Method, apparatus, device and medium for processing visual tasks using a general model

Abstract

Methods, devices, equipment and media for processing visual tasks using a general model are provided. In one method, visual data and prompt data associated with a visual task are received, and the visual task specifies determining a processing result associated with the prompt data from the visual data. A general prompt representation of the prompt data is obtained, and the prompt data includes any one of an image format and a language expression format. A general visual representation of the visual data is obtained, and the visual data includes any one of an image format and a video format. Based on the general prompt representation and the general visual representation, a processing result is determined. Using the exemplary implementation of the present disclosure, different visual tasks can be processed in a unified manner, and further training data can be shared across multiple visual tasks, and the processing performance of the general processing model can be improved.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure generally relate to visual task processing, and more particularly to methods, apparatuses, devices, and computer-readable storage media for processing visual tasks using a general processing model. Background Art

[0002] Machine learning technology has been widely used to process visual tasks related to instance perception. In order to improve the processing performance of visual tasks, visual tasks are usually subdivided into a large number of branches, such as object detection, object segmentation, object tracking, and so on. Although the refinement of tasks brings convenience to the development of specific applications, the overly diverse task definitions make it difficult for models designed independently for specific tasks to learn common knowledge between different tasks and fields. At this time, how to train machine learning models in a more effective way to improve the processing performance of various visual tasks has become a difficulty and hot spot in the field of visual processing. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for processing a visual task using a general processing model is provided. In the method, visual data and prompt data associated with the visual task are received, and the visual task specifies determining a processing result associated with the prompt data from the visual data. A general prompt representation of the prompt data is obtained, and the prompt data includes any one of an image format and a language expression format. A general visual representation of the visual data is obtained, and the visual data includes any one of an image format and a video format. Based on the general prompt representation and the general visual representation, a processing result is determined.

[0004] In a second aspect of the present disclosure, a device for processing a visual task using a general processing model is provided. The device comprises: a receiving module configured to receive visual data and prompt data associated with the visual task, wherein the visual task specifies determining a processing result associated with the prompt data from the visual data; a first acquisition module configured to acquire a general prompt representation of the prompt data, wherein the prompt data includes any one of an image format and a language expression format; a second acquisition module configured to acquire a general visual representation of the visual data, wherein the visual data includes any one of an image format and a video format; and a determination module configured to determine the processing result based on the general prompt representation and the general visual representation.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar figure labels represent the same or similar elements, wherein:

[0009] Figure 1 A block diagram showing the classification of visual tasks according to an exemplary implementation of the present disclosure;

[0010] Figure 2 A block diagram showing processing of visual tasks using a general processing model according to some implementations of the present disclosure;

[0011] Figure 3 A block diagram illustrating visual data and cue data associated with different visual tasks according to some implementations of the present disclosure;

[0012] Figure 4 A block diagram illustrating the structure of a general processing model according to some implementations of the present disclosure;

[0013] Figure 5 A block diagram of extracting a generic prompt representation according to some implementations of the present disclosure is shown;

[0014] Figure 6 A block diagram illustrating a process for performing multiple visual tasks according to some implementations of the present disclosure;

[0015] Figure 7 A flowchart of a method for processing visual tasks using a general processing model according to some implementations of the present disclosure is shown;

[0016] Figure 8 A block diagram showing an apparatus for processing visual tasks using a general processing model according to some implementations of the present disclosure; and

[0017] Fig. 9 A block diagram of a device capable of implementing multiple implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The implementation of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the implementations described herein. On the contrary, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the implementation methods of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly related to the time when the event occurs or the condition is satisfied. For example, in some cases, the subsequent action may be executed immediately when the event occurs or the condition is satisfied; while in other cases, the subsequent action may be executed some time after the event occurs or the condition is satisfied.

[0026] Example Environment

[0027] Instance perception is one of the basic tasks of computer vision, and it has a variety of downstream applications in autonomous driving, intelligent monitoring, content understanding, etc. In the context of the present disclosure, visual data may include image data and video data, and an object may represent an entity with a tangible shape in the visual data, such as but not limited to a person, an animal, an object, etc. For example, in an autonomous driving environment, individual vehicles in a road environment may be identified and tracked; in an intelligent monitoring system, individual products in a production process may be identified and tracked; and so on.

[0028] Instance-aware tasks aim to find specific objects specified by some query. Here, the query can include, for example, a classification name, a language expression, and an object annotation. Figure 1 Describe the outline of the visual task, the Figure 1 A block diagram 100 of the classification of visual tasks according to an exemplary implementation of the present disclosure is shown. Figure 1 As shown, the visual task can involve multiple tasks, for example, mainly involving 10 tasks and each task is distributed in Figure 1 The vertices of the cube shown. Figure 1 As shown, the x-axis of the coordinate system represents time, that is, whether the visual data includes image data collected at different time points. According to the direction indicated by the x-axis, the visual data can change from single-frame image data to video data. The y-axis represents the reference, that is, whether the visual task specifies to identify the object specified by the prompt from the visual data. According to the direction of the z-axis, the visual task can change from no prompt to including prompts in the format of language expression or image annotation. The z-axis represents the format, that is, the format of the object to be identified. According to the direction of the format accuracy indicated by the z-axis from low to high, the format can change from a bounding box to a mask.

[0029] Specifically, Figure 1As shown, starting from the object detection (OD) 110 at the origin of the coordinate axis, when moving along the format axis z, the coarse-grained bounding box of the object detection 110 can be changed to the fine mask of the instance segmentation (IS) 114. Moving along the time axis x, the visual data can be changed from static image data to dynamic video data. At this time, there can be multiple object tracking (MOT) 111, multi-object tracking and segmentation (MOTS) and video instance segmentation (VIS) 115. Moving along the reference axis y, referring to the language expression or bounding box / mask given in the previous video frame and other reference information, there can be three language-guided tasks: Referring Expression Comprehension (REC) 113, Referring Expression Segmentation (RES) 117 and Referring Video Object Segmentation (R-VOS), as well as two annotation-guided tasks: Single Object Tracking (SOT) 112 and Video Object Segmentation (VOS) 116.

[0030] Generally speaking, task processing models can be developed separately for refined tasks. Although refined tasks bring convenience to the development of specific applications, overly diverse task definitions divide the entire instance perception field into a large number of fragments. Currently, most instance perception technical solutions are only applicable to one or more specific task branches and are trained based on the sample data of the specific task branch. At this time, models designed independently for specific tasks are difficult to learn and share common knowledge between different tasks and fields, resulting in redundant parameters and may ignore the collaboration between different tasks. For example, object detection data enables the model to recognize common objects, which in turn can improve the performance of REC and RES. Furthermore, due to the limitations of fixed-size classifiers, traditional object detection models are difficult to perform joint training on multiple data sets with different label vocabularies, which results in the inability to dynamically change the classification of objects to be detected during the reasoning process.

[0031] At this point, how to train machine learning models in a more effective way and enable them to handle various types of visual tasks with higher performance has become a difficulty and hot spot in the field of visual processing.

[0032] Overview of visual processing tasks

[0033] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for processing visual tasks using a general processing model is proposed. Figure 2 Describes an overview of an exemplary implementation according to the present disclosure, Figure 2 A block diagram 200 for processing visual tasks using a general processing model according to some implementations of the present disclosure is shown. According to an exemplary implementation of the present disclosure, the specific types of each visual task are not distinguished, but a general processing model 220 is developed for multiple types of visual tasks 230. According to an exemplary implementation of the present disclosure, the visual task 230 can be any of the visual tasks described above, including but not limited to: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, reference expression understanding, reference expression segmentation, reference video object segmentation, single object tracking, video object segmentation, and the like.

[0034] like Figure 2 As shown, a sample 210 associated with a visual task 230 may be received, and the sample 210 may include visual data 212 and prompt data 214. Here, the visual task 230 may describe determining a processing result 240 associated with the prompt data 214 from the visual data 212. Further, a general prompt representation 224 of the prompt data 214 may be obtained, where the prompt data 214 includes any one of an image format and a language expression format. A general visual representation 222 of the visual data 212 may be obtained, where the visual data 212 may include any one of an image format and a video format. Further, a processing result 240 may be determined based on the general prompt representation 224 and the general visual representation 222.

[0035] In the context of the present disclosure, representation refers to features extracted for a certain entity, which can be implemented based on embedding, for example. Specifically, the visual representation can be features extracted from the visual data 212, and the prompt representation can be features extracted from the prompt data 214. Here, the general processing model 220 can provide a general instance-aware technical solution. Generally speaking, various instance-aware tasks are aimed at finding specific objects based on certain queries. Here, the general processing model 220 can provide a general instance-aware technical solution, and the general instance-aware model can be applied to specific downstream tasks.

[0036] According to an exemplary implementation of the present disclosure, a next generation universal instance perception (UNIversalINstance perception model of the NEXT generation, abbreviated as UNINEXT) technical solution is proposed, and the universal processing model 220 can be implemented based on the technical solution. UNINEXT can reformulate different instance perception tasks into a unified instance perception paradigm, and can flexibly perceive instances of different types of objects by simply changing the input prompt.

[0037] Using the exemplary implementation of the present disclosure, a large amount of data for different tasks can be used to jointly train the general processing model 220, which is particularly beneficial for tasks that lack training data. Further, the unified instance representation model can provide higher performance and reduce redundant calculations when processing multiple tasks simultaneously. The general processing model can be used to perform multiple independent tasks (for example, including classic image-level tasks (object detection and instance segmentation), visual and language tasks (reference expression understanding and segmentation), and video-level object tracking tasks, etc.), and higher accuracy is achieved.

[0038] Detailed process of handling visual tasks

[0039] Figure 3 A block diagram 300 of visual data and cue data associated with different visual tasks according to some implementations of the present disclosure is shown. Figure 3 As shown, visual data 212 may include image format, in which case each visual data 212 may include a single image frame. Alternatively and / or additionally, visual data 212 may include video format, in which case each visual data 212 may include multiple video frames in a video sequence.

[0040] According to an exemplary implementation of the present disclosure, the prompt data 214 may include multiple formats, for example, any one of an image format and a language expression format. Specifically, the language expression format may further include a format based on a classification name and a format of a description expressed in language. Prompt 310 relates to a format based on a classification name, and prompt 312 relates to a format of a description expressed in language. Further, prompt 314 represents a prompt based on an image format, where the annotation in the image indicates that the object "zebra" is expected to be recognized. Further, the task set 320 of the visual task 230 can be divided into three types according to different prompts: (1) tasks that use classification names as prompts (object detection, instance segmentation, VIS, MOT, MOTS); (2) tasks that use language expressions as prompts (REC, RES, R-VOS); and (3) tasks that use reference annotations as prompts (SOT, VOS).

[0041] Using the exemplary implementation of the present disclosure, different instances can be flexibly perceived by simply changing the input prompts, thereby achieving the corresponding visual task 230. In order to handle different prompting methods, the general processing model 220 may include a prompt encoder, which may include a reference text encoder and a reference visual encoder to process prompts in language expression format and prompts in image format, respectively. Further, the general processing model 220 may include a visual encoder for extracting a representation of visual data. Then, a fusion module can be used to enhance the original visual features and prompt features of the current image. In this way, deep information exchange can be achieved, and a highly discriminative representation is provided for the subsequent instance prediction step.

[0042] See also Figure 4 Describing more details of the general processing model 220, the Figure 4 A block diagram 400 is shown showing the structure of a general processing model according to some implementations of the present disclosure. Figure 4 As shown, visual data 212 and prompt data 214 may be received. A currently known and / or future developed image-based encoder may be used to determine a general visual representation 222 of the visual data 212 (e.g., using symbol F v Further, a hint encoder may be used to determine a generic hint representation 224 of the hint data 214 (eg, using the symbol F p to indicate).

[0043] According to an exemplary implementation of the present disclosure, in the process of obtaining the general prompt representation, the general prompt representation can be extracted based on the format of the prompt data. Here, the general processing model 220 is a general model set for multiple visual tasks 230, and the prompt data 214 can include multiple formats. By generating the corresponding general prompt representation 224 according to the specific format of the prompt data 214, the various features of the prompt data 214 can be determined in a more accurate manner, thereby improving the accuracy of the later processing visual tasks.

[0044] Here, the prompt encoder can convert the original prompts of multiple modalities (image-related prompts and language-related prompts) into a unified form. Specifically, if it is determined that the format of the prompt data 214 is a language expression format, the language expression encoder can be used to extract the prompt representation 224. In order to process the prompts related to the language, a language encoder currently known and / or to be developed in the future (for example, Enc L encoder).

[0045] Specifically, for tasks that use category names as prompts, the category names that appear in the current dataset can be connected as language expressions. Taking the COCO dataset as an example, assuming that the dataset involves category names: people, bicycles, ..., toothbrushes, then the category names in the dataset can be connected, and the language expression can be expressed as "people, bicycles, ..., toothbrushes". For datasets that include animal images, the following can be formed: Figure 3 The language expression shown in prompt 310 in the figure is “people, ..., giraffes, ..., zebras”. Then, the above language expression can be input into the language encoder Enc L , and then get the prompt embedding Here, L represents the length of the sequence and d represents the width, thereby obtaining the corresponding general prompt representation 224. For the prompt of the language expression format (e.g., Figure 3 Prompt 312 in the above example) can directly input the language expression into the language encoder Enc L .

[0046] According to an exemplary implementation of the present disclosure, if it is determined that the format of the prompt data 214 is an image format, a general prompt representation 224 is extracted based on an extended image including an annotation image specified by the prompt data. In this way, the features of the image as a prompt can be determined more accurately, thereby improving post-processing performance. Figure 5 More details on processing hint data 214 in image format are described in the Figure 5 A block diagram 500 is shown for extracting a generic hint representation according to some implementations of the present disclosure.

[0047] According to an exemplary implementation of the present disclosure, in order to extract fine-grained visual features and make full use of annotations, an additional reference visual encoder may be used. like Figure 5 As shown, the annotation image 520 can be determined in the reference image 510 based on the prompt, and further, a predetermined range (e.g., an integer multiple of the annotation image) around the annotation image 520 can be selected as the extended image 530. For example, the annotation image 520 in the reference image can be used as the center, and a region with 2 2 The extended image 530 may be used as a template and the universal prompt representation 224 may be determined. According to an exemplary implementation of the present disclosure, the extended image 530 may be adjusted to a fixed size of 256×256 (or other sizes).

[0048] Specifically, a first representation associated with the prompt data (eg, Figure 5The template portion 522 in the extended image 530). In order to introduce more accurate target information, an additional channel (called target prior) can be added to the extended image to form a 4-channel combined feature 540. Here, the prior value of the pixel data can indicate whether the pixel data belongs to the annotated image. Specifically, the prior value of the pixel in the annotated image 520 is 1 and the prior values ​​of other pixels can be set to 0. Alternatively and / or additionally, the prior data can be set based on other methods. A second representation (e.g., prior data 524) associated with the prompt data can be determined based on the prior values ​​of multiple pixel data in the extended image 530. Further, a splicing operation can be performed on the target portion 522 and the prior data 524, so that the extended image 530 can be expanded from three channels to four channels, and a combined feature 540 including four channels can be obtained.

[0049] Then, the combined feature 540 can be input into the reference visual encoder In order to obtain a hierarchical feature representation 550. Here, the hierarchical feature representation 550 can be represented as a hierarchical feature pyramid {C3, C4, C5, C6}, and the dimensions of the features of each layer can be 32×32, 16×16, 8×8 and 4×4, respectively. Further, in order to maintain the fine annotation information and obtain the hint embedding in the same format as other tasks, a merge operation can be performed. In other words, first convert the features of all levels to 32×32 (or other dimensions), then add and perform a flattening operation to form the final hint embedding

[0050] According to an exemplary implementation of the present disclosure, the prompt embedding may be generated based on the following formula 1:

[0051]

[0052] In Formula 1, F p Representative general prompts indicate that Language encoder, represents a visual encoder, expression represents a language expression in the prompt data, categories represents a category name in the prompt data, cancat represents a connection operation, template represents a template part, prior represents a priori part, and merge represents a combination operation. Using the exemplary implementation of the present disclosure, prompt data 214 in different formats can be processed separately, thereby obtaining a universal prompt representation 224 that can represent various aspects of various data formats.

[0053] return Figure 4Describe further processing for the general hint representation 224 and the general visual representation 222. According to an exemplary implementation of the present disclosure, a fusion module 410 can be used to enhance the original hint embedding through the image context, and make the original visual embedding have the function of hint perception. Specifically, the fusion module 410 can be based on a bidirectional attention operation, so that the general representation can consider the correlation between the general visual representation 222 and the general hint representation 224. That is, the attention operation can be performed on the general hint representation 224 and the general visual representation 222 to obtain an updated general hint representation and an updated general visual representation, respectively.

[0054] Specifically, a bidirectional attention operation may be performed on the general prompt representation 224 and the general visual representation 222, and then the result of the bidirectional attention operation may be used to update the general prompt representation 224 and the general visual representation 222 respectively. In this way, an updated general prompt representation F′ may be obtained. p and the updated general visual representation F′ v .

[0055] A bidirectional cross-attention module (Bi-XAtt) can be used to retrieve information from different inputs and then add the retrieved information to the original embedding. Along the first direction from the visual data to the prompt data, an attention operation can be performed on the general prompt representation and the general visual representation to determine the embedding F between the general prompt representation and the general visual representation. v2p (e.g., called the first attention representation), we can further use the embedding F v2p To update the general hint representation. In this way, the general hint representation 224 can be enhanced by the image context.

[0056] According to an exemplary implementation of the present disclosure, along the second direction from the prompt data to the visual data, an attention operation may be performed on the general prompt representation and the general visual representation to determine an embedding F between the general prompt representation and the general visual representation. p2v (e.g., called the second attention representation), and further utilize F p2v To update the general visual representation. In this way, the original visual embedding can be made to have the function of prompt perception. Specifically, the updated prompt embedding and visual embedding can be determined based on the following formula 2:

[0057] F p2v , F v2p =Bi-XAtt(F v , F p )

[0058] F′ v =F v +F p2v ; F′p =F p +F v2p

[0059] Formula 2

[0060] In Formula 2, F p represents the original prompt embedding, F v represents the original visual embedding, F p2v Represents the attention embedding of cue-visual direction, F v2p Attention embedding representing the visual-cue direction, F′ v represents the updated general visual embedding based on the attention operation, F′ p Represents the updated general cue embedding based on the attention operation. The expressiveness of cue embedding and visual embedding can be improved through the bidirectional cross-attention module.

[0061] The above has described the general hint representation F′ for determining the update p and the updated general visual representation F′ v , and then, the updated representation can be used to determine the result of the instance-aware task. According to an exemplary implementation of the present disclosure, the updated general prompt representation F′ can be used p and the updated general visual representation F′ v Specifically, the encoder 420 and decoder 450 architecture based on a transformer can be used to obtain the processing result using F′ p and F′ v to determine the processing results.

[0062] Specifically, multiple candidate query results corresponding to the prompt data can be queried in the visual data based on the updated general prompt representation and the updated general visual representation. In order to provide a more flexible instance query method, an object detector can be used as an instance decoder. At this time, N instance suggestions can be obtained, and then matching instances are retrieved from these suggestions based on the prompt. This flexible retrieval mechanism can overcome the shortcomings of traditional fixed-size classifiers and enable joint training of data from different tasks and fields.

[0063] Furthermore, the scores of the multiple candidate query results can be determined based on the multiple candidate query results and the weights associated with the prompt data, and then the processing results can be determined according to the scores of the multiple candidate query results. Here, the score can represent the degree of match between the candidate query result and the prompt data. In other words, the candidate query result that best matches the prompt data can be selected based on the score as the final processing result of the visual task.

[0064] According to an exemplary implementation of the present disclosure, an encoder-decoder architecture based on a detection transformer (DETECTIONTRANSFORMER, abbreviated DETR) can be used to achieve more flexible instance queries. Here, the encoder can use hierarchical cue-perceived visual features as input. With the help of multi-scale deformable self-attention, target information from different scales can be fully exchanged to obtain stronger instance features for subsequent instance decoding processes. In addition, an auxiliary prediction head can be attached to the end of the encoder to generate N initial reference points with the highest scores as input to the decoder.

[0065] The decoder takes as input the enhanced multi-scale features, N reference points from the encoder, and N object queries. According to an exemplary implementation of the present disclosure, object queries are crucial in instance perception tasks. Two query generation strategies can be used: (1) static queries, which do not change with images or prompts; (2) dynamic queries, which are conditioned on prompts. Static queries can be implemented based on the currently developed nn.Embedding() function. For dynamic queries, the enhanced F′ can be first pooled along the sequence dimension. v , obtain the global representation, and then repeat N times to execute.

[0066] According to an exemplary implementation of the present disclosure, static queries generally perform better than dynamic queries. The potential reason is that static queries contain richer information than dynamic queries and have better training stability. With the help of deformable attention, the retrieval module 452 can effectively retrieve visual features of perceptual cues and learn strong instance embeddings.

[0067] At the end of the decoder, a set of prediction modules can be used to obtain the final instance prediction. Specifically, the dynamic convolution-based regression module 454 and the mask module 456 can generate the bounding box and mask of the target respectively. In addition, a contrast module 458 is introduced to associate the current detection result with the previous track in MOT, MOTS and VIS based on contrast learning. According to an exemplary implementation of the present disclosure, N potential instance suggestions can be mined.

[0068] According to an exemplary implementation of the present disclosure, different types of prompt data can be processed in different ways to obtain corresponding weight matrices. In this way, the weight matrix can be made to describe the factors that are expected to be paid attention to in the prompt data in a more accurate way, thereby improving the accuracy of determining the degree of match between the candidate query and the prompt. Specifically, if the prompt data is determined to be a classification name defined in a language expression format, the weight is determined based on an updated general prompt representation. Alternatively and / or additionally, if the prompt data is determined to be any one of a description and an image format defined in a language expression format, the weight is determined based on the global average pooling (Global Average Polling, abbreviated GAP) of the updated prompt representation.

[0069] It will be appreciated that not all suggestions correspond to hints. Therefore, the hint embedding can be exploited to further retrieve instances of truly matching objects from these suggestions. Specifically, for the embedding F′ p For tasks that use categories as prompts, the embedding of each category name can be used as the weight matrix In addition, for both expression-based and annotation-based prompt tasks, the prompt embedding F′ can be aggregated by using global average pooling (GAP) along the sequence dimension. p , and then obtain the weight matrix, as shown in Formula 3 below.

[0070]

[0071] Finally, the instance cue matching score S can be calculated based on the matrix multiplication of the target feature and the transposed weight matrix S = F ins W T Here, the matching score can be supervised by the focal loss. Different from the fixed-size classifier in the existing technical solutions, the retrieval module here can select objects through the hint instance matching mechanism. This flexible design enables UNINEXT to jointly train on huge datasets with different label vocabularies from different tasks, and then learn universal instance representations.

[0072] Having described the architecture of the general processing model, according to an exemplary implementation of the present disclosure, the general processing model can be gradually trained in multiple different stages. For example, the training process can include three stages: (1) general perception pre-training (2) image-level joint training (3) video-level joint training. In this way, different training stages can enable the general processing model to gradually learn relevant knowledge of different types of visual tasks from easy to difficult, thereby enabling the sharing of training data across different visual tasks and improving the processing performance of visual tasks.

[0073] According to an exemplary implementation of the present disclosure, the first stage of training can be performed based on an object detection dataset so that the general processing model describes the association between the image data in the object detection dataset and the bounding box of the object in the image data. In the first stage, UNINEXT can be preprocessed on a large-scale object detection dataset (e.g., Objects365) to learn general knowledge about objects. Since there is no mask annotation in Objects365, two auxiliary losses based on the BoxInst technical solution can be introduced to train the mask branch. At this time, the loss function of the first stage can be expressed as follows:

[0074]

[0075] In formula 4, represents the loss function of the first stage, represents the loss function associated with object recognition in the object detection dataset, represents the loss function associated with the bounding box of the object, and Represents a mask-related loss function determined based on the BoxInst technical solution. Using the exemplary implementation of the present disclosure, the annotated data in the object detection dataset can be fully utilized to improve the performance of the general processing model in performing basic object detection-related tasks and improve the general perception ability of the model.

[0076] According to an exemplary implementation of the present disclosure, after the first stage of training, the second stage of training can be performed based on the mixed object detection dataset. In this way, the general processing model can describe the association between the image data in the mixed object detection dataset, the bounding box of the object in the image data, and the mask of the object. In the second stage, training can be performed based on the model parameters obtained in the first stage. Specifically, UNINEXT can be jointly fine-tuned on an image dataset that includes richer information (for example, a mixed dataset of COCO and RefCOCO, RefCOCO+ and RefCOOg).

[0077] At this point, manually labeled mask annotations can be used, for example, existing loss functions such as Dice loss and Focal loss can be used to determine the mask from the original video. In addition, in order to prevent the model from forgetting the previously learned image-level task knowledge, the image-level dataset can also be converted into a pseudo video for joint training with other video datasets. The loss function of the second stage can be expressed as follows:

[0078] In formula 5, represents the loss function of the second stage, represents the loss function associated with object recognition in the hybrid object detection dataset, represents the loss function associated with the bounding box of the object, and represents the loss function associated with the mask of the object.

[0079] According to an exemplary implementation of the present disclosure, after the second stage training, the third stage training is performed based on the video data set. In this way, the general processing model can describe the association between the image data in the video in the video data set, the bounding box of the object in the image data, and the mask of the object. In other words, with the help of richer training data in the video data set, the general processing model can obtain knowledge related to video processing. The training data for the third stage includes pseudo videos generated from COCO, RefCOCO / g / +, SOT&VOS data sets (GOT-10K, LaSOT, TrackingNet, etc.), MOT&VIS data sets (BDD100K, VIS19, OVIS) and R-VOS data sets. The loss function of the third stage can be expressed as follows:

[0080]

[0081] In formula 3, represents the loss function of the third stage, represents the loss function associated with object recognition in video datasets, represents the loss function associated with the bounding box of the object, and represents the loss function associated with the mask of the object.

[0082] According to an exemplary implementation of the present disclosure, in order to further learn the relevant knowledge of performing object tracking between multiple video frames in a video, a general processing model can be trained based on contrastive learning technology. Specifically, in the third stage, the general processing model can be used based on contrastive learning technology to narrow the distance between the general visual representations of two image data of the same object in the video, and push away the distance between the general visual representations of two image data of different objects in the video. At this time, positive sample pairs can be constructed using video frames including the same object, and negative sample pairs can be constructed using video frames including the same object, so that the general processing model can learn about object tracking. During this period, a reference visual encoder for SOT&VOS and an additional contrast module for association can be introduced and optimized. At this time, the loss function of the third stage can be expressed as follows:

[0083]

[0084] In Formula 7, Represents the related loss function of contrastive learning. At this time, the distance between the universal visual representations output by the universal processing model for the two image frames in the positive sample pair is small, and the distance between the universal visual representations output by the universal processing model for the two image frames in the negative sample pair is large. In this way, the performance of the universal processing model can be improved when processing video-related visual tasks in the future.

[0085] Once a general processing model has been obtained, it can be used to perform different types of visual tasks. Here, visual tasks may include but are not limited to: object detection, instance segmentation, multiple object tracking, multiple object tracking and segmentation, video instance segmentation, reference expression understanding, reference expression segmentation, reference video object segmentation, single object tracking, video object segmentation. Figure 4 As shown, the tasks may be divided into different task subsets, for example, task subset 460 may involve object detection related tasks, task subset 462 may involve reference related tasks, and task subset 464 may involve video related tasks.

[0086] During the inference phase, for tasks with categories as cues, UNINEXT can predict instances of different categories and associate them with previous trajectories. The association can be done in an online manner and can be based on learned instance embeddings. For tasks with expressions and annotations as cues, the object instance with the highest matching score for a given cue can be directly selected as the final result. See Figure 6 Describes more details about performing visual tasks, the Figure 6 Block diagram 600 is shown of a process for performing multiple visual tasks according to some implementations of the present disclosure.

[0087] like Figure 6 As shown, multiple query results 610 can be obtained from the visual data 212. At this time, different processing results will be output for different prompt data. For example, assuming that the prompt data is as follows Figure 3 The prompt 310 (“People, …, long Giraffe , …, zebra”). At this time, the prompt belongs to the type represented by the classification name, and the general processing model can output the processing result 620, that is, the processing result 620 includes a mask of the area where the giraffe is located identified from the visual data 212.

[0088] For another example, suppose the prompt data is Figure 3 320 ("giraffe on the right") shown in . At this time, the prompt is of the type directly represented by a language expression, and the general processing model can output a processing result 630, that is, the processing result 630 includes a mask of the area where the giraffe located on the right side is located, which is identified from the visual data 212.

[0089] For another example, suppose the prompt data is as follows Figure 3 . At this time, the prompt belongs to the image type represented by the annotation (that is, it is expected to identify "zebra" from the visual data 212). At this time, the general processing model can output the processing result 630, that is, the processing result 630 includes a mask of the area where the zebra is located identified from the visual data 212.

[0090] will understand, although Figure 6 Taking object detection as a specific example of visual task 230 only, the processing result of using a general processor to identify the object specified by the prompt data from the visual data 212 is described. Alternatively and / or additionally, the general processing model can be used to perform other tasks. According to an exemplary implementation of the present disclosure, the instance representation model described above can be implemented on different backbone networks. For example, ResNet-50 and ConvNeXtLarge can be used as visual encoders. The BERT model can be used as a prompt encoder, and the parameters of the encoder can be trained in the first stage and the second training. The existing transformer encoder-decoder architecture can be used, for example, 6 encoder layers and 6 decoder layers can be used. The number of queries N of the object query can be set to 300. The AdamW optimizer can be used and the weight decay is 0.05.

[0091] According to an exemplary implementation of the present disclosure, existing instance-aware tasks are divided into three categories. (1) Object detection, instance segmentation, MOT, MOTS, and VIS. These instance-aware tasks use the category name as a hint to find instances of a specific category. (2) REC, RES, and R-VOS. These instance-aware tasks use text expressions as hints to locate specific instances. (3) SOT and VOS use reference annotations as hints, for example, using the reference annotation given in the first frame as a hint to track instances associated with the reference annotation in the video. All of the above tasks aim to find instances of objects specified by certain hints. Based on the above commonalities, all instance-aware tasks can be restated as general object discovery and retrieval problems and expressed through a unified model architecture and learning paradigm.

[0092] The model configured as above can be used to process different visual tasks. Specifically, Tables 1 to 4 below respectively show the test results of performing different tasks using general instance representations. Table 1 shows a performance comparison of object detection tasks based on multiple technical solutions. As shown in Table 1, the first column shows the models adopted by multiple technical solutions as a comparison basis, the second column shows the structure of the backbone network adopted by each model, and the third column shows the average accuracy of each technical solution. As shown in bold in Table 1, when the UNINEXT technical solution according to the present disclosure is adopted, a higher average accuracy can be achieved.

[0093] Table 1 Performance comparison of object detection tasks

[0094]

[0095] Table 2 shows the performance comparison of the reference expression understanding task based on multiple technical solutions. As shown in bold in Table 2, when the UNINEXT technical solution according to the present disclosure is adopted, a higher average accuracy rate can be achieved.

[0096] Table 2 Performance comparison of reference expression understanding tasks

[0097]

[0098] Table 3 shows the performance comparison of the reference expression segmentation task based on multiple technical solutions. The first column on the left side of Table 3 shows multiple methods, and the right side shows multiple performance indicators for measuring the task execution performance. As shown in bold in Table 3, when the UNINEXT technical solution according to the present disclosure is adopted, each performance indicator is the optimal value.

[0099] Table 3 Performance comparison of reference expression segmentation tasks

[0100]

[0101] The experimental data in Tables 1 to 3 show that the proposed UNINEXT technical solution can unify the object discovery and retrieval paradigms related to multiple visual tasks. Extensive experiments show that compared with multiple existing technical solutions that have been proposed, UNINEXT can achieve better results when performing a large number of challenging visual tasks.

[0102] Using the exemplary implementation of the present disclosure, UNINEXT can learn powerful universal representations of massive data from various tasks and perform various instance perception tasks using a single model with the same model parameters. A large number of experiments have shown that UNINEXT has achieved excellent results on a large amount of test data. In this way, UNINEXT can integrate scattered instance perception tasks into a whole, and then conduct joint training on different tasks and fields without the need to develop models separately for each specific task. Furthermore, UNINEXT can use a unified model with the same model parameters and can achieve excellent performance on multiple instance perception tasks.

[0103] Example Process

[0104] Figure 7 A flow chart of a method 700 for processing a visual task using a general processing model according to some implementations of the present disclosure is shown. At box 710, visual data and prompt data associated with a visual task are received, and the visual task specifies determining a processing result associated with the prompt data from the visual data. At box 720, a general prompt representation of the prompt data is obtained, and the prompt data includes any one of an image format and a language expression format. At box 730, a general visual representation of the visual data is obtained, and the visual data includes any one of an image format and a video format. At box 740, a processing result is determined based on the general prompt representation and the general visual representation.

[0105] According to an exemplary implementation of the present disclosure, obtaining the general prompt representation includes: extracting the general prompt representation based on the format of the prompt data.

[0106] According to an exemplary implementation of the present disclosure, extracting a universal prompt representation includes at least any one of the following: in response to determining that the format of the prompt data is a language expression format, extracting the prompt representation using a language expression encoder; in response to determining that the format of the prompt data is an image format, extracting the universal prompt representation based on an extended image including an annotated image specified by the prompt data.

[0107] According to an exemplary implementation of the present disclosure, extracting a general prompt representation based on an extended image includes: determining a first representation associated with the prompt data based on multiple pixel data in the extended image; determining a second representation associated with the prompt data based on prior values ​​of the multiple pixel data in the extended image, the prior value of the pixel data in the multiple pixel data indicating whether the pixel data belongs to the annotated image; and combining the first representation and the second representation to form a general prompt representation.

[0108] According to an exemplary implementation of the present disclosure, determining a processing result includes: performing an attention operation on a general prompt representation and a general visual representation to respectively update the general prompt representation and the general visual representation; and obtaining a processing result using the updated general prompt representation and the updated general visual representation.

[0109] According to an exemplary implementation of the present disclosure, updating the general prompt representation includes: performing a first-direction attention operation on the general prompt representation and the general visual representation to determine a first attention representation between the general prompt representation and the general visual representation; and updating the general prompt representation using the first attention representation.

[0110] According to an exemplary implementation of the present disclosure, updating the general visual representation includes: performing a second-direction attention operation on the general prompt representation and the general visual representation to determine a second attention representation between the general prompt representation and the general visual representation; and updating the general visual representation using the second attention representation.

[0111] According to an exemplary implementation of the present disclosure, obtaining processing results includes: based on an updated general prompt representation and an updated general visual representation, querying the image data for multiple candidate query results corresponding to the prompt data; based on the multiple candidate query results and weights associated with the prompt data, respectively determining scores of the multiple candidate query results; and determining the processing results based on the scores of the multiple candidate query results.

[0112] According to an exemplary implementation of the present disclosure, it further includes at least any one of the following: in response to determining that the prompt data is a category name defined in a language expression format, determining the weight based on an updated general prompt representation; in response to determining that the prompt data is any one of a description and an image format defined in a language expression format, determining the weight based on global average pooling of the updated prompt representation.

[0113] According to an exemplary implementation of the present disclosure, it further includes performing multi-stage training for the general processing model, and the multi-stage training includes: performing a first stage training based on an object detection dataset so that the general processing model describes the association relationship between the image data in the object detection dataset and the bounding box of the object in the image data.

[0114] According to an exemplary implementation of the present disclosure, it further includes: after the first stage training, performing the second stage training based on the mixed object detection dataset so that the general processing model describes the association relationship between the image data in the mixed object detection dataset, the bounding box of the object in the image data, and the mask of the object.

[0115] According to an exemplary implementation of the present disclosure, it further includes: after the second stage training, performing the third stage training based on the video dataset so that the general processing model describes the association relationship between the image data in the video in the video dataset, the bounding box of the object in the image data, and the mask of the object.

[0116] According to an exemplary implementation of the present disclosure, it further includes: performing a third stage of training based on a video dataset, so that the general processing model shortens the distance between the general visual representations of two image data including the same object in the video, and pushes away the distance between the general visual representations of two image data including different objects in the video.

[0117] According to an exemplary implementation of the present disclosure, the visual tasks include at least any one of the following: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, reference expression understanding, reference expression segmentation, reference video object segmentation, single object tracking, and video object segmentation.

[0118] Example devices and equipment

[0119] Figure 8 A block diagram of an apparatus 800 for processing a visual task using a general processing model according to some implementations of the present disclosure is shown. The apparatus includes: a receiving module 810 configured to receive visual data and prompt data associated with a visual task, wherein the visual task specifies determining a processing result associated with the prompt data from the visual data; a first acquisition module 820 configured to acquire a general prompt representation of the prompt data, wherein the prompt data includes any one of an image format and a language expression format; a second acquisition module 830 configured to acquire a general visual representation of the visual data, wherein the visual data includes any one of an image format and a video format; and a determination module 840 configured to determine a processing result based on the general prompt representation and the general visual representation.

[0120] According to an exemplary implementation of the present disclosure, the first acquisition module acquires the general prompt representation, including: an extraction module configured to extract the general prompt representation based on the format of the prompt data.

[0121] According to an exemplary implementation of the present disclosure, the extraction module includes at least any one of the following: a first extraction module, configured to extract the prompt representation using a language expression encoder in response to determining that the format of the prompt data is a language expression format; a second extraction module, configured to extract the general prompt representation based on an extended image including an annotated image specified by the prompt data in response to determining that the format of the prompt data is an image format.

[0122] According to an exemplary implementation of the present disclosure, the second extraction module includes: a first representation determination module, configured to determine a first representation associated with prompt data based on multiple pixel data in the extended image; a second representation determination module, configured to determine a second representation associated with the prompt data based on a priori values ​​of multiple pixel data in the extended image, the prior value of the pixel data in the multiple pixel data indicating whether the pixel data belongs to the annotated image; and a combination module, configured to combine the first representation and the second representation to form a common prompt representation.

[0123] According to an exemplary implementation of the present disclosure, the determination module includes: an update module, configured to perform attention operations on the general prompt representation and the general visual representation to update the general prompt representation and the general visual representation, respectively; and a result acquisition module, configured to obtain processing results using the updated general prompt representation and the updated general visual representation.

[0124] According to an exemplary implementation of the present disclosure, the update module includes: a first attention module, configured to perform a first direction of attention operation on a general prompt representation and a general visual representation to determine a first attention representation between the general prompt representation and the general visual representation; and a first update module, configured to update the general prompt representation using the first attention representation.

[0125] According to an exemplary implementation of the present disclosure, the update module includes: a second attention module, configured to perform a second direction of attention operation on the general prompt representation and the general visual representation to determine a second attention representation between the general prompt representation and the general visual representation; and a second update module, configured to update the general visual representation using the second attention representation.

[0126] According to an exemplary implementation of the present disclosure, a result acquisition module includes: a query module, configured to query multiple candidate query results corresponding to prompt data in the image data based on an updated general prompt representation and an updated general visual representation; a score determination module, configured to determine scores of multiple candidate query results respectively based on the multiple candidate query results and weights associated with the prompt data; and a processing result determination module, configured to determine a processing result based on the scores of the multiple candidate query results.

[0127] According to an exemplary implementation of the present disclosure, the device further includes: a first weight determination module, configured to determine the weight based on the updated general prompt representation in response to determining that the prompt data is a classification name defined in a language expression format; a second weight determination module, configured to determine the weight based on the global average pooling of the updated prompt representation in response to determining that the prompt data is any one of a description and an image format defined in a language expression format.

[0128] According to an exemplary implementation of the present disclosure, it further includes a training module, which is configured to perform multi-stage training for the general processing model. The training module includes a first training module, which is configured to perform a first stage training based on an object detection dataset so that the general processing model describes the association relationship between the image data in the object detection dataset and the bounding box of the object in the image data.

[0129] According to an exemplary implementation of the present disclosure, the training module further includes: a second training module, configured to perform a second stage of training based on a mixed object detection dataset after the first stage of training, so that the general processing model describes the association relationship between the image data in the mixed object detection dataset, the bounding box of the object in the image data, and the mask of the object.

[0130] According to an exemplary implementation of the present disclosure, the training module further includes: a third training module, configured to perform a third stage of training based on the video dataset after the second stage of training, so that the general processing model describes the association relationship between the image data in the video in the video dataset, the bounding box of the object in the image data, and the mask of the object.

[0131] According to an exemplary implementation of the present disclosure, the third training module further includes: a contrastive learning module, configured to perform a third stage training based on a video data set, so that the general processing model shortens the distance between the general visual representations of two image data including the same object in the video, and pushes away the distance between the general visual representations of two image data including different objects in the video.

[0132] According to an exemplary implementation of the present disclosure, the visual tasks include at least any one of the following: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, reference expression understanding, reference expression segmentation, reference video object segmentation, single object tracking, and video object segmentation.

[0133] Fig. 9 9 is a block diagram of a device 900 capable of implementing multiple implementations of the present disclosure. It should be understood that Fig. 9The computing device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Fig. 9 The computing device 900 shown may be used to implement the methods described above.

[0134] like Fig. 9 As shown, computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of computing device 900.

[0135] The computing device 900 typically includes a plurality of computer storage media. Such media may be any available media accessible to the computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the computing device 900.

[0136] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig. 9 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules that are configured to perform various methods or actions of various implementations of the present disclosure.

[0137] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the functions of the components of the computing device 900 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0138] Input device 950 may be one or more input devices, such as a mouse, keyboard, tracking ball, etc. Output device 960 may be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 900 may also communicate with one or more external devices (not shown) through communication unit 940 as needed, such as storage devices, display devices, etc., communicate with one or more devices that allow a user to interact with computing device 900, or communicate with any device that allows computing device 900 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0139] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0140] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0142] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0143] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0144] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for processing visual tasks using a general processing model, comprising: receiving visual data and prompt data associated with a visual task, the visual task specifying determining a processing result associated with the prompt data from the visual data; Acquiring a general prompt representation of the prompt data includes: extracting the general prompt representation based on a format of the prompt data, the prompt data including any one of an image format and a language expression format; obtaining a generic visual representation of the visual data, the visual data comprising any one of an image format and a video format; and The processing result is determined based on the universal prompt representation and the universal visual representation.

2. The method according to claim 1, wherein extracting the general prompt representation comprises at least any one of the following: In response to determining that the format of the prompt data is a language expression format, extracting the prompt representation using a language expression encoder; In response to determining that the format of the hint data is an image format, the generic hint representation is extracted based on an extended image including an annotation image specified by the hint data.

3. The method of claim 2, wherein extracting the generic hint representation based on the extended image comprises: determining a first representation associated with the prompt data based on a plurality of pixel data in the extended image; determining a second representation associated with the prompt data based on a priori values ​​of the plurality of pixel data in the extended image, the a priori values ​​of pixel data in the plurality of pixel data representing whether the pixel data belongs to the annotated image; as well as The first representation and the second representation are combined to form the common prompt representation.

4. The method according to claim 1, wherein determining the processing result comprises: performing an attention operation on the universal hint representation and the universal visual representation to update the universal hint representation and the universal visual representation, respectively; as well as The processing result is obtained using the updated universal prompt representation and the updated universal visual representation.

5. The method of claim 4, wherein updating the general prompt representation comprises: performing an attention operation in a first direction on the universal prompt representation and the universal visual representation to determine a first attention representation between the universal prompt representation and the universal visual representation; as well as The general hint representation is updated using the first attention representation.

6. The method of claim 4, wherein updating the common visual representation comprises: performing an attention operation in a second direction on the universal prompt representation and the universal visual representation to determine a second attention representation between the universal prompt representation and the universal visual representation; as well as The general visual representation is updated using the second attention representation.

7. The method according to claim 4, wherein obtaining the processing result comprises: Based on the updated general prompt representation and the updated general visual representation, querying the visual data for a plurality of candidate query results corresponding to the prompt data; Determining scores for the plurality of candidate query results respectively based on the plurality of candidate query results and weights associated with the prompt data; as well as The processing result is determined based on the scores of the plurality of candidate query results.

8. The method according to claim 7, further comprising at least one of the following: In response to determining that the prompt data is a category name defined in the language expression format, determining the weight based on the updated universal prompt representation; In response to determining that the hint data is in any one of the description defined in the language expression format and the image format, the weight is determined based on global average pooling of the updated hint representation.

9. The method according to claim 4, further comprising performing multi-stage training for the general processing model, the multi-stage training comprising: The first stage training is performed based on the object detection dataset so that the general processing model describes the association relationship between the image data in the object detection dataset and the bounding box of the object in the image data.

10. The method according to claim 9, further comprising: After the first stage training, a second stage training is performed based on the hybrid object detection dataset so that the general processing model describes the association relationship between the image data in the hybrid object detection dataset, the bounding box of the object in the image data, and the mask of the object.

11. The method according to claim 10, further comprising: After the second stage training, a third stage training is performed based on the video dataset so that the general processing model describes the association relationship between the image data in the video in the video dataset, the bounding box of the object in the image data, and the mask of the object.

12. The method according to claim 11, further comprising: The third stage training is performed based on the video dataset so that the general processing model shortens the distance between the general visual representations of two image data including the same object in the video, and expands the distance between the general visual representations of two image data including different objects in the video.

13. The method of claim 1, wherein the visual tasks comprise at least any one of: object detection, instance segmentation, multi-object tracking, multi-object tracking and segmentation, video instance segmentation, reference expression understanding, reference expression segmentation, reference video object segmentation, single object tracking, video object segmentation.

14. A device for processing a visual task using a general processing model, comprising: a receiving module configured to receive visual data and prompt data associated with a visual task, wherein the visual task specifies determining a processing result associated with the prompt data from the visual data; A first acquisition module, configured to acquire a general prompt representation of the prompt data, comprising: an extraction module, configured to extract a general prompt representation based on a format of the prompt data, the prompt data comprising any one of an image format and a language expression format; A second acquisition module configured to acquire a generic visual representation of the visual data, the visual data comprising any one of an image format and a video format; and A determination module is configured to determine the processing result based on the general prompt representation and the general visual representation.

15. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 13 when executed by the at least one processing unit.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 13.

Images