Single-level open vocabulary panorama segmentation

Abstract

A single-level framework for open vocabulary panorama segmentation is provided. One aspect provides a computing system comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to: receive an image; extracting a plurality of feature maps from the image by using a visual language model based on a convolutional neural network; generating a plurality of pixel features from the plurality of feature maps; generating a plurality of mask predictions from the plurality of pixel features; generating a plurality of intra-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of pixel features; using the plurality of feature maps to generate a plurality of vocabulary class predictions; performing geometric integration on the plurality of intra-vocabulary class predictions and the plurality of extravocabulary class predictions to generate a plurality of final class predictions; and outputting the plurality of mask predictions and the plurality of final class predictions.

Description

Background Art

[0001] Panoptic segmentation is an image class segmentation technique that combines the different tasks of semantic segmentation and instance segmentation to produce an overall unified output. It is a complex computer vision problem that aims to predict a set of non-overlapping masks, each with its corresponding class label. Semantic segmentation interprets high-level class semantic concepts. The semantic segmentation task may involve detecting objects within an image and grouping objects based on semantic categories, usually by assigning a semantic category to each pixel in the image. Instance segmentation groups foreground pixels into different object instances, allowing identification and separation of different individual objects within the image. The instance segmentation task can be performed using mask classification, where a set of bounding boxes and binary masks identifying different object instances are predicted. Summary of the invention

[0002] A single-stage framework for open vocabulary panoptic segmentation is provided. One aspect provides a computing system including a processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to: receive an image; extract a plurality of feature maps from the image using a convolutional neural network-based visual language model; generate a plurality of pixel features from the plurality of feature maps; generate a plurality of mask predictions from the plurality of pixel features; generate a plurality of in-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of pixel features; generate a plurality of out-of-vocabulary class predictions using the plurality of feature maps; perform geometric integration on the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions to generate a plurality of final class predictions; and output the plurality of mask predictions and the plurality of final class predictions.

[0003] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to implementations that solve any or all of the shortcomings pointed out in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A schematic diagram of an example computing system for performing open vocabulary panoptic segmentation using a single-stage framework is shown.

[0005] Figure 2A A diagram showing an example two-stage framework for open vocabulary panoptic segmentation.

[0006] Figure 2B Diagram showing an example naive single-stage framework for open vocabulary panoptic segmentation using a trainable visual-language model.

[0007] Figure 2CA diagram showing an example single-stage framework for open vocabulary panoptic segmentation using a frozen convolutional neural network based visual language model that can be used Figure 1 system or other suitable hardware and software to implement.

[0008] Figure 3 A diagram depicting the process flow of an example single-stage framework for open vocabulary panoptic segmentation that can be used Figure 2C The framework and Figure 1 system or other suitable hardware and software to implement.

[0009] Figures 4 to 9 Tables and graphs showing performance results of an example single-stage framework implementation for open vocabulary panoptic segmentation compared against known methods, which can be implemented using Figure 1 system or other suitable hardware and software to implement.

[0010] Fig.10 A flowchart showing an example method for performing open vocabulary panoptic segmentation using a single-stage framework that can be used Figure 1 system or other suitable hardware and software to implement.

[0011] Fig.11 It shows that it is possible to Figure 1 Schematic diagram of an example computing environment of a computer device. DETAILED DESCRIPTION

[0012] Panoptic segmentation seeks holistic scene understanding, combining semantic segmentation and instance segmentation tasks. One approach to panoptic segmentation includes closed vocabulary models. Due to the high cost of annotating fine-grained datasets, the number of semantic classes limited in closed vocabulary panoptic segmentation models typically ranges from tens to hundreds. This hinders the further application of closed vocabulary methods in real-world settings, where the number of possible semantic classes may be infinite. In contrast, open vocabulary methods attempt to remedy this shortcoming by segmenting arbitrary classes, including classes that are not accessible during the training procedure.

[0013] Open vocabulary segmentation can be implemented to use text embeddings of category names (which can be expressed in natural language) as label embeddings, rather than learning them from a training dataset. This allows segmenting and identifying objects from an open set of categories in different environments, enabling classification of objects with a wider vocabulary. To provide meaningful embeddings, pre-trained text encoders can be used to effectively capture the semantic meaning of words and phrases.

[0014] Current methods for open vocabulary panoptic segmentation are typically implemented using a two-stage framework with a multimodal model. For example, a visual language model can be implemented to jointly encode vision and language in a fusion model. Such a model can be used for its ability to map image and text features to a shared embedding space and learn aligned image-text feature representations from large-scale Internet data. Some models use pre-trained object detectors to extract visual representations and fine-tune downstream tasks using language supervision. Examples of visual language models include contrastive language-image pre-training (CLIP) models and large-scale image and noisy text embedding (ALIGN) models. With the advancement of large language models, this field has made rapid progress. For example, CLIP and ALIGN show that pre-training a dual encoder model on large-scale noisy image-text pairs with contrastive objectives can learn representations with cross-modal alignment capabilities and show strong performance in zero-shot downstream tasks (such as open vocabulary image recognition).

[0015] In a typical two-stage framework for adapting a visual language model for open vocabulary panoptic segmentation, an image is typically first processed by a mask generator to obtain a mask proposal (e.g., a mask prediction). Then, each masked image crop is generated and fed into a visual language model (e.g., CLIP) for classification. For example, one approach includes a two-stage pipeline that uses a class-independent mask generator and a frozen (i.e., non-trainable) CLIP encoder for cross-modal alignment. In some approaches, mask proposals are used as attention masks in the backbone of a visual language model to effectively avoid multiple forward passes of masked crops. Many variations of these frameworks have been considered. For example, some techniques involve using a stable diffusion model as a frozen backbone for a mask generator, thereby significantly improving performance. However, even with such advances, these approaches still rely on a two-stage framework in which the mask generator and the visual language classifier extract features from the original image alone (i.e., extract features from the original image multiple times), resulting in inefficiency and ineffectiveness.

[0016] In view of the above observations, an implementation of open vocabulary panoramic segmentation using a single-stage framework is provided. The single-stage framework for open vocabulary panoramic segmentation can be implemented in a variety of ways. In some implementations, the single-stage framework implements a visual language model backbone that unifies the mask generator and the visual language classifier. For example, a CLIP-based solution can be used to provide predicted masks and open vocabulary recognition. Although a shared feature extractor is a simple solution, there are other considerations. First, fine-tuning the visual language model backbone destroys the alignment between image and text features, resulting in poor performance on out-of-vocabulary categories. Existing methods rely on another separate backbone for the mask generator, but this increases the model size and computational cost. Secondly, visual language models (such as CLIP models) are usually pre-trained on relatively low resolution inputs. However, dense prediction tasks perform better at higher resolution inputs. This makes it difficult to directly apply such a pre-trained visual language model backbone to downstream dense prediction tasks. Therefore, existing methods perform mask segmentation and CLIP classification at different input scales, resulting in poor performance.

[0017] To alleviate the above considerations, both the mask generator and the visual language classifier can be built on a shared frozen convolution (FC) visual language model backbone, thereby producing a single-stage framework. For example, in some implementations, the framework includes a single FC-CLIP model as a shared feature extractor. Using a shared model backbone provides a design with relatively fast training and inference speed (compared to a two-stage framework). The frozen visual language model backbone keeps the pre-trained image-text feature alignment intact, allowing out-of-vocabulary classification. In some implementations, the frozen visual language model backbone acts as a powerful mask generator by attaching a lightweight pixel decoder and a mask decoder. Compared with the visual language model based on the visual transformer (ViT), the convolutional visual language model (based on the convolutional neural network (CNN)) has better generalization ability, especially when the input size is expanded. Therefore, the model based on the convolutional network performs well in the dense prediction tasks of different input sizes.

[0018] Turning now to the accompanying drawings, a single-stage framework for open-vocabulary panoptic segmentation is provided. Figure 1A schematic diagram of an example computing system 100 is shown that is configured to perform open vocabulary panoptic segmentation using a single-stage framework. The computing system 100 includes a computing device 102 that also includes a processor 104 (e.g., a central processing unit or "CPU"), a volatile memory 106, a non-volatile memory 108, and an I / O module 110 that are operably coupled to each other. The non-volatile memory 108 stores an open vocabulary panoptic segmentation program 112 that contains instructions for various software modules described herein for execution by the processor 104.

[0019] When executed by the processor 104, the instructions stored in the open vocabulary panoptic segmentation program 112 cause the processor 104 to receive an image 114. The image 114 may be received in a variety of ways. For example, a user may provide and / or upload the image 114 to the computing device 102. The image 114 may be received in a variety of file formats. In some implementations, the open vocabulary panoptic segmentation program 112 is configured with the ability to perform panoptic segmentation on a video file. In such a case, the video data may be split into multiple images, and the open vocabulary panoptic segmentation program 112 performs panoptic segmentation on each image separately.

[0020] Figure 1 The open vocabulary panoramic segmentation program 112 depicted in is configured to perform panoramic segmentation using a single-stage framework. The framework relies on a CNN-based visual language model backbone 116, which provides shared features for various components within the framework. Various visual language models can be used. In some implementations, the framework uses a CNN-based CLIP model. In order to keep the pre-trained image-text feature alignment intact, the CNN-based visual language model backbone 116 can be configured to be non-trainable (i.e., frozen) to prevent fine-tuning that may destroy the pre-trained image-text feature alignment.

[0021] The single-stage framework includes extracting a feature map 118 from an image 114 using a CNN-based visual language model backbone 116. In some implementations, the CNN-based visual language model backbone 116 is configured to receive an image of a predetermined resolution as input. For example, the CNN-based visual language model backbone 116 can be trained on a dataset containing images of a specific resolution, and therefore, performs better when receiving similar images as input. Since the received image can vary in resolution, the framework can include a resizing process for resizing the received image 114 to a predetermined resolution. Resizing can be performed in a variety of ways. For example, dimensions that are too large can be cropped, and dimensions that are too small can also be filled (e.g., using black pixels). The predetermined resolution can be any resolution. In some implementations, the CNN-based visual language model backbone 116 is configured to receive a high-resolution image (e.g., a resolution of at least 1024×1024 pixels) as input. As used throughout this disclosure, "high resolution" and "low resolution" are used as relative descriptors and can include any resolution.

[0022] The extracted feature map 118 is fed into a pixel decoder 120, thereby producing pixel features 122, which can then be fed into a mask decoder 124 to provide mask predictions 126 (which may also be referred to as mask proposals). The mask predictions 126 provide instance segmentation, which identifies and describes the objects detected in the image 114. The framework also includes in-vocabulary classifiers and out-of-vocabulary classifiers to provide class predictions for the mask predictions 126.

[0023] The class prediction process includes performing an in-vocabulary mask classification for each of the mask predictions 126 using a mask pooling module 128. The mask pooling module 128 performs mask pooling on the pixel features 122 provided by the pixel decoder 120 to generate an in-vocabulary class prediction 130, which corresponds to the mask prediction 126. In another branch, the class prediction process includes performing an out-of-vocabulary mask classification for each of the mask predictions 126 using the mask pooling module 128. For the out-of-vocabulary mask classification, the mask pooling module 128 performs mask pooling on the feature map 118 provided by the CNN-based visual language model backbone 116 to generate an out-of-vocabulary class prediction 132. The in-vocabulary class predictions 130 and the out-of-vocabulary class predictions 132 are then fused by the geometric integration module 134 to provide the final class prediction 136.

[0024] The output of the open vocabulary panoptic segmentation process may include mask predictions 126 and associated predicted classifications 136. Out-of-vocabulary classifiers enable the panoptic segmentation process to provide class predictions beyond the limited number of semantic classes learned from the annotated dataset. This open vocabulary approach allows the panoptic segmentation process to classify objects with a wider vocabulary, allowing for more general application in real word settings. Figure 1 A general single-stage framework for open-vocabulary panoptic segmentation is presented. More details about the individual components and modules within the framework are provided in the following sections.

[0025] Mathematically, open vocabulary panoptic segmentation can be described as a process that aims to transform an image Segment into a set of non-overlapping masks with associated semantic labels:

[0026] K ground truth masks m i ∈{0, 1} H×W do not overlap with each other (i.e., ), and c i Indicates mask m i During training, a fixed set of class labels C can be used train During inference, another set of labels C can be used test In the open vocabulary setting, C test Can include novel categories not seen during training (i.e., C train ≠C test ). It can be assumed that C test The category names (in natural language) are available during testing.

[0027] Perform open vocabulary panoptic segmentation to transform the image Segmentation into sets of non-overlapping masks with associated semantic labels can be achieved in a number of ways. As mentioned above, the single-stage framework has several advantages over the two-stage framework. FIG. 2A to FIG. 2C A comparison between various open-vocabulary panoptic segmentation pipelines including a single-stage framework and a two-stage framework is shown. Figure 2A An example two-stage pipeline 200 for open vocabulary panoptic segmentation is shown. In the depicted example, the first stage takes a high-resolution image 202 to generate a class-independent mask 204 using a trainable model backbone 206 and a mask generator 208. The second stage feeds a low-resolution image 210 and the predicted class-independent mask 204 (mask prediction input not shown) to a frozen CLIP backbone 212, which operates as an out-of-vocabulary classifier 214 for open vocabulary recognition, thereby producing class predictions 216. Because image features need to be extracted multiple times, this generates a large amount of computation.

[0028] Mathematically, the two-stage framework pipeline can be described as comprising a first stage containing Class-independent mask generator for Mask Generator Generate a set of N mask proposals: Given an input image I:

[0029]

[0030] At the second level, the CLIP adapter The image I and the mask proposal Both are used as input, where the latter input is used to guide the frozen CLIP model CLIP* (* indicates frozen). The adapter performs mask classification using either masked cropping or masked attention via a forward pass:

[0031]

[0032] in Refers to the predicted class probability for the predicted N masks, C∈{C train , C test} is related to the training or testing phase, and |C| is the class size.

[0033] The two-stage framework for open vocabulary panoptic segmentation has several disadvantages. First, image features are extracted twice, once for mask generation and another for mask classification. Multiple feature extractions incur a large amount of computation, making it costly to scale the backbone parameters. Second, the mask generator is typically implemented to handle high-resolution inputs (e.g., resolution of at least 1024×1024), while the CLIP model is typically pre-trained using low-resolution images (e.g., 224×224). The two-stage pipeline therefore feeds high-resolution images into the mask generator and low-resolution images into the CLIP classifier, making the model inefficient.

[0034] To avoid increasing the model size and the computational cost of repeated feature extraction, the naive single-stage framework This can be achieved by formulating everything. Figure 2BA diagram of an example naive single-stage framework 220 for open vocabulary panoptic segmentation using a trainable CLIP backbone 222 is shown. The naive single-stage framework 220 builds a classifier 224 and a mask generator 226 together and fine-tunes the CLIP backbone 222. The framework 220 processes a high-resolution image 228 to produce mask predictions 230, which are then used with the classifier 224 (mask prediction input not shown) to generate associated class predictions 232. In such a framework, the classifier 224 and the mask generator 226 share a pre-trained CLIP backbone 222 (not frozen) for extracting features from an input image I:

[0035]

[0036] However, fine-tuning such a single-stage framework 220 may result in misalignment between image and text features in the pre-trained CLIP model 222, especially for unseen novel categories, and may lead to suboptimal performance. In addition, the training cost is significant.

[0037] Although the frozen CLIP backbone can provide sufficient features for mask generation while preserving image-text alignment representations, CLIP models are usually pre-trained on low-resolution images (e.g., 224×224), while segmentation models prefer higher-resolution inputs (e.g., 800×1333, 1024×2048, etc.). This difference can lead to significant performance degradation (e.g., when applying frozen CLIP on large input images). For example, CLIP models using the ViT backbone do not transfer well to different input sizes. Although these issues can be mitigated by particularly careful design (e.g., side adapters or cost aggregation), performance degradation is still a problem.

[0038] Compared to the ViT-based CLIP model, the CNN-based CLIP model shows better generalization ability to different input sizes due to its fully convolutional nature. Therefore, the single-stage open-vocabulary panoptic segmentation framework can use a single frozen convolutional CLIP (FC-CLIP) backbone. To achieve:

[0039]

[0040] Figure 2CA diagram of an example single-stage framework 240 for open vocabulary panoptic segmentation using a FC-CLIP model backbone 242 is shown. The "frozen CLIP" maintains open vocabulary recognition and can act as a strong mask generator, and the "convolutional CLIP" can generalize well to large input sizes. The single-stage framework 240 employs a shared FC-CLIP 242 that processes a high-resolution image 244 using three components: a class-agnostic mask generator 246 for generating mask predictions 248, an in-vocabulary classifier 250 for generating in-vocabulary class predictions 252 for the mask predictions 248 (mask prediction input not shown), and an out-of-vocabulary classifier 254 for generating out-of-vocabulary class predictions 256 for the mask predictions 248 (mask prediction input not shown).

[0041] FC-CLIP can be implemented to leverage the semantic features of a frozen CNN-based CLIP backbone for both mask generation and CLIP classification. Unlike previous approaches, which typically train separate mask generators and ignore potential reuse of CLIP semantic features, freezing and sharing backbone features enables more efficient training and testing by avoiding feature duplication. The CNN-based CLIP backbone not only transfers well to different input resolutions (according to the image sizes it was pre-trained on), but also generates multi-scale feature maps. Although FIG. 2A to FIG. 2C The example described in illustrates a framework utilizing the CLIP model, but other CNN-based visual-language models can also be implemented.

[0042] Figure 3 A diagram depicting the process flow of an example single-stage framework 300 for open vocabulary panoptic segmentation is shown. The example single-stage framework 300 may be based on Figure 2C The framework 240 described in . In the described example, the single-stage framework 300 is implemented using a shared frozen convolutional CLIP backbone 302. Other visual language models can also be used. The open vocabulary panoptic segmentation process is performed using three main components built on the shared FC-CLIP backbone 302. Figure 2C Similar to the framework 240 of , the three main components include a mask generator, an in-vocabulary classifier, and an out-of-vocabulary classifier.

[0043] The process flow begins with an input image 304. In the described example, the input image 304 is a high resolution image having a resolution of at least 1024×1024 pixels. The single-stage framework 300 can be configured to process images of any resolution. Higher resolution images enable more accurate mask prediction. In some implementations, the framework 300 includes resizing the input image 304 to a predetermined resolution (such as the resolution that the shared FC-CLIP backbone 302 is configured to receive) before performing panoptic segmentation. The resizing process can be performed in a variety of ways, including image cropping and padding.

[0044] The shared FC-CLIP backbone 302 processes the input image 304 to extract a feature map 306. A pixel decoder 308 is then used to generate pixel features 310 from the feature map 306. In some implementations, the pixel decoder 308 is enhanced with axial attention to improve the features extracted from the FC-CLIP backbone 302. The pixel features 310, along with a set of object queries (not shown), can be passed through a mask decoder 312 to generate class-independent mask predictions 314. In some implementations, matrix multiplication is performed between the object query and the pixel features 310 to generate the mask predictions 314. For the mask predictions 314, a mask merging scheme can also be implemented. The mask decoder 312 can be implemented in a variety of ways. In some implementations, the mask decoder 312 includes k-means cross attention, self-attention, and a feed-forward network. The mask decoder 312 can be implemented as a series of mask decoders implemented to generate class-independent mask predictions 314. For example, mask decoder 312 may be implemented as a series of six mask decoders.

[0045] During training, the predicted masks 314 are matched to the ground truth masks in a one-to-one manner through Hungarian matching and supervised accordingly. Since the number of object queries is usually larger than the number of labeled masks, usually only a subset of the predicted masks are optimized through this matching process. No penalty is applied to the remaining unmatched proposals, which provides more mask proposals to be obtained.

[0046] Once the mask predictions 314 are generated, the intra-vocabulary classifier produces a class embedding 316 by mask pooling 318 the pixel features 310 from the pixel decoder 308. Intra-vocabulary classification of the mask predictions 314 can be performed in a contrastive manner using the class text embeddings 320, where the class embeddings 316 and the class text embeddings 320 for each mask prediction 314 are projected into a common embedding space. In some implementations, the classification log (before performing SoftMax) is obtained by matrix multiplication between the predicted class embeddings 316 and the class text embeddings 320. The text embeddings 320 can be obtained by feeding the class names to the CLIP text encoder, which can be executed in advance and cached in memory as a text classifier, resulting in negligible additional cost.

[0047] The class probabilities predicted by the in-vocabulary classifier can be defined as follows:

[0048] where T is a learnable temperature parameter initialized to 0.07 to control the sharpness of the distribution, cos is the cosine distance measure, and v i is the class embedding for the predicted i-th mask obtained by mask pooling 318 on the final pixel features 310 from the pixel decoder 308. j is the text embedding 320 of the category name of class j, which can be obtained by feeding the category name to a text encoder pre-trained by CLIP.

[0049] During inference, the in-vocabulary classifier itself cannot generalize to completely novel unseen classes because the model 302 is trained on a limited set of categories and therefore cannot recognize different novel concepts. To address this problem, the out-of-vocabulary classifier provides out-of-vocabulary classification of mask prediction 314. The out-of-vocabulary classifier generates class embedding 322 by masking pooling 318 of frozen CLIP backbone features 306, aiming to borrow the pre-trained (complete) open vocabulary recognition ability of CLIP. Similar to the in-vocabulary classification, the out-of-vocabulary classification of mask prediction 314 can be performed in a contrasting manner using the category text embedding 320.

[0050] Unlike the two-stage framework that requires one or more forward passes of CLIP, the adopted out-of-vocabulary classifier introduces a small additional cost because the backbone features 306 have already been extracted (and only lightweight mask pooling 318 is performed). Then the out-of-vocabulary classifier The predicted class probabilities are compared with the in-vocabulary classifier In a similar way, v iThe features are replaced with masked pooled features on the frozen CLIP backbone features 306. This classifier maintains the original CLIP feature distribution, allowing for better recognition of new categories.

[0051] The framework 300 includes using geometric ensemble 324 to fuse classification scores between in-vocabulary and out-of-vocabulary classifiers to generate a final class prediction 326 for mask prediction 314. That is,

[0052]

[0053] Where the subscripts in and out refer to the in-vocabulary and out-of-vocabulary classifiers, respectively, and α, β∈[0,1] balance the predictions of the in-vocabulary and out-of-vocabulary classifiers for seen and novel unseen categories. In some implementations, the classification process is initialized with α=0.4 and β=0.8. The mask prediction 314 and the final class prediction 326 provide an output 328 for an open vocabulary panoptic segmentation of the input image 304.

[0054] Figures 4 to 9 An example open vocabulary panoptic segmentation framework (such as Figure 3 ) performance results for the framework described in . Figure 4 is a table comparing the panoramic quality (PQ) and mean intersection over union (mIoU) of the example FC-CLIP single-stage framework and other known methods. The performance results are based on zero-shot learning of open vocabulary panoramic segmentation on the ADE20K dataset. All methods are trained on the COCO dataset. As shown, the example FC-CLIP framework achieves significantly better performance compared to MaskCLIP (a transformer-based method using a visual encoder). With +9.4PQ and +7.2mIoU, the example FC-CLIP outperforms MaskCLIP despite using fewer frozen parameters (-104M) and trainable parameters (-29M). The advantage of FC-CLIP remains when compared to the concurrent methods FreeSeg (a general framework for unified, universal, and open vocabulary image segmentation) and ODISE (a diffusion-based model). Under the same training setting (without using COCO-Stuff annotations), the example FC-CLIP is +8.2PQ better than FreeSeg and +1.9PQ better than ODISE. Compared to ODISE with supervised captions, Example FC-CLIP still outperforms it by +1.1PQ. Moreover, Example FC-CLIP achieves a significant reduction of 7.5× (6.5×) in frozen (total) parameters compared to ODISE, which leverages a powerful large backbone from stable diffusion for feature extraction.

[0055] Figure 5is a table comparing PQ, segmentation quality (SQ), and recognition quality (RQ) of the example FC-CLIP single-stage framework and ODISE. The performance results are based on zero-shot learning open vocabulary panoptic segmentation on the Cityscapes and Mapillary Vistas datasets, which focus on street driving scenes. Compared to ODISE, the example FC-CLIP achieves better performance on both datasets, outperforming ODISE by +2.8PQ and +19.1PQ on Mapillary Vistas and Cityscapes, respectively. In the described example, the SQ for FC-CLIP is lower because the mask generator is weaker compared to ODISE, which uses a larger backbone and larger input size (e.g., on Cityscapes, ODISE resizes the shorter edges to 1024, which results in a 1024×2048 input resolution, while the example FC-CLIP resizes the longer edges to 1281, which results in an effective input size of 641×1281).

[0056] Figure 6 is a table comparing the mIoU across various datasets for an example FC-CLIP single-stage framework and other known methods. Using the same training annotations, FC-CLIP outperforms MaskCLIP by +3.2, +5.0, +7.2, +11.5 mIoU across datasets A-847, PC-459, A-150, and PC-59, respectively. Compared to methods with title annotations, FC-CLIP maintains its advantage, outperforming ODISE (title) by +0.4, +1.2, +2.2, +2.1, +2.0 mIoU on datasets A-847, PC-459, A-150, PC-59, and PAS-21, respectively.

[0057] Figure 7 is a table comparing the inference speed of the example FC-CLIP single-stage framework and ODISE. The performance results are shown in frames per second (FPS). ODISE uses a larger backbone than the example FC-CLIP. It also uses a larger input size, resizing the shorter edge size to 1024, which results in a typical size of 1024×1706 on COCO or 1024×2048 on Cityscapes. Nevertheless, despite these shortcomings, the example FC-CLIP provides significantly faster inference time and better performance (on the ADE20K and COCO datasets, the example FC-CLIP runs 4.44× and 5.56× faster than ODISE, respectively).

[0058] Figure 8is a table comparing the PQ, SQ, and RQ of the example FC-CLIP single-stage framework and other known methods. The performance results are based on a model trained on the ADE20K dataset using panoramic annotations and evaluated on the COCO panoramic dataset. As shown in the figure, on the COCO dataset, the example FC-CLIP performs better than FreeSeg by +10.2PQ and better than ODISE by +1.7PQ.

[0059] Fig. 9 It is a chart comparing the RQ performance of trainable CLIP variants and frozen CLIP variants. The same mask proposal is used for two models. The performance results for 10 seen classes (left side) and 10 unseen classes (right side) are shown. The 10 seen classes are shared by both COCO and ADE20K data sets, and the 10 unseen classes are only included in the ADE20K data set. As shown in the figure, adjusting the CLIP backbone causes similar performance for seen classes, and worse performance on unseen concepts, because the adjustment destroys the CLIP feature alignment, thereby losing its recognition ability on a wider vocabulary.

[0060] Fig.10 A flow chart of an example method 1000 for performing open vocabulary panoptic segmentation using a single-stage framework is shown. At step 1002, method 1000 includes receiving an image. The image can be received in various formats and in various ways, such as uploaded from a user. In some implementations, media content is received and the image is retrieved from the media content. For example, some implementations can be configured to perform panoptic segmentation on a video file. In such a case, frames of the video file can be processed individually as images.

[0061] At step 1004, method 1000 includes extracting multiple feature maps from the image using a CNN-based visual language model. Various visual language models can be used. For example, the visual language model can be implemented as a CLIP model. In some implementations, the CNN-based visual language model is a non-trainable (i.e., frozen) model. The CNN-based visual language model can be configured to process images of a predetermined resolution. Typically, the predetermined resolution is a high resolution (e.g., a resolution of at least 1024×1024 pixels). In some implementations, the image is resized to a predetermined resolution. Resizing can be performed in a variety of ways. For example, image cropping and / or padding can be performed to adjust the image size.

[0062] At step 1006, method 1000 includes using a pixel decoder to generate a plurality of pixel features from a plurality of feature maps. Various types of pixel decoders may be used. For example, a lightweight pixel decoder may be used to improve efficiency. In some implementations, the pixel decoder is enhanced using axial attention, which may improve the feature maps extracted from the CNN-based visual language model.

[0063] At step 1008, method 1000 includes generating multiple mask predictions from multiple pixel features. Mask predictions can be generated in a variety of ways. In some implementations, mask predictions are generated by feeding multiple pixel features and object query sets into one or more mask decoders. In other implementations, matrix multiplication is performed between the object query and the pixel features to generate the mask predictions. Different mask decoders can be used. The mask decoder can be implemented as a series of mask decoders. In some implementations, each mask decoder includes k-means cross attention, self-attention, and a feedforward network.

[0064] At step 1010, method 1000 includes using multiple pixel features to generate multiple intra-lexical class predictions corresponding to multiple mask predictions. Generating multiple intra-lexical class predictions may include mask pooling the multiple pixel features to generate multiple intra-lexical class embeddings for each mask prediction in the mask prediction. The intra-lexical class embeddings may then be used in a contrasting manner with multiple text embeddings to generate multiple intra-lexical class predictions. The text embeddings may be generated in a variety of ways. In some implementations, the text embeddings are generated by feeding the category names into a pre-trained text encoder (such as a CLIP text encoder). The text embeddings may be generated in advance and cached in a memory.

[0065] At step 1012, method 1000 includes generating a plurality of out-of-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of feature maps. Generating the plurality of out-of-vocabulary class predictions may include mask pooling the plurality of feature maps to generate a plurality of in-vocabulary class embeddings for each of the mask predictions. The out-of-vocabulary class embeddings may then be used in a contrastive manner with the plurality of text embeddings to generate the plurality of out-of-vocabulary class predictions.

[0066] At step 1014, method 1000 includes performing geometric integration of the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions to generate a plurality of final class predictions. In some implementations, the geometric integration includes fusing the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions using a weighted formula. In further implementations, the out-of-vocabulary class predictions are weighted higher than the in-vocabulary class predictions.

[0067] At step 1016, method 1000 includes outputting a plurality of mask predictions and a plurality of final class predictions. Such information provides a panoptic segmentation result that identifies instances of objects and predicted classifications of the instances.

[0068] The single-stage frozen convolutional visual language model framework described in this paper provides a simple yet effective framework for open vocabulary panoptic segmentation. For example, FC-CLIP improves efficiency by building everything on a shared frozen convolutional CLIP backbone, which not only significantly reduces training and testing costs, but also performs well against other known methods for open vocabulary panoptic segmentation.

[0069] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.

[0070] Fig.11 A non-limiting embodiment of a computing system 1100 is schematically shown, and the computing system 1100 can implement one or more of the above methods and processes. The computing system 1100 is shown in simplified form. The computing system 1100 can embody the above description and in Figure 1 Components of computing system 1100 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices and wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0071] The computing system 1100 includes a logic processor 1102, a volatile memory 1104, and a non-volatile storage device 1106. The computing system 1100 may optionally include a display subsystem 1108, an input subsystem 1110, a communication subsystem 1112, and / or Fig.11 Other components not shown.

[0072] Logical processor 1102 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0073] The logical processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices, which are configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 1102 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logical processor may optionally be distributed between two or more individual devices, which may be remotely located and / or configured to coordinate processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessed networked computing device configured in a cloud computing configuration. In such a case, it will be understood that these virtualized aspects are run on different physical logical processors of various different machines.

[0074] The non-volatile storage device 1106 includes one or more physical devices that are configured to store instructions that can be executed by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 1106 can be transformed to, for example, store different data.

[0075] The non-volatile storage device 1106 may include a removable and / or built-in physical device. The non-volatile storage device 1106 may include an optical memory, a semiconductor memory, and / or a magnetic memory or other mass storage device technology. The non-volatile storage device 1106 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable and / or content addressable device. It is understood that the non-volatile storage device 1106 is configured to save instructions even when the power to the non-volatile storage device 1106 is cut off.

[0076] The volatile memory 1104 may include a physical device with random access memory. The volatile memory 1104 is typically used by the logical processor 1102 to temporarily store information during processing of software instructions. It will be appreciated that when the power to the volatile memory 1104 is cut off, the volatile memory 1104 typically does not continue to store instructions.

[0077] Aspects of the logic processor 1102, volatile memory 1104, and non-volatile storage device 1106 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0078] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 1100, which is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a transformation process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 1102 that uses portions of volatile memory 1104 to execute instructions stored by a non-volatile storage device 1106. It is understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may include individual executable files, data files, libraries, drivers, scripts, database records, etc. or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0079] When included, the display subsystem 1108 can be used to present a visual representation of the data stored by the non-volatile storage device 1106. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device, thereby transforming the state of the non-volatile storage device, the state of the display subsystem 1108 can also be transformed to intuitively represent the changes in the underlying data. The display subsystem 1108 may include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 1102, the volatile memory 1104, and / or the non-volatile storage device 1106 in a shared housing, or such a display device can be a peripheral display device.

[0080] When included, input subsystem 1110 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, camera, or microphone.

[0081] When included, the communication subsystem 1112 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 1112 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wired or wireless local or wide area network, a broadband cellular network, or the like. In some embodiments, the communication subsystem can allow the computing system 1100 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0082] As used herein, "and / or" is defined as an inclusive OR as specified by the following truth table:

[0083] A B A∨B real real real real Fake real Fake real real Fake Fake Fake

[0084] The following paragraphs provide additional descriptions of the subject matter of the present disclosure. One aspect provides a computing system for performing open vocabulary panoramic segmentation, the computing system comprising: a processor and a memory, the memory storing instructions, the instructions, when executed by the processor, causing the processor to: receive an image; extract multiple feature maps from the image using a convolutional neural network (CNN) based visual language model; generate multiple pixel features from the multiple feature maps using a pixel decoder; generate multiple mask predictions from the multiple pixel features using a mask decoder; generate multiple in-vocabulary class predictions corresponding to the multiple mask predictions using the multiple pixel features; generate multiple out-of-vocabulary class predictions corresponding to the multiple mask predictions using the multiple feature maps; perform geometric integration on the multiple in-vocabulary class predictions and the multiple out-of-vocabulary class predictions to generate multiple final class predictions; and output the multiple mask predictions and the multiple final class predictions. In this aspect, additionally or alternatively, the CNN based visual language model includes a non-trainable model. In this aspect, additionally or alternatively, the CNN based visual language model includes a contrastive language image pre-trained model. In this aspect, additionally or alternatively, extracting multiple feature maps from an image using a CNN-based visual language model includes: resizing the image to a predetermined resolution of at least 1024×1024 pixels; and feeding the resized image into a CNN-based visual language model to generate multiple feature maps. In this aspect, additionally or alternatively, the image is at least 1024×1024 pixels. In this aspect, additionally or alternatively, generating multiple mask predictions includes feeding multiple pixel features and an object query set into a mask decoder. In this aspect, additionally or alternatively, generating multiple intra-lexical class predictions includes: mask pooling the multiple pixel features to generate multiple intra-lexical class embeddings for each mask prediction in the mask prediction; and generating multiple intra-lexical class predictions using the multiple intra-lexical class embeddings and the multiple text embeddings. In this aspect, additionally or alternatively, the multiple text embeddings are generated by feeding the category names into a pre-trained text encoder. In this aspect, additionally or alternatively, generating multiple out-of-vocabulary predictions includes: mask pooling the multiple feature maps to generate multiple out-of-vocabulary embeddings for each mask prediction in the mask predictions; and generating multiple out-of-vocabulary predictions using the multiple out-of-vocabulary embeddings and the multiple text embeddings. In this aspect, additionally or alternatively, performing geometric integration includes fusing the multiple in-vocabulary predictions and the multiple out-of-vocabulary predictions using a weighted formula.

[0085] Another aspect provides a method for performing open vocabulary panoramic segmentation, the method comprising: receiving an image; extracting a plurality of feature maps from the image using a convolutional neural network (CNN)-based visual language model; generating a plurality of pixel features from the plurality of feature maps using a pixel decoder; generating a plurality of mask predictions from the plurality of pixel features using a mask decoder; generating a plurality of in-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of pixel features; generating a plurality of out-of-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of feature maps; performing geometric integration of the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions to generate a plurality of final class predictions; and outputting the plurality of mask predictions and the plurality of final class predictions. In this aspect, additionally or alternatively, the CNN-based visual language model further comprises a non-trainable contrastive language image pre-trained model. In this aspect, additionally or alternatively, extracting a plurality of feature maps from the image using the CNN-based visual language model comprises: resizing the image to a predetermined resolution of at least 1024×1024 pixels; and feeding the resized image into the CNN-based visual language model to generate a plurality of feature maps. In this aspect, additionally or alternatively, generating multiple mask predictions includes feeding multiple pixel features and an object query set into a mask decoder. In this aspect, additionally or alternatively, generating multiple in-vocabulary class predictions includes: mask pooling multiple pixel features to generate multiple in-vocabulary class embeddings for each mask prediction in the mask prediction; and generating multiple in-vocabulary class predictions using multiple in-vocabulary class embeddings and multiple text embeddings. In this aspect, additionally or alternatively, multiple text embeddings are generated by feeding category names into a pre-trained text encoder. In this aspect, additionally or alternatively, generating multiple out-of-vocabulary class predictions includes: mask pooling multiple feature maps to generate multiple out-of-vocabulary class embeddings for each mask prediction in the mask prediction; and generating multiple out-of-vocabulary class predictions using multiple out-of-vocabulary class embeddings and multiple text embeddings. In this aspect, additionally or alternatively, performing geometric integration includes fusing multiple in-vocabulary class predictions and multiple out-of-vocabulary class predictions using a weighted method.

[0086] Another aspect provides a non-transitory computer-readable medium for media content editing, wherein the non-transitory computer-readable medium includes instructions, and when the instructions are executed by a computing device, the computing device implements the method described in the above aspects.

[0087] On the other hand, a computing system for performing open vocabulary panoramic segmentation is provided, the computing system including: a processor and a memory, the memory storing instructions, which, when executed by the processor, causes the processor to: receive media content including at least one image; and for each image in the media content, perform panoramic segmentation in the following manner: extract multiple feature maps from the image using a non-trainable convolutional neural network (CNN-based) contrastive language image pre-trained model; generate multiple pixel features from the multiple feature maps using a pixel decoder; generate multiple mask predictions from the multiple pixel features using a mask decoder; and for each mask prediction, generate a final class prediction in the following manner: generate an in-vocabulary class prediction by mask pooling the multiple pixel features; generate an out-of-vocabulary class prediction by mask pooling the multiple pixel features and multiple feature maps; and generate a final class prediction using the in-vocabulary class prediction and the out-of-vocabulary class prediction.

[0088] It is understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered in a limiting sense, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various behaviors illustrated and / or described may be performed in the illustrated and / or described sequence, in other sequences, in parallel, or omitted. Likewise, the order of the above process may be changed.

[0089] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system for performing open vocabulary panoptic segmentation, the computing system comprising: A processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to: receiving an image; Extracting a plurality of feature maps from the image using a visual language model based on a convolutional neural network (CNN); Using a pixel decoder, generating a plurality of pixel features from the plurality of feature maps; Using a mask decoder, generating a plurality of mask predictions from the plurality of pixel features; generating a plurality of intra-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of pixel features; generating a plurality of out-of-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of feature maps; performing geometric integration of the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions to generate a plurality of final class predictions; as well as The plurality of mask predictions and the plurality of final class predictions are output.

2. The computing system of claim 1 , wherein the CNN-based visual-language model comprises a non-trainable model.

3. The computing system of claim 1, wherein the CNN-based visual language model comprises a contrastive language image pre-trained model.

4. The computing system of claim 1 , wherein extracting the plurality of feature maps from the image using the CNN-based visual language model comprises: resizing the image to a predetermined resolution of at least 1024×1024 pixels; as well as The resized image is fed into the CNN-based visual language model to generate the plurality of feature maps.

5. The computing system of claim 1, wherein the image is at least 1024 x 1024 pixels.

6. The computing system of claim 1, wherein generating the plurality of mask predictions comprises feeding the plurality of pixel features and an object query set into the mask decoder.

7. The computing system of claim 1 , wherein generating the plurality of intra-vocabulary class predictions comprises: performing mask pooling on the plurality of pixel features to generate a plurality of intra-vocabulary class embeddings for each of the mask predictions; as well as The plurality of within-vocabulary class predictions are generated using the plurality of within-vocabulary class embeddings and the plurality of text embeddings.

8. The computing system of claim 7, wherein the plurality of text embeddings are generated by feeding category names into a pre-trained text encoder.

9. The computing system of claim 1 , wherein generating the plurality of out-of-vocabulary class predictions comprises: performing mask pooling on the plurality of feature maps to generate a plurality of out-of-vocabulary class embeddings for each of the mask predictions; as well as The plurality of out-of-vocabulary class predictions are generated using the plurality of out-of-vocabulary class embeddings and the plurality of text embeddings.

10. The computing system of claim 1, wherein performing the geometric integration comprises fusing the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions using a weighted formula.

11. A method for performing open vocabulary panoptic segmentation, the method comprising: receiving an image; Extracting a plurality of feature maps from the image using a visual language model based on a convolutional neural network (CNN); Using a pixel decoder, generating a plurality of pixel features from the plurality of feature maps; Using a mask decoder, generating a plurality of mask predictions from the plurality of pixel features; generating a plurality of intra-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of pixel features; generating a plurality of out-of-vocabulary class predictions corresponding to the plurality of mask predictions using the plurality of feature maps; performing geometric integration of the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions to generate a plurality of final class predictions; as well as The plurality of mask predictions and the plurality of final class predictions are output.

12. The method of claim 11, wherein the CNN-based visual language model comprises a non-trainable contrastive language image pre-trained model.

13. The method of claim 11, wherein extracting the plurality of feature maps from the image using the CNN-based visual language model comprises: resizing the image to a predetermined resolution of at least 1024×1024 pixels; as well as The resized image is fed into the CNN-based visual language model to generate the plurality of feature maps.

14. The method of claim 11, wherein generating the plurality of mask predictions comprises feeding the plurality of pixel features and an object query set into the mask decoder.

15. The method of claim 11, wherein generating the plurality of intra-vocabulary class predictions comprises: performing mask pooling on the plurality of pixel features to generate a plurality of intra-vocabulary class embeddings for each of the mask predictions; as well as The plurality of within-vocabulary class predictions are generated using the plurality of within-vocabulary class embeddings and the plurality of text embeddings.

16. The method of claim 15, wherein the plurality of text embeddings are generated by feeding category names into a pre-trained text encoder.

17. The method of claim 11, wherein generating the plurality of out-of-vocabulary class predictions comprises: performing mask pooling on the plurality of feature maps to generate a plurality of out-of-vocabulary class embeddings for each of the mask predictions; as well as The plurality of out-of-vocabulary class predictions are generated using the plurality of out-of-vocabulary class embeddings and the plurality of text embeddings.

18. The method of claim 11, wherein performing the geometric integration comprises fusing the plurality of in-vocabulary class predictions and the plurality of out-of-vocabulary class predictions using a weighted approach.

19. A non-transitory computer-readable medium for media content editing, the non-transitory computer-readable medium comprising instructions which, when executed by a computing device, cause the computing device to implement the method of claim 11.

20. A computing system for performing open vocabulary panoptic segmentation, the computing system comprising: A processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to: receiving media content including at least one image; as well as For each image in the media content, panoptic segmentation is performed in the following manner: Extracting a plurality of feature maps from the image using a non-trainable convolutional neural network (CNN) based contrastive language image pre-trained model; Using a pixel decoder, generating a plurality of pixel features from the plurality of feature maps; Using a mask decoder, generating a plurality of mask predictions from the plurality of pixel features; as well as For each mask prediction, the final class prediction is generated as follows: generating intra-vocabulary class predictions by mask pooling the plurality of pixel features; generating an out-of-vocabulary class prediction by mask pooling the plurality of pixel features and the plurality of feature maps; as well as Using the in-vocabulary class predictions and the out-of-vocabulary class predictions, the final class prediction is generated.

Images