Panoptic segmentation using multi-dataset training and part-to-whole recognition

Abstract

A system and method are provided for multi-dataset panoptic segmentation, which includes processing images received from multiple datasets, each containing its own unique label space (702), extracting multiscale features using a backbone network (704), generating text embeddings of class names from the unique label space for each of the multiple datasets (706), and integrating the visual features extracted from the received images with the text embeddings (708) to generate a unified semantic space. A transformer-based segmentation model is trained using the unified semantic space (710), predicts the segmentation masks and classes of the received images, and performs inference using a panoptic interference algorithm to generate a unified panoptic segmentation map from the predicted segmentation masks and classes (712).

Description

Technical Field

[0001] Related Application Information This application claims priority to U.S. Provisional Application No. 63 / 465,627, filed May 11, 2023; U.S. Provisional Application No. 63 / 466,831, filed May 16, 2023; U.S. Provisional Application No. 63 / 599,175, filed Nov. 15, 2023; and U.S. Patent Application No. 18 / 659,785, filed May 9, 2024, the entire contents of each of which are hereby incorporated by reference in their entirety.

Background Art

[0002] The present invention relates to image processing using panoptic segmentation, and more particularly to a system and method for improving the accuracy of multi-dataset segmentation by integrating language-based embedding and dataset-specific adaptation into a transformer-based model. Description of Related Technologies

[0003] In the field of computer vision, particularly panoptic segmentation, traditional methods have focused on single-dataset training approaches that leverage the distinct visual features and semantic annotations specific to a given dataset. While these traditional systems are effective within a specific label space, they falter when faced with the task of integrating and interpreting semantic information across multiple datasets with diverse and potentially conflicting annotations. The challenge is further exacerbated by the fact that these methods struggle to handle the label space conflicts and overlaps that naturally occur when training across diverse datasets. This limitation significantly restricts their usability in real-world applications where the ability to operate across heterogeneous data sources is critical. Furthermore, training on relatively large, manually annotated datasets is not only labor-intensive but also limits the scalability and adaptability of segmentation models, especially in scenarios requiring rapid, real-time deployment across various operational environments. Therefore, there is a need for advanced segmentation techniques that can robustly integrate and analyze panoptic data from multiple datasets, improving both the accuracy and applicability of segmentation results in diverse and dynamic environments. [Overview of the project]

[0004] According to one aspect of the present invention, a method for multi-dataset panoptic segmentation is provided, comprising: processing images received from a plurality of datasets, each containing its own unique label space, to extract multiscale features using a backbone network; generating text embeddings of class names from the unique label space for each of the plurality of datasets; and integrating the visual features extracted from the received images with the text embeddings to generate a unified semantic space. A converter-based segmentation model is trained using the unified semantic space to predict segmentation masks and classes of the received images, and inference is performed using a panoptic interference algorithm to generate a unified panoptic segmentation map from the predicted segmentation masks and classes.

[0005] According to another aspect of the present invention, a system for multi-dataset panoptic segmentation is provided. When executed by a processor device, the system includes a memory that stores instructions for the system to process images received from multiple datasets, each having its own unique label space; to extract multiscale features using a backbone network; to generate text embeddings of class names from the unique label space for each of the multiple datasets; and to integrate the visual features extracted from the received images with the text embeddings to generate a unified semantic space. A converter-based segmentation model is trained using the unified semantic space to predict segmentation masks and classes of the received images and to perform inference using a panoptic interference algorithm to generate a unified panoptic segmentation map from the predicted segmentation masks and classes.

[0006] According to another aspect of the present invention, a computer program product for multi-dataset panoptic segmentation is provided, which includes instructions for processing images received from multiple datasets, each containing its own unique label space, extracting multiscale features using a backbone network, generating text embeddings of class names from the unique label space for each of the multiple datasets, and integrating the visual features extracted from the received images with the text embeddings to generate a unified semantic space. A converter-based segmentation model is trained using the unified semantic space to predict segmentation masks and classes of the received images, and inference is performed using a panoptic interference algorithm to generate a unified panoptic segmentation map from the predicted segmentation masks and classes.

[0007] These and other features and advantages will become apparent from the following detailed description of the exemplary embodiment, which will be read in conjunction with the attached drawings. [Brief explanation of the drawing]

[0008] This disclosure provides further details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] This block diagram illustrates an exemplary processing system to which the present invention may be applied, according to embodiments of the present invention.

[0010] [Figure 2] This figure illustrates a high-level diagram of a system and method for panoptic segmentation training using a multi-dataset partial-whole recognition model according to an embodiment of the present invention.

[0011] [Figure 3]This figure illustrates an exemplary multi-dataset partial-to-whole recognition pipeline for panoptic segmentation according to an embodiment of the present invention.

[0012] [Figure 4] This figure illustrates an exemplary high-level system and method for multi-dataset panoptic segmentation according to an embodiment of the present invention.

[0013] [Figure 5A] This figure illustrates an exemplary system and method for multi-dataset panoptic segmentation training according to an embodiment of the present invention.

[0014] [Figure 5B] This figure illustrates an exemplary system and method for multi-dataset panoptic segmentation inference according to an embodiment of the present invention.

[0015] [Figure 6] This figure illustrates an exemplary method for training multi-dataset panoptic image segmentation according to an embodiment of the present invention.

[0016] [Figure 7] This figure illustrates an exemplary method for multi-dataset panoptic segmentation according to an embodiment of the present invention.

[0017] [Figure 8] This figure illustrates a high-level diagram of a system and method for image processing based on multi-dataset panoptic segmentation across various computer vision and image processing tasks in multiple exemplary environments, according to embodiments of the present invention.

[0018] [Figure 9]FIG. is an exemplary diagram showing a high-level view of a system for image processing based on multi-dataset panoptic segmentation according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0019] According to an embodiment of the present invention, a system and method for enhancing a panoptic segmentation model are provided by integrating the learning functions of multiple datasets. This system and method can handle images from multiple datasets, each dataset being characterized by a unique and potentially conflicting label space, improving semantic understanding and model robustness. The core of this invention is to integrate an advanced transformer-based segmentation model with a new language-based embedding and a dataset-specific query embedding. This integration enables the system to integrate heterogeneous semantic categories into a coherent semantic space, significantly improving the accuracy and applicability of segmentation results across diverse datasets.

[0020] The capabilities of the present invention extend beyond conventional single-dataset segmentation methods, skillfully resolving label space contradictions and overlaps encountered during multi-dataset training. Through enhanced decoding processing, this model can dynamically adapt segmentation predictions based on the semantics of the specific dataset being used, ensuring a high fidelity of the generated panoptic map. In some embodiments, the present invention incorporates a new inference algorithm that optimizes the handling of overlapping segmentation masks, thereby improving the final segmentation output. This mechanism ensures that the system not only performs segmentation but also intelligently resolves conflicts between competing annotations. Additionally, this system is supported by a robust computational framework that manages the complex tasks of feature extraction, embedding integration, and panoptic map generation. The system can include multiple components such as a data preprocessing unit, a feature extraction module, and an inference engine, all of which are organized to utilize state-of-the-art AI technologies for managing and interpreting complex visual data across various panoptic segmentation environments according to aspects of the present invention.

[0021] The embodiments described herein may be wholly hardware, wholly software, or may include both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0022] Embodiments may include computer program products accessible from computer-enabled or computer-readable media that provide program code for use by or in connection with a computer or any instruction execution system. Computer-enabled or computer-readable media may include any device that stores, communicates, propagates, or transports programs for use by or in connection with an instruction execution system, apparatus, or device. The medium may be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor systems (or apparatus or devices), or propagation media. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random-access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0023] Each computer program can be substantially stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) that is readable by a general-purpose or special-purpose programmable computer, in order to configure and control the operation of the computer when the storage medium or device is read by the computer in order to perform the procedures described herein. The system of the present invention can also be considered to be implemented on a computer-readable storage medium configured with a computer program, in which case the configured storage medium causes the computer to operate in a specific predetermined manner to perform the functions described herein.

[0024] A data processing system suitable for storing and / or executing program code may include at least one processor directly or indirectly coupled to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory that provides at least some temporary storage for the program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediary I / O controller.

[0025] Network adapters can also be integrated into a system to enable a data processing system to connect to other data processing systems or remote printers or storage devices via an intervening private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0026] Aspects of the present invention will be described below with reference to flowcharts and / or block diagrams of methods, systems, and computer program products according to embodiments of the present invention. Note that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0027] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code consisting of one or more executable instructions for implementing a specified logical function, and in some alternative implementations of the present invention, the functions described in the blocks may occur in a different order than shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, in reverse order, or in any other order depending on the functionality of a particular embodiment.

[0028] Furthermore, it should be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in a block diagram and / or flowchart, can be implemented by a purpose-specific hardware system that performs a specific function / operation, or by a combination of purpose-specific hardware and computer instructions that conform to this principle.

[0029] Here, referring to drawings where similar numbers represent the same or similar elements, and first referring to Figure 1, an exemplary processing system 100 to which the present principle may be applied is illustrated exemplary according to an embodiment of the present principle.

[0030] In some embodiments, the processing system 100 may include at least one processor (CPU) 104 operably coupled with other components via a system bus 102. A cache 106, read-only memory (ROM) 108, random access memory (RAM) 110, input / output (I / O) adapters 120, audio adapters 130, network adapters 140, user interface adapters 150, and display adapters 160 are operably coupled to the system bus 102.

[0031] The first storage device 122 and the second storage device 124 are operably coupled to the system bus 102 by the I / O adapter 120. Storage devices 122 and 124 may be disk storage devices (e.g., magnetic or optical disk storage devices), solid-state magnetic devices, etc. Storage devices 122 and 124 may be of the same type or different types.

[0032] Speaker 132 is operably coupled to system bus 102 by audio adapter 130. Transceiver 142 is operably coupled to system bus 102 by network adapter 140. Display device 162 is operably coupled to system bus 102 by display adapter 160. Vision language (VL) models can be used in combination with prediction device 164 for input text processing tasks and can be further coupled to system bus 102 by any suitable connection system or method (e.g., Wi-Fi, wired, network adapter, etc.) according to aspects of the present invention.

[0033] The first user input device 152 and the second user input device 154 are operably coupled to the system bus 102 by the user interface adapter 150. The user input devices 152, 154 can be one or more of the following: a keyboard, mouse, keypad, image capture device, motion sensor, microphone, or a device incorporating at least two of the functions of the aforementioned devices. The encoder and / or decoder 156 can process the received input and, according to aspects of the present invention, can be included in a system with one or more storage devices, communication / network devices (e.g., WiFi, 4G, 5G, wired connection), hardware processors, etc. In various embodiments, other types of input devices can also be used while maintaining the spirit of the principles of the present invention. The user input devices 152, 154 can be the same type of user input device or different types of user input devices. The user input devices 152, 154 are used to input and output information to and from the system 100, according to aspects of the present invention. The encoder and / or decoder 156 may be used in conjunction with a trainable neural network to work with a training device / panoptic segmentation device 164 which can be operably connected to the system 100 for any of a plurality of tasks (e.g., image classification, object detection, segmentation, etc.) according to aspects of the present invention.

[0034] Of course, the processing system 100 may include other elements (not shown) and may omit certain elements, as can be readily contemplated by those skilled in the art. For example, various other input and / or output devices may be included in the processing system 100, depending on their specific implementation, as can be readily understood by those skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc., in various configurations may also be available, as can be readily understood by those skilled in the art. These and other variations of the processing system 100 will be readily contemplated by those skilled in the art, given the teachings of the present principle provided herein.

[0035] Furthermore, it should be understood that systems 200, 300, 400, 501, 503, 800, and 900, described later with respect to Figures 2, 3, 4, 5A, 5B, 8, and 9 respectively, are systems for carrying out each embodiment of the present invention. Part or all of the processing system 100 can be implemented in one or more elements of systems 200, 300, 400, 501, 503, 800, and 900 according to aspects of the present invention.

[0036] Furthermore, it should be understood that the processing system 100 can perform at least a portion of the methods described herein, including, for example, at least a portion of methods 200, 300, 400, 501, 503, 600, 700, and 800, which will be described later with respect to Figures 2, 3, 4, 5A, 5B, 6, 7, and 8, respectively. Similarly, some or all of the systems 200, 300, 400, 501, 503, 800, and 900 can be used, in accordance with aspects of the present invention, to perform at least a portion of methods 200, 300, 400, 501, 503, 600, 700, and 800, respectively.

[0037] As used herein, the terms “hardware processor subsystem,” “processor,” or “hardware processor” may refer to a processor, memory, software, or combination thereof that works together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or arithmetic element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.) that may be onboard or offboard, or that may be dedicated for use by the hardware processor subsystem.

[0038] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0039] In other embodiments, the hardware processor subsystem may include dedicated circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs). These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0040] Next, referring to Figure 2, a high-level diagram of a system and method 200 for panoptic segmentation training using a multi-dataset part-to-whole recognition model, according to an embodiment of the present invention, is illustrated exemplarily.

[0041] In various embodiments, in block 202, the multi-dataset label space may be organized to include various object categories from multiple datasets, such as D1 which may encompass categories like [person, car, ...] and D2 which may encompass categories like [face, shoe, ...]. This structure allows the system to manage and interpret diverse semantic annotations, facilitating a comprehensive approach to recognizing and classifying a wide range of object classifications that may differ significantly across datasets. This capability enhances the generalization of the model across datasets with different label definitions, supporting effective training and accurate predictions. In blocks 201, 203, 205, 207, and 209, text descriptions of objects can be generated or retrieved and mapped to categories identified in the multi-dataset label space. Blocks 201 and 203 can handle categories from D1, while blocks 205, 207, and 209 can handle categories from D2. These blocks can generate detailed text descriptions essential for generating text embeddings, which can then be used in subsequent steps to enhance the alignment and integration of text and image data for more accurate object recognition and segmentation.

[0042] Block 204 allows the system to input images featuring objects for segmentation. This block manages the initial processing of these images, including standardization of image formats, adjustment of dimensions, and improvement of image quality through techniques such as contrast adjustment and color correction. These preprocessing steps can be tailored to optimize the images for feature extraction, ensuring that the visual data is in optimal condition for accurate analysis and processing by the encoder. Block 206 allows the use of the CLIP (Contrastive Language Image Pretraining) model, where CLIP text objects can be processed by integrating both text and visual data using an advanced neural network architecture. This integration leverages cutting-edge techniques in natural language processing and computer vision to create robust embeddings that capture the subtle interactions between text descriptions and the visual features of objects. These embeddings are crucial for understanding and classifying objects within images, providing rich semantic context that significantly improves the model's segmentation capabilities.

[0043] In block 211, the output from CLIP text objects, including text embeddings generated from the detailed descriptions in blocks 201, 203, 205, 207, and 209, can be formatted and aligned for integration with visual features. This alignment is important to ensure that the text embeddings correspond accurately to the visual data and to facilitate the seamless combination of these two data types in subsequent processing steps. In block 213, the preprocessed images from block 204 can undergo further processing to extract visual features suitable for integration with the text embeddings from block 211. This processing can apply convolutional neural networks or other feature extraction methods that can analyze visual information and extract it into a format compatible with text data, thereby enhancing the system's ability to accurately match and integrate these different formats of data.

[0044] In block 208, an image / text product matrix can be created by performing a dot product between text embeddings from block 211 and visual features from block 213. This matrix serves as a crucial element of the system's processing pipeline, enabling the model to effectively reconcile and integrate text and visual data. This matrix can comprehensively represent the relationships between text and image data, helping to predict object categories within images with greater accuracy and reliability. In block 210, predictive labels can be generated based on a sophisticated analysis of the image / text product matrix from block 208. These labels, influenced by a unified understanding of text and image data, may be essential for identifying object categories within images. The accuracy of these predictions plays a critical role in the overall effectiveness of the segmentation process, guiding the system to produce reliable and semantically consistent segmentation results.

[0045] Block 212 aggregates source images from multiple datasets and prepares them for encoding. This block manages the collection, standardization, and preliminary processing of these images, ensuring they are in a uniform format suitable for detailed analysis and feature extraction by the encoder in Block 214. Block 214 utilizes the encoder to extract and encode features from the images provided by Block 212. This encoder can analyze image data using various advanced deep learning techniques and extract it into a condensed, feature-rich format, which is crucial for accurate object segmentation and classification.

[0046] In block 216, the Dataset Embedding and Overlap Mask Selection Module (MPA module) can be activated to manage the integration of features from the multi-dataset label space with encoded image features. This module can address and resolve potential conflicts arising from overlapping label definitions by applying a selective masking strategy, ensuring that semantic consistency of dataset-specific classifications is maintained. In block 218, output embeddings and masks are generated, representing the final segmentation masks and their corresponding embeddings, based on predictions from block 210. These outputs are essential for evaluating the model's segmentation accuracy and are used for further evaluation against ground truth data.

[0047] In block 220, ground truth labels can serve as a reference criterion for evaluating predicted labels from block 210. These labels are crucial for training the model to accurately predict object categories and serve as a benchmark for measuring the effectiveness of the training process and tuning the model's performance accordingly. In block 222, ground truth masks can define the precise boundaries of objects annotated in the training dataset. These masks are essential for evaluating the spatial accuracy of the model's segmentation output and ensure that the model's predictions accurately reflect the actual contours of the objects.

[0048] In block 224, a bipartite matching loss can be calculated to evaluate the consistency between the predicted labels and the masks from block 218, and between the ground truth labels from block 220 and the masks from block 222. This loss function can be used to quantitatively measure the performance of the model and provides an index for optimizing the model during training to improve accuracy and reduce segmentation errors. The present invention effectively addresses the challenges of training panoptic segmentation models across multiple datasets by managing the relationship between parts and wholes and the competition of dataset-specific labels, and, according to aspects of the present invention, generates robust and accurate segmentation outputs applicable to a wide range of visual contexts.

[0049] Referring here to Figure 3, an illustrative diagram shows a system and method 300 for panoptic segmentation using a multi-dataset partial-to-whole recognition pipeline according to an embodiment of the present invention.

[0050] In various embodiments, block 302 allows images to be input to the system for processing. This block can receive images from various sources, ensuring compatibility with subsequent processing stages. These images can serve as the primary data source for panoptic segmentation tasks and potentially contain multiple objects and scene types from diverse environments. The image input unit can handle various image formats and conditions, adapting them as needed to maintain the integrity of the visual data for further analysis. In block 304, an encoder can process the input image to extract features. This encoder can utilize deep neural networks, such as convolutional neural networks (CNNs) or converter-based models, to transform raw image data into a high-dimensional feature space. These features can capture important visual cues such as texture, shape, and context, which are essential for accurate segmentation. The encoder output provides a rich and condensed representation of the original image data, optimized for effective segmentation in subsequent stages.

[0051] In various embodiments, block 306 may contain the core functionality of the system, where multiple processes can be orchestrated to handle the complexity of panoptic segmentation across diverse datasets. This pipeline may include cross-attention and thresholding mechanisms that can improve the segmentation process by focusing on relevant features and minimizing the impact of dataset-specific anomalies. Block 310 may generate dataset embeddings to encode the unique properties and label spaces of the various datasets involved in the training process. This embedding process can leverage pre-trained models or custom algorithms to map dataset identifiers and their associated properties into a continuous vector space. These embeddings help to reconcile and normalize features extracted from diverse datasets, facilitating consistent handling across the entire multi-dataset training environment. Blocks 308 and 312 may include integration of processed image features with dataset embeddings. This integration allows the system to maintain recognition of the source dataset for each image, preserving the context necessary for accurate segmentation while handling potentially conflicting annotations from different datasets.

[0052] In block 314, thresholding and cross-attention mechanisms can be applied to the integrated embeddings and features. These processes can enhance the distinction between different semantic categories and instances within an image, especially when there are overlapping or closely related labels across datasets. By dynamically adjusting thresholds and focusing attention, the system can improve the accuracy and consistency of segmentation. Blocks 311, 313, and 315 can represent dataset-specific processing units for datasets D1, D2, and Dx, respectively. Each unit can handle the specificity of its corresponding dataset, applying customized rules and adjustments to accommodate its unique label space and annotation criteria. These units ensure that the training and inference phases of the segmentation model are sensitive to the nuances of each dataset, improving the overall performance and adaptability of the model. In block 316, predictive masks can be generated based on the processed and integrated features and embeddings. These masks can define the boundaries and categories of various objects within each image, in line with the combined knowledge extracted from multiple datasets.

[0053] In block 318, the POMP (Panoptic Overlapping Mask Prediction) module can finalize the segmentation process. This module can selectively combine and refine the predicted masks from block 316, resolving any conflicts and ensuring that the most accurate label is assigned to each pixel in the output. The POMP module is crucial for achieving high-quality panoptic segmentation output, especially in multi-dataset training scenarios where part-whole relationships and overlapping categories may present further challenges. In block 320, the final output can be generated, containing a unified panoptic segmentation map. These maps can display a comprehensive and instance-aware segmentation of the input image, reflecting a combined and harmonized understanding of the multiple datasets processed by the system. This output can be used in a variety of applications requiring a detailed semantic understanding of visual scenes, such as autonomous driving, robot navigation, and advanced image analysis systems. This output can be a final modified mask image, the result of panoptic segmentation processing, and may include a unified panoptic segmentation map that integrates predictive masks for each category across multiple datasets. These maps can, for example, display both "things" and "events" with clear instance boundaries, providing a comprehensive visual representation of the segmented elements in the image. Each pixel in the output image can be classified into the most appropriate category by applying a clear segmentation mask to distinguish between overlapping and adjacent segments. This output is particularly useful for applications that rely on accurate and detailed image segmentation, such as autonomous driving systems that require a precise understanding of the surrounding environment for real-time navigation and decision-making, according to aspects of the present invention.

[0054] Next, referring to Figure 4, an illustrative diagram shows a system and method 400 for multi-dataset panoptic segmentation according to an embodiment of the present invention.

[0055] In various embodiments, in block 402, a panoptic segmentation dataset is used as the basic input to the segmentation system. These datasets can encompass diverse image categories and settings, and each dataset may follow its own set of category definitions and annotations. In this block, these datasets can be managed and preprocessed to prepare them for feature extraction and further analysis, while addressing the variation in the label space resulting from the heterogeneous nature of the data sources. In block 404, a panoptic segmentation model functions as the core processing unit in the system. This model may be based on a transformer architecture optimized for processing complex image data and extracting detailed segmentation predictions. This model consists of several key elements: a backbone for initial feature extraction, a decoder for deriving the segmentation mask, and an inference mechanism for final mask adjustment and output generation.

[0056] In block 406, the segmentation model's backbone can process the input image to extract key visual features. This element can be analyzed using advanced neural network architectures, such as deep convolutional networks, and condensed into a comprehensive feature set that captures both the texture and contextual nuances of the visual input. In block 408, the decoder can take in the features extracted by the backbone and apply a series of transformations to predict a detailed segmentation mask for each object in the image. This decoder can process a fixed set of object queries and leverage dataset-specific embeddings to improve the accuracy of distinguishing similar categories across different datasets and mitigate potential label space competition.

[0057] In block 410, the panoptic inference unit can process the segmentation masks predicted by the decoder to generate a unified panoptic map. This element may include a novel inference algorithm designed to effectively manage the relationship between parts and wholes by allowing the smaller mask to override the larger mask when both have high confidence levels and the smaller mask completely encloses the larger mask. This approach assigns the most accurate class and instance to each pixel, improving the overall quality and usability of the segmentation output. In block 412, dataset-specific queries can be generated to tailor the segmentation process to the unique characteristics of each dataset used for training. These queries influence the decoder by providing context about which dataset the image originates from, allowing the model to adjust its predictions to align with the specific semantic and instance annotations of that dataset. This customization can significantly improve the model's ability to handle diverse datasets without sacrificing accuracy.

[0058] In block 414, the system output can ultimately be a segmented image in which each pixel is precisely classified into a semantic category and assigned to a specific instance based on a unified panoptic map generated by the inference unit. This output can be used in a variety of applications requiring detailed image understanding, such as autonomous driving, robot navigation, and advanced surveillance systems, providing comprehensive insights into the visual scene. The system and method 400 are designed to handle complex segmentation tasks across various datasets with high efficiency and accuracy, and according to aspects of the present invention, they are valuable tools for applications requiring advanced image analysis capabilities.

[0059] Next, referring to Figure 5A, an illustrative diagram shows a system and method 501 for multi-dataset panoptic segmentation training according to an embodiment of the present invention.

[0060] In various embodiments, block 502 can compile a diverse collection of training datasets, each characterized by a unique label space containing various object categories such as "people," "cars," "faces," and "shoes." This compilation serves as a foundation for training a panoptic segmentation model, aiming to equip the model with the ability to handle a wide range of visual and semantic inconsistencies between datasets. This block ensures that each label is accurately represented during the training process and can enhance the model's ability to generalize across datasets with potentially conflicting annotations. Furthermore, integrating multiple label spaces can facilitate a comprehensive understanding of the relationships and distinctions between categories, which is particularly useful for the model's performance in real-world scenarios requiring real-time decision-making and tuning of system components for optimal system safety and / or performance.

[0061] Block 504 incorporates a CLIP text encoder that can convert the text descriptions of dataset labels into high-dimensional embeddings. By leveraging natural language processing techniques within the CLIP framework, this encoder can generate semantic embeddings that capture the contextual nuances of each label. This process enhances the model's interpretability and enables seamless correlation between visual data and textual information. This encoding step is crucial for generating robust connections between diverse semantic labels and their visual correspondences, ensuring that the embeddings are optimized for subsequent integration with visual features. In Block 506, the output from the CLIP text encoder is text embeddings that encapsulate the semantic essence of the dataset labels. These embeddings are essential for training segmentation models, providing a rich semantic layer that complements the visual input. By integrating these embeddings, the model can gain a deeper understanding of the semantic content of images, which can then be used for accurate classification and segmentation of complex scenes.

[0062] Block 508 allows for the management of embedding structure and indexing to map embeddings to specific dataset labels. This functionality can include mapping each embedding data to a label identifier, facilitating the structural handling of embedding data during the training phase. Proper indexing is crucial to ensure the model leverages the correct semantic information of each image and maintains consistency and accuracy in the training output. In Block 510, the embedding layer can refine the indexed embeddings from Block 508. This layer can further process the embeddings to align them with the neural network architecture and optimize their interaction with visual features extracted from the images. The refined embeddings can be utilized more effectively within the model, improving the semantic resolution of the segmentation task.

[0063] Block 512 generates Label Space-Specific Query Embeddings (LSQEs) that tailor the model's response to the unique characteristics of each dataset's label space. These query embeddings can direct the segmentation process by adjusting the model's focus according to the specific semantic requirements of each label space, facilitating accurate segmentation across diverse datasets. Block 514 derives object queries from the LSQEs to guide the segmentation model in identifying and classifying various objects within an image. These queries act as the model's focus, highlighting areas of interest or concern within the visual data and ensuring the segmentation process is accurate and relevant to the specific characteristics of each dataset. Block 516 includes input source images from a compiled multi-dataset collection. These images provide the visual data necessary for the model to apply the learned features and embeddings to actionable segmentation tasks. The diversity of image sources necessitates the model adapting its strategy across various visual contexts, which is crucial for developing a versatile and robust segmentation capability.

[0064] In block 518, the image encoder processes the source image to extract essential visual features. This encoder may employ advanced convolutional networks and can analyze the image to generate feature maps that highlight key visual patterns and structures for use in performing accurate and effective segmentation. Block 520 generates image embeddings from the visual features extracted by the image encoder in block 518. These embeddings can encapsulate important visual information in a format that can be easily integrated with text embeddings, facilitating a deeper, multimodal understanding of the image. In block 522, the decoder can utilize object queries, along with both text and image embeddings, to perform the segmentation task. This element can synthesize diverse inputs to predict a segmentation mask that accurately reflects the semantic and visual content of the image. Block 524 calculates the training loss, which measures the effectiveness of the segmentation prediction against known ground truth. This metric can guide model optimization, pinpointing areas that need adjustment to minimize errors and improve segmentation accuracy. This feedback loop is used to improve the model's performance and ensure that it meets the stringent requirements of multi-dataset segmentation according to aspects of the present invention.

[0065] Referring next to Figure 5B, an illustrative diagram shows a system and method 503 for multi-dataset panoptic segmentation inference according to an embodiment of the present invention. This system and method 503 is designed to integrate various processes, including image and text encoding, embedding refinement, and decoding, in order to achieve accurate segmentation. This framework ensures that each element is tailored to effectively handle the complexities of image processing across diverse label spaces.

[0066] In various embodiments, block 532 can collect a sequence of labels from this exemplary designated D3 label space to guide the inference behavior of the segmentation model. This label space includes diverse categories such as "person" and "face," and is crucial in providing the semantic framework necessary for accurately processing the input image. This block can enhance the model's ability to interpret image data in relation to specific label characteristics, which is crucial for maintaining consistency and relevance of segmentation results across different image sets. In block 534, the CLIP text encoder can be used to convert text descriptions related to the test label space 532 into detailed semantic embeddings. This encoder leverages advanced natural language processing techniques to capture subtle meanings embedded in the text and convert them into a format that the segmentation model can process. The encoder's ability to generate high-quality, meaningful embeddings is fundamental to ensuring that the model's interpretation of the visual content is deeply understood by the corresponding text metadata, thereby improving the overall accuracy of the segmentation process.

[0067] Block 536 dynamically predicts the most appropriate label space for the image being analyzed using complete label space embeddings. This process allows for adaptive selection and adjustment of labels based on specific visual and contextual cues present in the image, significantly improving the flexibility and accuracy of the model in real-time applications. The prediction mechanism in this block can optimize label selection by evaluating the compatibility of various label configurations with the input image, thereby improving the accuracy and applicability of the segmentation output. Block 538 stores a comprehensive set of vector embeddings containing all available labels in the system's database. This block serves as a critical resource during the predictive labeling process, providing a rich repository of semantic information that the model can use to enhance its predictions. The embeddings in this block are meticulously maintained to reflect the latest semantic developments and label additions, thereby supporting the continuous adaptability and learning capabilities of the segmentation model.

[0068] Block 539 allows for the generation of a predicted label space based on an advanced prediction algorithm that utilizes improved embeddings from the complete label space embeddings 538. This sophisticated prediction mechanism considers the visual characteristics of the current image, along with semantic insights from the text embeddings, to select the most appropriate label for each image. This block ensures that the label space used during segmentation is optimally suited to the input specifications by dynamically aligning the segmentation process with the complexity of the visual content, thereby promoting higher accuracy and relevance in the model's output. Block 540 includes an embedding layer that further processes and refines the predicted embeddings to precisely match the requirements of the segmentation model. This layer can tune the embeddings for optimal compatibility, increasing their usefulness by improving their accuracy and detail. This step is crucial for the embeddings to effectively communicate the necessary semantic and visual cues to the model, enabling the model to perform segmentation with greater accuracy and efficiency.

[0069] In block 542, label space-specific query embeddings (LSQEs) can be generated to provide specific and detailed queries to the segmentation model. These embeddings can be tailored to the unique requirements of each predicted label space, allowing the model to concentrate computational resources on relevant segments of the image. The LSQE processing includes advanced algorithms that interpret the improved embeddings and translate them into actionable queries, thereby directing the segmentation task with enhanced specificity and effectiveness. In block 544, object queries derived from the LSQEs can be used to guide the segmentation model in identifying and classifying various objects within the image. These queries can identify specific features or regions within the image that are crucial for accurate classification and segmentation. The accuracy of these queries is critical to the model's ability to identify similar objects and correctly apply predicted labels, thereby ensuring that the segmentation is accurate and relevant to the input characteristics.

[0070] In block 546, the image can be input to the system for processing. This block can handle various image formats and conditions and applies initial preprocessing steps to standardize the image for consistent analysis. Preprocessing may include adjustments for illumination, alignment, and scaling, which are essential for preparing the image for subsequent detailed feature extraction and embedding processes. In block 548, the image encoder processes the input image to extract key visual features for use in segmentation. This encoder employs advanced techniques to analyze the visual data and generates high-quality image embeddings that capture the essential features of each segment in the image. The effect of this block ensures that the visual data is accurately represented, complementing the text embeddings used in the segmentation model. In block 550, image embeddings can be created from the visual features extracted by the image encoder. These embeddings encapsulate detailed visual information that is important for the segmentation process. These embeddings provide a comprehensive visual representation that the decoder can use together with text embeddings to perform detailed and accurate segmentation.

[0071] In block 552, the decoder can utilize both object queries and image embeddings to perform the segmentation task. This element synthesizes diverse inputs to generate accurate segmentation masks and classifications. The decoder's ability to integrate and interpret complex data is effectively utilized in generating detailed segmentation maps that accurately reflect the combination of semantic and visual content of the image. In block 554, classes can be determined based on the output from the decoder. This process can classify the segmented portions of the image into defined classes according to the predicted label space. This classification can be used to structure the segmentation output into a usable format, facilitating further analysis or practical application of the segmented data. In block 556, masks can be generated and / or applied that accurately define the boundaries of each classified segment in the image. These masks can be used to visualize the segmentation results, providing a clear and distinct representation of each object or region classified by the system. The detailed and precise properties of these masks are particularly useful, according to aspects of the present invention, for applications requiring accurate and / or real-time segmentation, such as medical image processing, autonomous navigation, or detailed geographic image processing.

[0072] Next, referring to Figure 6, an illustrative diagram shows a method 600 for training multi-dataset panoptic image segmentation according to an embodiment of the present invention.

[0073] In various embodiments, block 602 can be used as input for multiple datasets, each with its own label space and ground truth annotations. These datasets are selected to cover a diverse range of categories and semantic labels, ensuring that the model encounters a variety of semantic scenarios during training. This block lays the foundation for handling inconsistent semantics across datasets by providing a comprehensive input dataset that reflects the complexity of real-world visual scenes. Block 604 allows for the processing of individual datasets to identify and classify existing label spaces and semantics. This processing may include analyzing label overlaps and classifying objects as "things" or "events" based on countability and semantic significance. The insights gained here help better prepare the model for the complexity of multi-dataset training.

[0074] In block 606, a CLIP-based model can be used to integrate language-based embeddings of class names, creating a unified semantic space. This step allows the model to handle semantically similar categories cohesively across different datasets. Integrating language embeddings maintains a high level of semantic understanding despite the presence of label inconsistencies. In block 608, label space-specific query embeddings can be generated. These embeddings are designed to condition the transform-decoder to specific data semantics, enhancing the model's ability to handle competing label spaces. This approach allows the model's behavior to dynamically adapt based on the active label space, both during training and inference. In block 610, a segmentation model can be trained using a modified Mask2Former framework that incorporates enhancements from blocks 606 and 608. This model employs a multi-layer transform-decoder that processes images using both visual features and the newly integrated query embeddings. This training step is crucial for aligning the model's output with the complex and overlapping label spaces encountered in multi-dataset environments.

[0075] In block 612, inference can be performed on new images using a combination of label spaces from the training dataset. The model can predict the label space to apply by matching text embeddings of class names with query embeddings. This process ensures the versatility of the model, allowing it to handle any combination of labels during inference and significantly improving its usability in a variety of applications. In block 614, the model's performance can be evaluated across various benchmarks, including those specifically designed to assess its capabilities in mixed label space scenarios. Performance improvements can be quantified and the effectiveness of the RESI framework validated using metrics such as mloU for semantic segmentation, PQ for panoptic segmentation, and AP for instance segmentation. This method 600 demonstrates that the novel RESI framework can be used for training multi-dataset image segmentation to address the challenge of semantic inconsistencies across combined datasets. This approach, according to aspects of the present invention, leverages advanced techniques such as language-based embeddings and label space-specific query embeddings to ensure robust performance even when faced with semantic inconsistencies.

[0076] Next, referring to Figure 7, an illustrative diagram shows a method 700 for multi-dataset panoptic segmentation according to an embodiment of the present invention.

[0077] In various embodiments, in block 702, images may be processed from multiple datasets, each containing its own unique label space. This block manages the preliminary processing of these images, which may include tasks such as resizing, color correction, and other normalization processes to prepare the data for effective feature extraction. The diversity of datasets allows the model to encounter various visual scenarios, which can help develop robust segmentation capabilities. In block 704, a backbone network can be used to extract multiscale features from the processed images. This network can be a convolutional neural network (CNN) or a transformer network that processes the computation of visual features at various scales and depths. Multiscale feature extraction can be used to capture both detailed and broad aspects of an image, providing a comprehensive set of features to support complex segmentation tasks.

[0078] In block 706, text embeddings of class names can be generated from the unique label space of each dataset. This process utilizes a pre-trained visual-language model to transform textual categorical descriptions into a dense vector form that encapsulates semantic meaning. These embeddings facilitate consistent handling of categorical names across various datasets, fill semantic gaps, and enhance the model's ability to generalize across diverse annotation criteria. In block 708, text embeddings can be integrated with visual features extracted from images to generate a unified semantic space. This integration can leverage methods such as concatenation or feature fusion, enabling the model to effectively correlate and combine text and visual data. The unified semantic space allows the model to more accurately align semantic concepts with visual patterns, which is crucial for predicting accurate segmentation output.

[0079] Block 710 allows training a transformer-based segmentation model using a unified semantic space. This training involves fitting the model to accurately predict the segmentation mask and class of an incoming image using a rich feature set developed from integrated text and visual data. Transformer architectures are particularly effective when processing this type of data due to their ability to model complex dependencies and relationships within the data. Block 712 allows performing inference using a novel panoptic segmentation inference algorithm to generate a unified panoptic segmentation map from the predicted segmentation mask and class. This block can handle the task of resolving competition in segmentation prediction, particularly competition arising from the diverse label spaces of the training dataset. The inference process can ensure that each pixel is assigned the most appropriate semantic category and instance ID, resulting in a consistent and accurate panoptic map.

[0080] In block 714, the inference algorithm can resolve conflicting annotations from multiple datasets by allowing the smaller mask to override the larger mask if both have a confidence level above a certain threshold, the smaller mask is completely contained within the larger mask, and they belong to different classes. This specific mechanism ensures that detailed features, such as faces in the context of the whole person, are not obscured by broader segmentation, thus maintaining the completeness and granularity of the panoptic output according to aspects of the present invention.

[0081] Next, referring to Figure 8, an exemplary diagram is shown illustrating a high-level diagram of a system and method 800 for image processing based on multi-dataset panoptic segmentation across various computer vision and image processing tasks in multiple exemplary environments, according to embodiments of the present invention.

[0082] In various embodiments, the computing network 801 can function as a communication infrastructure connecting multiple devices and environments. This network can support various data transmission protocols and handle high-bandwidth operations to facilitate real-time data exchange and coordination between various system components. An end-user 802 within this system can interact with the technology using user equipment 804. The user can initiate image processing tasks, provide inputs for customizing multi-dataset panoptic segmentation, resolve inconsistent semantics, and receive processed outputs. The end-user can interact with the system through a user interface that enables real-time feedback that can be used to specify parameters, transmit images, or fine-tune processing algorithms.

[0083] User devices 804 can encompass a wide range of technologies, including smartphones, tablets, laptops, and desktop computers. These devices may be equipped with dedicated software that allows users to upload images for processing and make adjustments to images, queries, etc., according to aspects of the present invention. Furthermore, the devices may have varying processing capabilities, some capable of performing basic image processing tasks locally, while others may rely on remote servers for more complex calculations. To perform complex calculations involved in multi-dataset panoptic segmentation and other image processing tasks, computing devices 806 (e.g., a server, user device, local, remote, etc.) can be utilized. This device may be a server, user device, or a combination of local and remote processing units equipped with a powerful CPU or GPU capable of performing the intensive calculations required for real-time use of the present invention. The computing devices may operate backend processing according to aspects of the present invention and may also host algorithms for performing various tasks.

[0084] Block 808 employs a multi-dataset panoptic segmentation system to enhance urban planning and smart city management. This system can process diverse image data from the urban environment, including traffic conditions, pedestrian flow, and infrastructure status, across multiple datasets, including satellite imagery, CCTV feeds, and drone aerial footage. By integrating and segmenting these datasets with high precision, it provides comprehensive insights into urban dynamics, facilitating efficient urban planning, resource allocation, and optimized emergency response. The ability to process datasets with diverse label spaces is particularly valuable in urban environments where various systems may be used to classify and annotate urban functions.

[0085] In Block 810, the system can make a significant contribution to autonomous vehicle navigation systems. It can process real-time visual data from multiple sources, including on-board cameras and existing geographic information systems, to generate accurate panoptic maps of the vehicle's surroundings. These maps include detailed classification and localization of all visible objects, such as other vehicles, pedestrians, road signs, and lane markings, which are essential for safe and efficient navigation. The system's robustness against label space inconsistencies ensures reliable performance even when integrating datasets from various geographic regions and manufacturers. In Block 812, this system can be applied to enhance surveillance and security systems. It can analyze video from multiple security cameras in various locations and segment and identify various elements such as individuals, vehicles, and objects in complex scenes. The ability to be trained on diverse datasets allows the system to adapt to various scenarios and lighting conditions, improving threat detection and situational awareness in security-critical environments such as airports, shopping centers, and public squares.

[0086] Block 814 allows the system to be used for agricultural monitoring and management. By processing images from satellites, drones, and field cameras, the system can segment and classify various crops, assess plant health, and monitor pest and disease outbreaks. Integrating multiple agricultural datasets enables more accurate and detailed panoptic segmentation, facilitating effective intervention strategies, optimizing resource use, and enhancing yield forecasting. Furthermore, it includes supporting environmental monitoring by analyzing images from diverse ecosystem datasets, including forest areas, water systems, and urban biomes. The system can detect changes in vegetation cover, water levels, and pollution patterns, contributing to efforts in climate change research, habitat conservation, and disaster management. The ability to seamlessly integrate and interpret panoptic data from various sources is essential for accurately tracking environmental changes and implementing timely conservation measures.

[0087] Block 816 can transform medical image processing and analysis by applying segmentation capabilities to diverse medical image datasets, such as MRI scans, X-rays, and ultrasound images. This system can assist in the detection and segmentation of tumors, fractures, and other pathological features from various image modalities, improving diagnostic accuracy and personalized treatment planning. The system's robustness in handling various medical annotation standards and imaging techniques ensures high reliability and adaptability in clinical settings. Block 818 allows the system to be applied to retail and inventory management, analyzing images from store cameras to monitor product placement, shelf arrangement, and customer interaction patterns. The ability to train on panoptic segmentation and apply it to various retail environments helps optimize store layouts, improve customer experience, and automate inventory level assessment, thereby improving operational efficiency and profitability. While System and Method 800 are described exemplarily to be applied to the specific environments described above, it should be understood that the present invention is versatile and therefore can be used in all kinds of environments across multiple different real-world applications according to aspects of the invention.

[0088] Next, referring to Figure 9, an illustrative diagram shows an image processing based on multi-dataset panoptic segmentation of system 900 according to an embodiment of the present invention.

[0089] In various embodiments, in block 902, the image receiving unit can process the acquisition of images from diverse datasets, ensuring that visual data from various sources is correctly formatted and synchronized for consistent processing. This unit supports a variety of image types and sources, which is crucial for maintaining a versatile and robust input stream for segmentation tasks. In block 904, the feature extraction engine can analyze the received images to derive multi-scale visual features essential for detailed segmentation. This engine utilizes either a deep convolutional neural network or an advanced transformation model to extract layers of features that capture both macro and micro elements of the visual data, which are important for understanding complex scenes. In block 906, the text embedding generator can process text information from the label space of each dataset to generate semantic embeddings. By employing a pre-trained visual-language model, this unit transforms class names and other text descriptors into a dense vector form that captures the unique semantic characteristics of each class, facilitating deeper integration of text and visual data.

[0090] In block 908, the semantic space integration unit can integrate text embeddings with extracted visual features to form a unified semantic space. This integration forms the basis for generating a coherent and unified representation of the visual and text data, enhancing the model's ability to accurately interpret and segment complex datasets. In block 910, the segmentation model training unit can train a converter-based segmentation model using the unified semantic space. This training involves adapting the model to effectively predict accurate segmentation masks and classes by leveraging the rich and integrated feature set provided by the previous stage. In block 912, the panoptic segmentation map generator can compile the output from the segmentation model (particularly the mask and class predictions) into a comprehensive panoptic map. This unit uses sophisticated algorithms to ensure all elements are correctly placed and classified, providing a detailed and actionable segmentation output.

[0091] In block 914, the storage device can archive all relevant data, including raw input images, processed features, intermediate data, and the final segmentation map. This device ensures data availability and integrity for both real-time processing and historical analysis, supporting the needs of system operation and evaluation. In block 916, the data synchronization unit can manage the consistency and timing of data flow between processing stages. This unit ensures that data moves smoothly from one stage to another without bottlenecks or data loss, optimizing the overall efficiency of the segmentation process. In block 918, the inference algorithm optimizer can refine the algorithm used in the final inference stage that generates the panoptic segmentation map. This unit adjusts algorithm parameters in response to feedback from the system's output, aiming to improve accuracy and reduce inconsistencies between overlapping segmentation labels from different datasets.

[0092] In block 920, a model update interface facilitates the feeding of new data and insights back into the model training process. This interface enables continuous learning and model refinement, maintaining system effectiveness even as new datasets are added or existing datasets evolve. In block 922, a performance monitoring unit tracks the effectiveness of segmentation tasks and provides analysis of accuracy, speed, and reliability. This unit helps identify areas for improvement and supports continuous system enhancement to maintain high levels of performance. In block 924, a user interaction interface can provide system users with the ability to input parameters, receive outputs, and interact with the segmentation system. This interface supports customization of segmentation tasks and allows users to access detailed results for analysis or further processing. The system integration bus 901 supports communication and data transfer across all elements of the system, enabling each system element to exchange information efficiently in real time. This bus can be used, according to aspects of the present invention, to maintain consistency and timing of segmentation processing across the entire system architecture.

[0093] In this specification, any reference to “one embodiment” or “one embodiment” of the present invention, and to other modifications, means that certain features, structures, properties, etc., described in relation to the embodiments are included in at least one embodiment of the present invention. Therefore, expressions such as “in one embodiment” or “in one embodiment” appearing elsewhere in this specification, and any other modifications, do not necessarily all refer to the same embodiment. However, it should be understood that, considering the teachings of the present invention provided herein, features of one or more embodiments can be combined.

[0094] For example, in the case of "A / B," the use of any of the following " / ," "and / or," or "at least one," such as "A and / or B" or "at least one of A and B," will be understood as intended to include the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to include the selection of only the first listed option (A), only the second listed option (B), only the third listed option (C), only the first and second listed options (A and B), only the first and third listed options (A and C), only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended as many times as there are listed items.

[0095] The foregoing is to be understood in all respects to be illustrative and not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description but from the claims as interpreted in accordance with the full width permitted by patent law. The embodiments shown and described herein are merely illustrative of the invention, and those skilled in the art should understand that various modifications can be implemented without departing from the scope and spirit of the invention. Those skilled in the art can implement various other combinations of features without departing from the scope and spirit of the invention. Thus, while aspects of the invention have been described with the detail and specificity required by patent law, what is claimed and intended to be protected by the patent is as stated in the appended claims.

Claims

1. A computer-implemented method for multi-dataset panoptic segmentation, This involves processing images received from multiple datasets, each containing its own unique label space (702), and extracting multiscale features using a backbone network (704), For each of the aforementioned multiple datasets, generate text embeddings of class names from the unique label space (706), The visual features extracted from the received image and the text embedding are integrated (708) to generate a unified semantic space, Using the unified semantic space, a converter-based segmentation model is trained (710) to predict the segmentation mask and class of the received image, A method comprising generating a unified panoptic segmentation map from the predicted segmentation mask and classes by performing inference using a panoptic interference algorithm (712).

2. In the method according to claim 1, The backbone network is a method comprising a convolutional neural network or a transformer network that processes images to extract multi-scale visual features.

3. In the method according to claim 1, Furthermore, the method includes conditioning the transform-decoder with the semantics of a particular dataset by applying dataset-specific query embeddings to object queries during the training of the segmentation model.

4. In the method according to claim 1, The aforementioned text embedding is a method that generates text using a pre-trained visual-language model that maps category names from various datasets into a single, consistent space where semantic relationships are preserved.

5. In the method according to claim 4, The aforementioned pre-trained visual-language model is a CLIP (Contrast Language-Image Pretraining) model for generating text embeddings that map category names from various datasets into a consistent semantic space.

6. In the method according to claim 1, A method for resolving conflicting annotations from multiple datasets by allowing the inference algorithm to override a larger mask if both have a confidence level above a certain threshold, the smaller mask is completely contained within the larger mask, and they belong to different classes.

7. In the method according to claim 1, A method for enhancing the robustness and semantic understanding of the converter-based segmentation model, wherein the converter-based segmentation model is trained to handle overlapping label spaces from the multiple datasets.

8. A system for multi-dataset panoptic segmentation, Processor device (104) and The system has a memory (110) for storing instructions, and when an instruction is executed by the processor device, it is executed by the system. Images received from multiple datasets, each containing its own unique label space, are processed (702), and multiscale features are extracted using a backbone network (704). For each of the aforementioned datasets, text embeddings of class names are generated from the unique label space (706), The visual features extracted from the received image and the text embedding are integrated (708) to generate a unified semantic space. Using the unified semantic space, a converter-based segmentation model is trained (710) to predict the segmentation mask and class of the received image. A system that generates a unified panoptic segmentation map from the predicted segmentation masks and classes by performing inference using a panoptic interference algorithm (712).

9. In the system described in claim 8, The aforementioned text embedding is a system that generates category names from various datasets using a pre-trained visual-language model that maps them into a single, consistent space where semantic relationships are preserved.

10. In the system described in claim 9, The aforementioned pre-trained visual and linguistic model is a CLIP (Contrast Language-Image Pretraining) model system for generating text embeddings that map category names from various datasets into a consistent semantic space.

11. In the system described in claim 8, The aforementioned processor device is a system that executes instructions to apply a panoptic inference algorithm that resolves conflicting annotations by prioritizing smaller masks over larger masks based on differences in inclusion and class.

12. In the system described in claim 8, The system is characterized in that the memory stores instructions that enable the system to adapt to a changing label space from the dataset by employing language-based embedding, thereby enabling the system to process images from new or unknown datasets without retraining.

13. In the system described in claim 8, The converter-based segmentation model is trained to handle overlapping label spaces from the multiple datasets, and the system enhances the robustness and semantic understanding of the converter-based segmentation model.

14. In the system described in claim 8, A system for resolving conflicting annotations from multiple datasets by allowing a smaller mask to override a larger mask when both have a confidence level above a certain threshold, the smaller mask is completely contained within the larger mask, and they belong to different classes.

15. A computer program product for multi-dataset panoptic segmentation, the computer program product includes a computer-readable storage medium in which program instructions are embodied, and the program instructions are transmitted to a hardware processor. Images received from multiple datasets, each containing its own unique label space, are processed (702), and multiscale features are extracted using a backbone network (704). For each of the aforementioned datasets, text embeddings of class names are generated from the unique label space (706), The visual features extracted from the received image and the text embedding are integrated (708) to generate a unified semantic space. Using the unified semantic space, a converter-based segmentation model is trained (710) to predict the segmentation mask and class of the received image. A computer program product that generates a unified panoptic segmentation map from the predicted segmentation mask and class by performing inference using a panoptic interference algorithm (712).

16. In the computer program product described in claim 15, The aforementioned text embedding is a computer program product generated using a pre-trained visual-language model that maps category names from various datasets into a single, consistent space where semantic relationships are preserved.

17. In the computer program product described in claim 16, The aforementioned pre-trained visual and linguistic model is a computer program product that is a CLIP (Contrast Language-Image Pretraining) model for generating text embeddings that map category names from various datasets into a consistent semantic space.

18. In the computer program product described in claim 15, The program instructions are a computer program product that causes the hardware processor to resolve annotation conflicts from different datasets during inference by applying the panoptic algorithm to prioritize smaller masks when smaller masks are entirely contained within larger masks of different classes.

19. In the computer program product described in claim 15, The program instructions are a computer program product that enables adaptation to a changing label space from a dataset by employing language-based embeddings to process images from a new or unknown dataset without retraining.

20. In the computer program product described in claim 15, The program instructions enable the hardware processor to evaluate the performance of a trained model using an index that evaluates the model performance in overlapping label spaces, thereby enhancing its usefulness in diverse application scenarios, and the panoptic inference algorithm includes steps for sequentially arranging segmentation masks based on confidence scores and sizes in order to effectively manage overlapping masks and ensure accurate segmentation results, in this computer program product.

Images