System and method with language-guided self-supervised semantic segmentation
Through the language-guided self-supervised semantic segmentation method, combined with pre-trained visual language model and machine learning model, consistent pixel embedding is generated, solving the problems of high cost of manual annotation and limited recognition capabilities, and achieving high-quality semantic segmentation without manual annotation and unknown categories.
Patent Information
- Application Number
- CN202410588015.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing semantic segmentation technology relies on manual pixel-by-pixel annotation, which consumes labor and is costly, and the model recognition ability is limited to predefined semantic categories, making it difficult to deal with unknown categories.
Using language-guided self-supervised semantic segmentation method, the pre-trained visual language model and machine learning model is combined with data augmentation and contrast learning to generate consistent pixel embeddings to achieve high-quality semantic segmentation without manual annotation, which is suitable for known and Semantic segmentation task for unknown categories.
High-quality semantic segmentation without manual annotation is realized, the ability to discover unknown categories without predefined category names and significantly improves the segmentation accuracy of unknown categories and the clarity of pixel embeddings.
Smart Images

Figure CN118941827A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to computer vision, and more particularly to digital image processing, visual analysis, and semantic segmentation. Background Art
[0002] In general, semantic segmentation involves dividing an input image into semantically meaningful regions at the pixel level and assigning a semantic category label to each region. Recent advances in semantic segmentation rely heavily on pixel-by-pixel manual annotations. However, obtaining pixel-by-pixel annotations is extremely labor-intensive and costly. For example, it is known that pixel-by-pixel annotations take 1.5 hours to label an image. In addition, manual annotations are usually limited to predefined semantic categories, which may limit the learned model recognition capabilities to these predefined semantic categories. Summary of the invention
[0003] The following is an overview of the specific embodiments described in detail below. The described aspects are presented only to provide the reader with a brief overview of these specific embodiments, and the description of these aspects is not intended to limit the scope of the present disclosure. In fact, the present disclosure may cover various aspects that may not be explicitly set forth below.
[0004] According to at least one aspect, a computer-implemented method relates to language-guided self-supervised semantic segmentation. The method includes receiving a source image. The method includes generating a modified image by performing data augmentation on the source image. The modified image is a modified version of the source image. The method includes generating a first pixel embedding based on the modified image via a machine learning model. The method includes generating a first fragment embedding using the first pixel embedding. The method includes generating a second pixel embedding based on the source image via a pre-trained visual language model. The method includes generating a second fragment embedding by applying fragment contour data of the first fragment embedding to corresponding clusters of the second pixel embedding after performing data augmentation on the second pixel embedding. The method includes generating embedding consistency loss data by at least comparing the first fragment embedding with the second fragment embedding. The method includes generating combined loss data including the embedding consistency loss data. The method includes updating parameters of the machine learning model based on the combined loss data. The method includes providing or outputting a framework for language-guided self-supervised semantic segmentation. The framework includes (i) a pre-trained visual language model and (ii) the machine learning model after the parameters have been updated.
[0005] According to at least one aspect, a system relates to language-guided self-supervised semantic segmentation. The system includes at least a processor and a non-transitory computer-readable medium. The non-transitory computer-readable medium is in data communication with the processor. The non-transitory computer-readable medium has computer-readable data including instructions stored thereon, which instructions, when executed by the processor, are configured to cause the processor to perform a method. The method includes receiving a source image. The method includes generating a modified image by performing data augmentation on the source image. The modified image is a modified version of the source image. The method includes generating a first pixel embedding based on the modified image via a machine learning model. The method includes generating a first fragment embedding using the first pixel embedding. The method includes generating a second pixel embedding via a pre-trained visual language model based on the source image. The method includes generating a second fragment embedding by applying fragment contour data of the first fragment embedding to a corresponding cluster of the second pixel embedding after performing data augmentation on the second pixel embedding. The method includes generating embedding consistency loss data at least by comparing the first fragment embedding with the second fragment embedding. The method includes generating combined loss data including the embedding consistency loss data. The method includes updating parameters of the machine learning model based on the combined loss data. The method includes providing or outputting a framework for language-guided self-supervised semantic segmentation. The framework includes (i) a pre-trained visual language model and (ii) a machine learning model after parameters have been updated.
[0006] According to at least one aspect, a non-transitory computer-readable medium has computer-readable data including instructions stored thereon. The computer-readable data is executable by one or more processors to perform a method. The method includes receiving a source image. The method includes generating a modified image by performing data augmentation on the source image. The modified image is a modified version of the source image. The method includes generating a first pixel embedding based on the modified image via a machine learning model. The method includes generating a first fragment embedding using the first pixel embedding. The method includes generating a second pixel embedding based on the source image via a pre-trained visual language model. The method includes generating a second fragment embedding by applying fragment contour data of the first fragment embedding to corresponding clusters of the second pixel embedding after performing data augmentation on the second pixel embedding. The method includes generating embedding consistency loss data by at least comparing the first fragment embedding with the second fragment embedding. The method includes generating combined loss data including the embedding consistency loss data. The method includes updating parameters of the machine learning model based on the combined loss data. The method includes providing or outputting a framework for language-guided self-supervised semantic segmentation. The framework includes (i) a pre-trained visual language model and (ii) a machine learning model after the parameters have been updated.
[0007] These and other features, aspects and advantages of the present invention are discussed in the following detailed description with reference to the accompanying drawings, wherein like characters indicate like or similar parts throughout the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a diagram of an example of a system having a framework for language-guided self-supervised semantic segmentation according to an example embodiment of the present disclosure.
[0009] Figure 2A is a flowchart illustrating a training Figure 1 aspects of the framework.
[0010] Figure 2B is a diagram of a non-limiting example according to an example embodiment of the present disclosure, which illustrates Figure 2A The framework generates various aspects of the first loss data.
[0011] Figure 2C is a diagram of a non-limiting example according to an example embodiment of the present disclosure, which illustrates Figure 2A The framework generates aspects of the second loss data.
[0012] Figure 2D is a diagram of a non-limiting example according to an example embodiment of the present disclosure, which illustrates Figure 2A The framework generates aspects of third-party loss data.
[0013] Figure 3 is a diagram of a non-limiting example according to an example embodiment of the present disclosure illustrating aspects of computing a target class prototype.
[0014] Figure 4A is a diagram of a non-limiting example of an embedding space including semantics where a set of pixel embeddings from a pre-trained visual language model is not aligned with a set of pixel embeddings from a machine learning model according to an example embodiment of the present disclosure.
[0015] Figure 4B is a diagram of a non-limiting example of an embedding space including semantics in which a set of pixel embeddings from a pre-trained visual language model is aligned with a set of pixel embeddings from a machine learning model according to an example embodiment of the present disclosure.
[0016] Figure 5 is a diagram of a non-limiting example according to an example embodiment of the present disclosure, which illustrates that during inference Figure 2A aspects of the framework.
[0017] Figure 6 According to an exemplary embodiment of the present disclosure, Figure 2AA diagram of an example of a control system of the framework.
[0018] Figure 7 According to an exemplary embodiment of the present disclosure Figure 6 Schematic diagram of a control system for mobile machine technology.
[0019] Figure 8 According to an exemplary embodiment of the present disclosure Figure 6 Schematic diagram of a control system with regard to safety technology.
[0020] Fig. 9 According to an exemplary embodiment of the present disclosure Figure 6 Schematic diagram of the control system regarding imaging technology. DETAILED DESCRIPTION
[0021] From the foregoing description, the embodiments described herein (which have been shown and described by way of example) and their many advantages will be understood, and it will be apparent that various changes may be made to the form, construction, and arrangement of components without departing from the disclosed subject matter or sacrificing one or more of its advantages. Indeed, the description of these embodiments is merely illustrative. These embodiments are susceptible of various modifications and alternative forms, and the following claims are intended to cover and include such changes, and are not limited to the specific forms disclosed, but rather cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure.
[0022] Figure 1 is a diagram of a non-limiting example of a system 100 having a framework 200 configured for language guided self-supervised semantic segmentation (“CLIP-S 4 ”). The framework 200 is configured to learn consistent pixel embeddings about visual and conceptual semantics using self-supervised learning and guidance from a pre-trained visual language model 210 (e.g., contrastive language image pre-training (CLIP)). The framework 200 provides a self-supervised semantic segmentation method that combines pixel-fragment contrastive learning with guidance from a pre-trained visual language model 210. The framework 200 is configured to generate high-quality pixel embeddings without any human annotations. The framework 200 is applicable to various semantic segmentation tasks (e.g., language-driven semantic segmentation for known and unknown categories, unsupervised semantic segmentation using k-means clustering / linear classification, transfer learning for generating pixel embeddings such as mask tracking, etc.). In addition, the framework 200 provides language-driven semantic segmentation without any human annotations by introducing and solving a new task of category-free semantic segmentation. The advantage of the framework 200 is that it is configured to discover unknown categories from unlabeled image data without even knowing the unknown category names, rather than assuming that all category names are known during training.
[0023] The system 100 includes at least one processing system 110 having at least one processing device. For example, the processing system 110 includes at least an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any suitable processing technology, or any number and combination thereof. The processing system 110 is operable to provide the functionality as described herein.
[0024] The system 100 includes a memory system 120 that is operably connected to the processing system 110. In an example embodiment, the memory system 120 includes at least one non-transitory computer-readable storage medium that is configured to store various data and provide access to various data to enable at least the processing system 110 to perform operations and functions as disclosed herein. In an example embodiment, the memory system 120 includes a single memory device or multiple memory devices. The memory system 120 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology that can operate with the framework 200. For example, in an example embodiment, the memory system 120 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drive, memory card, optical storage device, magnetic storage device, memory module, any suitable type of memory device, or any number and combination thereof. Relative to the processing system 110 and / or other components of the framework 200, the memory system 120 is local, remote, or a combination thereof (e.g., partially local and partially remote). For example, the memory system 120 may include at least a cloud-based storage system (eg, a cloud-based database system) that is remote from the processing system 110 and / or other components of the framework 200 .
[0025] The memory system 120 includes at least a framework 200, machine learning data 130, and other related data 140 stored thereon. More specifically, the framework 200 includes computer-readable data having instructions that, when executed by the processing system 110, are configured to provide and implement language-guided self-supervised semantic segmentation via the framework 200. The computer-readable data may include instructions, codes, routines, various related data, any software technology, or any number and combination thereof. The framework 200 includes at least one pre-trained visual language model 210 and at least one machine learning model 220. The pre-trained visual language model 210 is pre-trained to perform image classification using open vocabulary categories. For example, the pre-trained visual language model 210 includes a pyramid scene parsing network (PSPNet or a semantic segmentation model using a pyramid parsing module) with an extended residual neural network (e.g., ResNet50 or a convolutional neural network with 50 layers). For example, the pre-trained visual language model 210 includes CLIP. For example, in Figure 2A In the embodiment of the present invention, the pre-trained visual language model 210 includes a modified image encoder 210A (e.g., MaskCLIP) configured to directly extract pixel embeddings related to text from the pre-trained visual language model 210 (e.g., CLIP). Figure 3 and Figure 5 As shown in , the pre-trained visual language model 210 includes a text encoder 210B, which is configured to generate a text embedding as an output in response to receiving one or more text / character data (e.g., category names) as input.
[0026] In addition, the machine learning model 220 is configured to generate a pixel embedding as an output when receiving at least one digital image as an input. The machine learning model 220 includes at least a convolutional neural network (CNN), any suitable encoding network, any suitable artificial neural network model, or any number and combination thereof. For example, in Figure 2A In the embodiment of the present invention, the machine learning model 220 includes at least one encoder 220A. In addition, the machine learning data 130 includes sensor data, source image data, various enhanced image data, various enhanced data, various class prototype data, various loss data, various weight data, various parameter data, and any related data that enables the framework 200 to learn and / or perform language-guided self-supervised semantic segmentation, as described in the present disclosure. Meanwhile, other related data 140 provides various data (e.g., operating system, etc.), which enables the framework 200 to perform functions as discussed herein.
[0027] The system 100 may include at least one sensor system 150. The sensor system 150 includes one or more sensors. For example, the sensor system 150 includes an image sensor, a camera, a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, an audio sensor, an inertial measurement unit (IMU), any suitable sensor, or any number and combination thereof. The sensor system 150 is operable to communicate with one or more other components of the system 100 (e.g., the processing system 110 and the memory system 120). For example, the sensor system 150 may provide sensor data, which the processing system 110 then uses to generate image data (e.g., a digital image) based on the sensor data. In this regard, the processing system 110 is configured to obtain sensor data directly or indirectly from one or more sensors of the sensor system 150. The sensor system 150 is local, remote, or a combination thereof (e.g., partially local and partially remote). Upon receiving the sensor data, the processing system 110 is configured to process the sensor data and / or generate image data (e.g., a digital image) associated with the framework 200, the machine learning data 130, other related data 140, any suitable software and / or hardware components, or any number and combination thereof.
[0028] In addition, the system 100 may include at least one other component. Figure 1 As shown in , the memory system 120 is also configured to store other related data 140, which is related to the operation of the framework 200 associated with one or more components (e.g., sensor system 150, I / O device 160 and other functional modules 170). In addition, the system 100 is configured to include one or more I / O devices 160 associated with the framework 200 (e.g., display device, keyboard device, speaker device, etc.). In addition, the system 100 includes other functional modules 170, such as any appropriate hardware, software, or combination thereof that helps or contributes to the operation of the framework 200. For example, the other functional modules 170 include communication technologies that enable the components of the system 100 to communicate with each other as described herein. At this point, the system 100 is operable to at least train, adopt and / or deploy a framework 200 for language-guided self-supervised semantic segmentation, as described herein.
[0029] Figure 2A , Figure 2B , Figure 2C and Figure 2D Figure 2 shows a method for language-guided self-supervised semantic segmentation (hereinafter referred to as “CLIP-S 4 ”) in various aspects of an example of a framework 200, wherein “CLIP” refers to Contrastive Language Image Pre-Training (“CLIP”), and “S 4” refers to “self-supervised semantic segmentation”. As described above, the framework 200 includes a pre-trained visual language model 210 and a machine learning model 220. One or more processors of the processing system 110 are configured to execute the framework 200.
[0030] In an example embodiment, the pre-trained visual language model 210 includes at least one CLIP model. In this regard, the framework 200 directly uses the pre-trained visual language model, which is pre-trained for the classification task. For example, in Figure 2A In the embodiment of the present invention, the framework 200 includes at least a modified image encoder 210A (e.g., MaskCLIP) of a pre-trained visual language model 210 (e.g., CLIP). The modified image encoder 210A is configured to directly extract pixel embeddings related to text from the pre-trained visual language model 210 (e.g., CLIP model). In addition, the framework 200 includes at least one text encoder 210B ( Figure 3 and Figure 5 ). The pre-trained visual language model 210 is configured to receive a training dataset of training images 10 including source images 12. The source images 12 refer to the "original images" of the training dataset and / or the training images 10 without any data augmentation (e.g., flipping, cropping, etc.). In this example, the source images 12 are digital images. As a non-limiting example, in Figure 2A , the source image 12 is a digital image showing a jet and the moon against a sky background. Figure 2A As shown in , the modified image encoder 210A is configured to generate a pixel embedding as an output in response to receiving one or more digital images (eg, source image 12) as input.
[0031] In addition, the machine learning model 220 is configured to generate pixel embeddings as outputs upon receiving one or more digital images (e.g., image 14 and image 16) as inputs. In this regard, the machine learning model 220 is configured to map image data into pixel embeddings for semantic segmentation. For example, in Figure 2A , the machine learning model 220 includes a Siamese coding network. The machine learning model 220 includes at least one encoder 220A. The encoder 220A may include a convolutional neural network (CNN), any suitable coding network, any suitable machine learning model, or any number and combination thereof.
[0032] Figure 2AAn example of a framework 200 is illustrated, which involves at least the following two components: (i) a self-supervised contrastive learning component 230 and (ii) a visual language model guided consistency component 240. Specifically, the self-supervised contrastive learning component 230 enforces pixel embeddings to be consistent within visually coherent regions of the same source image 12 and between different enhanced views (e.g., image 14 and image 16) via a first loss component 270 of the machine learning model 220. For example, in Figure 2A In the embodiment of the present invention, the self-supervised contrastive learning component 230 includes at least a first loss component 270 to achieve such contrastive consistency.
[0033] The visual language model guided consistency component 240 introduces (i) embedding consistency, via a second loss component 280 of the machine learning model 220, so that the pixel embedding generated by the machine learning model 220 is aligned with the joint feature space of text and image provided by the pre-trained visual language model 210 (e.g., CLIP), and (ii) semantic consistency, via a third loss component 290 of the machine learning model 220, so that the machine learning model 220 is forced to make the same predictions as the pre-trained visual language model 210 for a set of target categories having known category prototypes 262 and unknown category prototypes 264. For known categories, the known category prototypes 262 are pre-computed and fixed during training. For unknown categories, the unknown category prototypes 264 are learned via clustering during training. In this regard, the self-supervised contrastive learning component 230 and the visual language model guided consistency component 240 complement each other. These two components improve pixel embedding quality and enable language-driven semantic segmentation for both known categories (e.g., a set of categories that are predefined during training and used as a set of text embeddings by the framework 200) and unknown categories (e.g., another set of categories that are not predefined during training and therefore learned by the framework 200).
[0034] exist Figure 2AIn the present invention, the framework 200 segments a digital image by enabling a machine learning model 220 to learn a pixel embedding function using (i) self-supervised contrastive learning and (ii) guidance from a pre-trained visual language model 210 (e.g., CLIP). The framework 200 uses self-supervised contrastive learning to enforce consistency of pixel embeddings within visually coherent regions (e.g., superpixels) and between different enhanced views (e.g., image 14 and image 16) obtained from the same source image 12. The framework 200 introduces two visual language model-guided consistency components (i.e., embedding consistency and semantic consistency) to further regularize the machine learning model 220. The two components are complementary to each other. In addition, contrastive learning mitigates the noise introduced by the pre-trained visual language model 210 (e.g., CLIP). In addition, using the knowledge extracted from the pre-trained visual language model 210 (e.g., CLIP), the framework 200 is configured to improve the quality of pixel embeddings. The framework 200 is configured to perform language-driven semantic segmentation using target category prototypes 260 , which include known category prototypes 262 and unknown category prototypes 264 .
[0035] The framework 200 trains a pixel embedding function to generate consistent pixel embeddings in visually coherent regions through pixel segment contrast learning. Specifically, the framework 200 employs an embedding function that transforms each pixel p of a digital image into a unit-length embedding vector z of dimension d via a deep neural network (such as the machine learning model 220). p For example, in Figure 2A In , the machine learning model 220 generates a set of pixel embeddings based on the pixels of the image 14, as shown in image 22. Specifically, the encoder 220A generates the pixel embeddings as output when receiving the image 14 as input. In this regard, image 22 is provided as a visualization of the pixel embeddings that are generated by the machine learning model 220 (e.g., encoder 220A) based on the image 14. The image 14 is a modified version of the source image 12 based on the data augmentation. As a non-limiting example, in Figure 2A In the embodiment of the present invention, the image 14 is generated by at least performing a color space transform on the source image 12 so that the image 14 is a differently colored version of the source image 12.
[0036] In addition, the machine learning model 220 generates a set of pixel embeddings based on the image 16, as shown in the image 24. Specifically, the encoder 220A generates the pixel embeddings as output when receiving the image 16 as input. In this regard, the image 24 is provided as a visualization of the pixel embeddings, which are generated by the machine learning model 220 (e.g., the encoder 220A) based on the image 16. The image 16 is another modified version of the source image 12 based on the data augmentation. As a non-limiting example, in Figure 2AIn the embodiment, the image 16 is generated by at least performing a flip operation on the source image 12 , such that the image 16 is a flipped version of the source image 12 .
[0037] Framework 200 then partitions the pixel embeddings into |S| segments by clustering the pixel embeddings into sets of clusters. Figure 2A In relation to the pixel embedding shown in image 22, framework 200 generates a set of clusters as defined by the segment outline data and as shown by the cluster outlines shown in image 32. Figure 2A In , image 32 is provided as a visualization of segment contour data, which is generated based on the pixel embedding shown in image 22. In addition, with respect to the pixel embedding of image 24, framework 200 generates a set of clusters, which are defined by the segment contour data and are illustrated by the cluster contours shown in image 34. Figure 2A In FIG. 2 , image 34 is provided as a visualization of the segment contour data, which is generated based on the pixel embedding shown in image 24. In addition, framework 200 converts the embedding v of each segment s (or embedded segment) into s Calculated as the average of pixel embeddings such that v s =∑ p∈ s z p / |s|, which is also normalized to a unit length vector v s =v s / ||v s ||.
[0038] Furthermore, for each pixel p, the fragments are grouped into two sets, which include the positive set S + and negative set S - The positive set S of pixels + Segments that are within the same visually coherent region of pixels. Visually coherent regions can be derived from superpixels or contours. As described above, framework 200 uses data augmentation (e.g., random resizing, cropping, flipping, color jittering, Gaussian blurring, etc.) to generate consistent pixel embeddings between different augmented views of the same source image 12 (e.g., image 14 and image 16). Therefore, segments that are within the same visually coherent region of pixels in any augmented view are considered to be a positive set S. + Other segments in the image and other segments from other images in the same batch are included in the negative set S - middle.
[0039] Figure 2B A non-limiting example of a positive set 272 is illustrated, which includes a portion of a jet aircraft segment embedded in 44A as shown in image 44, and another portion of the same jet aircraft segment embedded in 44A as shown in image 44. In addition, Figure 2BAn example of another positive set 274 is illustrated, which includes a jet aircraft segment embedding 44A as shown in image 44 and another jet aircraft segment embedding 46A as shown in image 46 . Figure 2B Also illustrated is an example of a negative set 276 , which includes unknown segment embedding 44B as shown in image 44 and jet segment embedding 46A as shown in image 46 .
[0040] Then, the pixel embedding z p Attracted to the positive set S via contrast loss + The fragments in the negative set S - The fragment rejection in is expressed in Equation 1. More specifically, in Equation 1, κ is a concentration constant, and sim(z p ,v s ) is the pixel embedding z p and the fragment embeds v s At this point, framework 200 computes the pixel embedding z p and the fragment embeds v s The similarity data between , as expressed in Equation 1, is used to generate contrast loss data.
[0041]
[0042] In addition, if Figure 2A As shown in , the framework 200 obtains the pixel embedding of the source image 12 from the pre-trained visual language model 210 via a modified image encoder 210A (e.g., MaskCLIP). The modified image encoder 210A includes a modification of the attention-based pooling layer of the image encoder of the pre-trained visual language model 210. More specifically, the modification includes (i) removing the query projection layer and the key projection layer, and (ii) reconstructing the value projection layer and the last linear layer into two consecutive fully connected layers. As an example, for example, the framework 200 uses clip-i(·) as the modified image encoder 210A and uses clip-t(·) as the text encoder 210B. Refer back to Figure 2A , the pixel embedding generated by modified image encoder 210A and shown in image 20 is coarser and noisier than the pixel embeddings generated by encoder 220A and shown in images 22 and 24, respectively.
[0043] The framework 200 then obtains pixel embeddings generated by the pre-trained visual language model 210 for different enhanced views of the source image 12. Specifically, upon receiving the source image 12 as input, the framework 200 generates pixel embeddings via the pre-trained visual language model 210, as shown in the image 20. Figure 2A, an image 20 is provided as a visualization of pixel embeddings generated by a pre-trained visual language model 210 (e.g., a modified image encoder 210A such as MaskCLIP).
[0044] The framework 200 is configured to subsequently perform data augmentation to ensure that the pixel embeddings of the pre-trained visual language model 210 correspond to the pixel embeddings of the machine learning model 220 among the different augmented views. The framework 200 also minimizes the embedding distance between segments (not pixels) from the self-supervised and visual language embedding spaces. Figure 2A As shown in , these actions are performed on the segment embeddings, because the pixel embeddings are noisy, which is mitigated by aggregating on the segments. Therefore, the framework 200 uses the pixel embeddings generated by the machine learning model 220 to derive segments (or clusters) of visually coherent regions and generates segment contour data defining these segments (or clusters).
[0045] Framework 200 includes a segment applier 250. Segment applier 250 is configured to (i) receive data augmentation data for creating image 14 from source image 12 and segment contour data defining clusters of image 32, and (ii) generate segment embeddings of image 40 by applying the data augmentation data and the segment contour data to pixel embeddings of image 20. In addition, segment applier 250 is configured to (i) receive data augmentation data for creating image 16 from source image 12 and segment contour data defining clusters of image 34, and (ii) generate segment embeddings of image 42 by applying the data augmentation data and the segment contour data to pixel embeddings of image 20.
[0046] For each segment s, framework 200 defines the embedding consistency loss via Equation 2, where v s and i s are the segment embeddings derived from the embedding function of the machine learning model 220 and the embedding function of the pre-trained visual language model 210 (e.g., CLIP), respectively. Here, the average of the pixel embeddings from the pre-trained visual language model for segment s is i s =∑ p∈s clip-i(s) / |s|. Framework 200 generates embedding consistency loss data via Equation 2, which involves the clip embedding v via the machine learning model 220 s and the fragment embeddings of the pre-trained visual language model 210 s The cosine similarity between them is used to calculate the similarity data.
[0047]
[0048] Figure 2C2 illustrates various aspects of computing an embedding consistency loss. In this non-limiting example, the framework 200 generates embedding consistency loss data by comparing the similarity between a segment embedding 46A derived from the machine learning model 220 and a corresponding segment embedding 42A derived from the pre-trained visual language model 210. Figure 2C As shown in , the segment embedding 46A from the machine learning model 220 is of higher quality than the segment embedding 42A of the pre-trained visual language model 210. The segment embedding 46A shown in image 46 and the segment embedding 42A shown in image 42 both correspond to visually coherent regions or segments related to the known category of jet aircraft.
[0049] In addition to embedding consistency, the framework 200 provides semantic consistency by forcing the machine learning model 220 to make the same semantic category predictions as the pre-trained visual language model 210. The basic principle is that the framework 200 is configured to generate better pixel embeddings if different clusters are formed to correspond to different semantic categories, because the goal of semantic segmentation is to perform pixel-by-pixel classification. Semantic consistency is achieved via pseudo-labeling. Again, the framework 200 directly enforces semantic consistency at the segment level (not directly at the pixel level) to reduce pseudo-label y s The noise in the
[0050] Specifically, for each segment s, framework 200 generates its pseudo label y on the set of target categories using the pre-trained visual language model 210 s , the target category set includes both known categories and unknown categories. Target categories will also be discussed later in this disclosure. s A set of category prototypes 260 of target categories in the pixel embedding space of the pre-trained visual language model 210 The maximum similarity or highest similarity between them is used to generate the pseudo label y s , so that y s = argmax l∈L (sim(i s , c l )).
[0051] Then, as expressed in Equation 3, the framework 200 defines the semantic consistency loss as the prediction of the target class by the machine learning model 220 and pseudo-label y s The cross entropy between . In Equation 3, Framework 200 generates semantic consistency loss data via Equation 3.
[0052]
[0053] Figure 2D2 illustrates various aspects of computing semantic consistency loss using target category prototypes 260. More specifically, Figure 2D In the non-limiting example shown in , the framework 200 compares the similarity between the segment embedding 42A derived from the pre-trained visual language model 210 and each target category prototype 260, including the known category prototype 262 and the unknown category prototype 264. The framework 200 generates similarity data 296 via a cosine similarity function (denoted by "Sim (first feature, second feature)"). The similarity data 296 indicates a measure of how similar (or dissimilar) the segment embedding 42A is to each target category prototype 260. As described above, when the similarity value is closer to or equal to 1, the similarity between the two features is greater. In addition, when the similarity is closer to or equal to 0, the dissimilarity between the two features is greater. For example, in this non-limiting example, the framework 200 generates similarity data 296 indicating that the segment embedding 42A is more similar to the known category prototype of a jet aircraft than other known category prototypes and unknown category prototypes. The framework 200 also generates additional data 294 to locate the category with the maximum predicted probability via the argmax function. In this case, the framework 200 determines that the class prototype of jet airplane exhibits the maximum predicted probability of the segment embedding 42A compared to other class prototypes (e.g., the class prototype of cat, the class prototype of the first unknown class, the class prototype of the second unknown class, etc.). Therefore, the framework 200 uses the known class prototype of jet airplane as the pseudo label y on the target class prototype set 260. s .
[0054] In addition, the framework 200 compares the similarity between the segment embedding 46A derived from the machine learning model 220 and each target class prototype 260, including the known class prototype 262 and the unknown class prototype 264. The framework 200 generates similarity data 292 via a cosine similarity function (denoted by "Sim (first feature, second feature)"). The similarity data 292 (or similarity measure) indicates a measure of how similar (or dissimilar) the segment embedding 46A is to each target class prototype 260. When the similarity value is closer to or equal to 1, the similarity between the two features is greater. In addition, when the similarity is closer to or equal to 0, the dissimilarity between the two features is greater. For example, in this non-limiting example, the framework 200 generates similarity data indicating that the segment embedding 46A is more similar to the known class prototype of a jet aircraft than to other known class prototypes and unknown class prototypes. The framework 200 provides the known class prototype of a jet aircraft as a class prediction for the target class. The framework then uses Equation 3 and calculates the semantic consistency loss as the predicted class prediction and pseudo label y of the machine learning model 220 s The cross entropy between .
[0055] Figure 3 3 is a flow chart 300 illustrating various aspects of the target class prototype 260. Figure 3 As shown in FIG. 2 , the target category prototype 260 includes the target category prototype 260 as represented by C={C k , C u} represents the known category prototype 262 and the unknown category prototype 264, where C represents the target category prototype set 260, C k represents the known category prototype set 262, and C u Represents the unknown category prototype set 264. For the known category set 302 (e.g., jet, cat, etc.), where K = {0, ..., k-1}, the framework 200 first generates a text embedding set 306 by using the prompt template set 304 via the text encoder 210B of the pre-trained visual language model 210, T = {t k Specifically, the text encoder 210B receives the known category set as text input via the hint template 304. Then, the framework 200 calculates the text embedding 306T and all clip embeddings 308I based on the pre-trained visual language model 210 of the training image 10 via softmax. s}, and the first m similar fragment embeddings 308 of the pre-trained visual language model 210 are averaged as the category prototype of the category,
[0056] For each unknown category c u , the framework 200 randomly selects a segment embedding 308 of an image segment (e.g., i r ) as the initial prototype, c u =i r The random selection may be made with respect to a subset taken from the set of all segment embeddings 308. The subset includes segment embeddings that are not associated with the set of known categories 302. For example, i0 is associated with the known category of jet aircraft, while i3 is not associated with a known category in the set of known categories 302. In other words, the subset includes multiple segment embeddings 308 corresponding to unknown categories (e.g., currently unnamed categories associated with the placeholder category). For example, in Figure 3 In the example, framework 200 randomly selects an embedding segment i3 from the subset such that i r =i3. Instead of using only the text embedding 306 of the predefined / known category set 302, the framework 200 includes two target category sets with both known category prototypes 262 and unknown category prototypes 264, which enables language-driven category-free semantic segmentation.
[0057] Design of target categories and associated category prototypes is the key to achieving semantic consistency. Here, the category prototype c l is an embedding vector that can represent a category in an embedding space (e.g., a visual-linguistic embedding space). For example, a category prototype can be the mean vector of all segment embeddings of the category "car" (or the mean vector of all segment embeddings of the category "car"). Assuming that the target category is already predefined is not feasible in real use cases without any human annotations. Therefore, methods with these assumptions cannot handle unknown categories hidden in the data. To solve this problem, the framework 200 includes two category prototype sets 260. The first category prototype set 260 includes known category prototypes 262, C k ={c0, ..., c k-1}, which are associated with the known class set 302. The known class set 302 is predetermined and given to the framework 200 during training. The second class prototype set includes unknown class prototypes 264, C u ={c k , ..., c k+u}, which are associated with unknown categories. Unknown category prototypes 264 are associated with unknown categories (which may sometimes be referred to as "placeholder categories"). Unknown categories act as placeholders until the input data is obtained later (e.g., during inference) because the input data is not provided as the known category set 302 during training. Known category prototypes 262 are predefined by utilizing the pre-trained visual language model 210. Meanwhile, unknown category prototypes 264 are learned from segment embeddings during training. Therefore, the framework 200 has a target category prototype 260, which can be represented as Where L = k + u.
[0058] For known categories, a natural choice is to use the text embeddings (e.g., contour 406) generated by the pre-trained visual-language model 210 as their category prototype embeddings. However, even if the text embeddings (e.g., contour 406) are trained to align with the image / pixel embeddings (e.g., contour 402), there is still a huge gap between the text embeddings (e.g., contour 406) and the image / pixel embeddings (e.g., contour 402) in the joint visual-language embedding space 400, as shown in Figure 4A and Figure 4B As shown in . Therefore, when using text embeddings (such as outline 402) as category prototypes, there are some challenges in learning meaningful unknown categories from image features. Therefore, the framework 200 utilizes pixel embeddings from a pre-trained visual language model 210 to represent each known category.
[0059] To this end, for a set of known categories 302 (e.g., jet, cat, etc.), K = {0, ..., k-1}, the framework 200 first obtains a set of text embeddings 306 via a set of hint templates 304, T = {t k}={clip-t k}. For example, during training, a user (e.g., a software developer, etc.) can input the known category set 302 into a pre-trained visual language model 210 (e.g., CLIP) via a hint template 304. The framework 200 also obtains a segment embedding set I = {i s}: a) Feed the training image 10 into the modified image encoder 210A to generate pixel embeddings; b) Cluster the pixel embeddings into segments and c) for each segment The pixel embeddings in are averaged. Accordingly, the framework 200 is configured to Each fragment of the representation generates a fragment embedding.
[0060] The framework 200 (i) computes the similarity between the text embedding 306 of the known categories generated by the pre-trained visual language model 210 and the embedding 308 of all segments generated by the pre-trained visual language model 210, and (ii) normalizes these similarities across all known categories by softmax. Finally, the framework 200 averages the top m similar segment embeddings to generate a known category prototype 262 for each known category, In addition, Figure 3 In, unknown category C u The unknown category prototype 264 (e.g., embedding vector) of is randomly initialized by sampling the segment embedding 308 generated by the pre-trained visual language model 210. For example, the framework 200 performs random sampling such that where the size of the unknown class is u.
[0061] During training, the segment embedding c of each unknown class prototype 264 is updated by minimizing the distance of each unknown class prototype 264 to all segments classified as the unknown class. u (similar to updating the centroid in k-means clustering), as expressed in Equation 4. In this example, the framework 200 uses a cosine similarity function to generate the segment embedding c u and the similarity data (or distance data) between all segment embeddings (or all segments of pixel embeddings) classified as the unknown category. More specifically, in Equation 4, S urepresents a segment (or segment embedding) classified as an unknown category. In this regard, the framework 200 is configured to generate prototype loss data via Equation 4 and learn pixel representations of unknown categories. During training, the framework 200 generates prototype loss data via Equation 4 and learns pixel representations of unknown categories via Equation 4. u Update the embedding of the unknown class (or update the unknown class prototype 264).
[0062]
[0063] In summary, the framework 200 trains the pixel embedding function by combining the loss L, which is a combination of pixel segment contrastive loss data (e.g., L t ), embedding consistent loss data (e.g., L e ) and semantic consistency loss data (e.g., L s ), as indicated in Equation 5. In this regard, the framework 200 generates combined loss data by calculating the combined loss L expressed in Equation 5. In addition, the framework 200 is configured to update the parameters of the machine learning model 220 based on the combined loss data.
[0064]
[0065] For inference, the system 100 uses a trained framework 200 (e.g., a trained machine learning model 220) to generate pixel embeddings for each input image (e.g., a digital image), and uses the pixel embeddings for different downstream tasks (including language-driven and unsupervised semantic segmentation). For language-driven semantic segmentation, the framework 200 first obtains the text embedding of any inferred category by feeding the prompt engineering text into the prompt template 304 of the text encoder 210B for the pre-trained visual language model 210. The framework 200 then assigns each pixel or fragment the category name (or category label) whose text embedding is closest to the pixel embedding or fragment embedding generated by the framework 200 (e.g., the trained machine learning model 220). For unsupervised semantic segmentation, the framework 200 uses a k-nearest neighbor search or a linear classifier to perform semantic segmentation.
[0066] Figure 4A and Figure 4B The visual language embedding space 400 is illustrated, which includes two different sets of pixel embeddings and their semantics. More specifically, Figure 4A and Figure 4B The pixel embedding set of the pre-trained visual language model 210 is outlined by outline 402 and the pixel embedding set of the machine learning model 220 is outlined by outline 404. In addition, outline 402 includes, for example, C for representing a class prototype. i and C for another class prototype j For example, as a non-limiting example, Ci can represent a class prototype associated with the jet aircraft class, and C j Can represent the category prototype associated with the moon category, as discussed in other non-limiting examples of the present disclosure. At the same time, the outline 404 also includes, for example, C for representing a category prototype. i and “unknown” for representing another category prototype. In this regard, the C of the contour 402 generated from the pixel embedding of the pre-trained visual language model 210 is i C corresponding to the contour 404 generated from the pixel embedding of the machine learning model 220 i In addition, the C of the contour 402 generated from the pixel embedding of the pre-trained visual language model 210 j The “unknown” corresponds to the contour 404 generated from the pixel embedding of the machine learning model 220.
[0067] Figure 4A A non-limiting example is illustrated in which the contours 402 of the pixel embedding set of the pre-trained visual language model 210 are not aligned with the contours 404 of the pixel embedding set of the machine learning model 220. Figure 4A As shown in , when contour 402 is not aligned with contour 404, the category prototype C of the pre-trained visual language model 210 i There is no category prototype C with the machine learning model 220 in the visual language embedding space 400 i Furthermore, when contour 402 is not aligned with contour 404, the category prototype C of the pre-trained visual language model 210 j does not align with the corresponding “unknown” category prototype of the machine learning model 220 in the visual-language embedding space 400. In addition, Figure 4A and Figure 4B Illustrated is a contour 406 of a text embedding of the pre-trained visual-language model 210 , separated from the contour 402 in the visual-language embedding space 400 . Figure 4A and Figure 4B Also illustrated is that contours 406 of the text embeddings of the pre-trained visual-language model 210 are separated from contours 404 in the visual-language embedding space 400 .
[0068] As mentioned earlier in this disclosure, to achieve language-driven semantic segmentation and improve the quality of pixel embeddings, the framework 200 uses a pre-trained visual language model 210 (e.g., CLIP) to guide the training of the pixel embedding function of the machine learning model 220. The framework 200 aligns the output space of the pixel embedding function of the machine learning model 220 with the feature space of the pre-trained visual language model 210. The framework 200 is configured to guide and / or align the pixel embedding generated from the self-supervised method of the machine learning model 220 with the pixel embedding generated from the pre-trained visual language model 210. The framework 200 performs this alignment by minimizing the distance between the pixel embedding of the machine learning model 220 and the pixel embedding of the pre-trained visual language model 210 in the visual language embedding space 400.
[0069] Figure 4B A non-limiting example is illustrated in which a contour 402 of a pixel embedding of a pre-trained visual-language model 210 is aligned or at least partially aligned (or at least partially overlapped) with a contour 404 of a pixel embedding of a machine learning model 220 in a visual-language embedding space 400 via the framework 200. In this non-limiting example, due at least to the training provided by the framework 200, Figure 4B The profile 404 is greater than Figure 4A The profile 404. In addition, Figure 4B The class prototype C of the machine learning model 220 is shown i The category prototype C of the pre-trained visual language model 210 in the visual language embedding space 400 i In addition, the “unknown” category prototype of the machine learning model 220 is aligned with the category prototype C of the pre-trained visual language model 210 in the visual language embedding space 400. j Aligned or at least partially aligned (or at least partially overlapping). Figure 4A and Figure 4B As shown and discussed, the framework 200 is advantageous in at least partially aligning (or partially overlapping) pixel embeddings and their semantics (e.g., outlines 406 of text embeddings) in a visual-linguistic embedding space 400, resulting in improved embedding quality and learning of representations for unknown categories.
[0070] Figure 5500 is a flow chart of a non-limiting example of the framework 200 during inference. More specifically, during inference, the framework 200 is configured to receive a digital image 502 as input data. In this non-limiting example, the digital image 502 shows a jet 502A, a moon 502B, and a sky 502C. Upon receiving the digital image 502, the machine learning model 220 is configured to generate a pixel embedding as an output upon receiving the digital image 502 as input. The framework 200 is then configured to generate a set of segment embeddings using the pixel embeddings of the machine learning model 220, as shown in image 504. The set of segment embeddings is generated based on clustering of pixel embeddings that share similar semantic meanings or belong to similar / same visually coherent regions. For example, the set of segment embeddings includes a segment embedding 504A for a jet, a segment embedding 504B for the moon, and a plurality of segment embeddings 504C for the sky.
[0071] In addition, a user (e.g., a software developer) can input a set of categories 506 (e.g., jets, cats, moons, etc.) via the prompt template 304. In this non-limiting example, the prompt template 304 is "a photo of {category name}," where "{category name}" represents the user input (e.g., a category name such as jets). In this non-limiting example, for example, the set of categories 506 includes a jet category 506A, a moon category 506B, and a plurality of other categories (e.g., a cat category, etc.). In addition, as Figure 5 As shown in , the modified image encoder 210A (e.g., mask CLIP) is configured to generate a set of text embeddings 508 as output upon receiving a set of categories 506 as input. For example, the modified image encoder 210A is configured to generate a text embedding 508A for the jet category 506A, a text embedding 508B for the moon category 506B, and a plurality of other text embeddings for a plurality of other categories.
[0072] The framework 200 then determines the similarity of each segment embedding relative to each text embedding. As a non-limiting example, for example, the framework 200 calculates the cosine similarity between the segment embedding 504A (corresponding to the jet segment) and the text embedding 508B (corresponding to the moon category). As another non-limiting example, for example, the framework 200 calculates the cosine similarity between the segment embedding 504B (corresponding to the moon segment) and the text embedding 508B (corresponding to the moon category). Figure 5As shown in , in this case, framework 200 determines that the probability that segment embedding 504A (corresponding to the jet segment) is associated with text embedding 508B (corresponding to the moon category) is low (e.g., the similarity data is less than 0.5), and also determines that the probability that segment embedding 504B is associated with text embedding 508B (corresponding to the moon category) is high (e.g., the similarity data is greater than 0.5).
[0073] After performing the cosine similarity calculation, the framework 200 generates semantic segmentation data, which includes at least assigning a set of categories 506 to a set of image segments (e.g., mask segments), as shown in image 510. For example, upon determining that the similarity data is within a predetermined threshold range, the framework 200 assigns the jet category 506A to the image segment 510A (and the pixels including the image segment 510A). Upon determining that the similarity data is within a predetermined threshold range, the framework 200 assigns the moon category 506B to the image segment 510B (and the pixels including the image segment 510B). In addition, the framework 200 does not assign a category name from the set of categories 506 to the image segment 510C because the corresponding segment embedding 504C does not exhibit sufficient probability / correlation with the set of text embeddings 508 relative to the calculated similarity data relative to the predetermined threshold range. In such a case, the framework 200 can assign a default label (e.g., a background label) to these image segments 510C.
[0074] If the category name "sky" is input via the hint template 304 during a later inference time, the framework 200 is configured to assign the sky category to each of the plurality of image segments 510C corresponding to the sky, at least because the framework 200 learned these segments as unknown category prototypes during training. The framework 200 is able to perform this action, similar to the framework 200's ability to assign the moon category 506B to the image segment 510B during inference, even though the moon category name was not provided as input to the text encoder 210B during training ( Figure 3). In this regard, the framework 200 is advantageously configured to generate semantic segmentation data (e.g., semantic labels or category labels) during inference time from any segment of a digital image that is: (i) a known category provided to the framework 200 (e.g., the text encoder 210B) during training, or (ii) an unknown category. The unknown category can be considered a placeholder for new input data that can be received and transformed by the framework 200. For example, during inference, a new category name, such as "sky", can be appended to the category set 506. This new category name "sky" is treated as an unknown category during training and is associated with a particular unknown category prototype during training. However, the framework 200 can receive this new category name (e.g., sky) from a user during inference time and enable the framework 200 to generate semantic segmentation data (e.g., semantic labels or category labels) for all image segments 510C of the digital image 502 that correspond to the new input data (e.g., the category name of sky). In this regard, during inference time, the framework 200 is configured to (i) enable a user to input an arbitrary category name that is unknown during training, and (ii) encode the at least one arbitrary category name into a text embedding to match the most similar snippet embedding.
[0075] Figure 6 6 is a diagram of a system 600, which includes a framework 200 configured for language-guided self-supervised semantic segmentation. In this example, the system 600 includes at least a sensor system 610, a control system 620, and an actuator system 630. The system 600 is configured so that the control system 620 controls the actuator system 630 based on sensor data from the sensor system 610. More specifically, the sensor system 610 includes one or more sensors and / or corresponding devices that generate sensor data. For example, the sensor system 610 includes an image sensor, a camera, a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, a satellite-based navigation sensor (e.g., a global positioning system (GPS) sensor), an optical sensor, an audio sensor, any suitable sensor, or any number and combination thereof. When obtaining detection from the environment, the sensor system 610 is operable to communicate with the control system 620 via an input / output (I / O) system 670 and / or other functional modules 650 including communication technology.
[0076] The control system 620 is configured to obtain sensor data directly or indirectly from one or more sensors of the sensor system 610. In this regard, the sensor data may include sensor data from a single sensor or sensor fusion data from multiple sensors. Upon receiving an input including at least sensor data, the control system 620 is operable to process the sensor data via a processing system 640. In this regard, the processing system 640 includes at least one processor. For example, the processing system 640 includes an electronic processor, a CPU, a GPU, a microprocessor, an FPGA, an ASIC, a processing circuit, any suitable processing technology, or any combination thereof. When processing at least the sensor data, the processing system 640 is configured to extract, generate and / or obtain appropriate input data (e.g., digital image data) of the framework 200. In addition, the processing system 640 is operable to generate output data (e.g., semantic segmentation data about an object displayed in a digital image) via the framework 200 based on communication with the memory system 660. In addition, the processing system 640 is operable to provide actuator control data to the actuator system 630 based on the output data, the semantic segmentation data and / or the object recognition data.
[0077] The memory system 660 is a computer or electronic memory system that is configured to store various data and provide access to various data to implement at least the operations and functions disclosed herein. The memory system 660 includes a single device or multiple devices. The memory system 660 includes electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology, or any combination thereof. For example, the memory system 660 may include RAM, ROM, flash memory, disk drive, memory card, optical storage device, magnetic storage device, memory module, any suitable type of memory device, or any number and combination thereof. In an example embodiment, the memory system 660 is local, remote, or a combination thereof (e.g., partially local and partially remote) relative to the control system 620 and / or the processing system 640. For example, the memory system 660 may include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from the processing system 640 and / or other components of the control system 620.
[0078] The memory system 660 includes at least the framework 200 executed via the processing system 640. The framework 200 is configured to receive or obtain input data including at least one digital image. In addition, the framework 200 is configured via the processing system 640 to generate output data (e.g., semantic segmentation data) based on the input data (e.g., digital image). As discussed earlier, the framework 200 includes at least a pre-trained visual language model 210 and a machine learning model 220. In this example, the machine learning model 220 has been used as discussed above using the combined loss data L and the prototype loss data L uIn addition, the memory system 660 includes a computer vision application 680, which includes computer readable data including instructions for providing a plurality of computer vision services to the control system 620 using the output data of the framework 200. The computer vision application 680 works with the framework 200 to provide a plurality of computer vision services (e.g., object recognition) to the control system 620, so that the control system 620 can control the actuator system 630 according to the computer vision services. The memory system 660 is also configured to store other related data 690, which relates to the operation of the system 600 related to one or more components (e.g., the sensor system 610, the actuator system 630, etc.).
[0079] In addition, if Figure 6 As shown in , system 600 includes other components that facilitate the operation of control system 620 associated with sensor system 610 and actuator system 630. In addition, as shown in Figure 6 As shown in , the control system 620 includes an I / O system 670, which includes one or more interfaces for one or more I / O devices involved in the system 600. For example, the I / O system 670 provides at least one interface to the sensor system 610 and at least one interface to the actuator system 630. In addition, the control system 620 is configured to provide other functional modules 650, such as any suitable hardware technology, software technology, or any combination thereof that assists and / or facilitates the operation of the system 600. For example, the other functional modules 650 include operating systems and communication technologies that enable the components of the system 600 to communicate with each other as described herein. At least using Figure 6 In the configuration discussed in the examples, system 600 can be applied in various technologies.
[0080] Figure 7 is a diagram of system 600 with respect to mobile machine technology 700 according to an example embodiment. As non-limiting examples, mobile machine technology 700 includes at least a partially autonomous vehicle or robot. Figure 7 In the embodiment of the present invention, the mobile machine technology 700 is at least a partially autonomous vehicle that includes a sensor system 610. The sensor system 610 includes an optical sensor, an image sensor, a video sensor, an ultrasonic sensor, a position sensor (e.g., a GPS sensor), a radar sensor, a LIDAR sensor, any suitable sensor, or any number and combination thereof. One or more sensors may be integrated with respect to the vehicle. The sensor system 610 is configured to provide sensor data to a control system 620.
[0081] The control system 620 is configured to obtain or generate image data based on sensor data or sensor fusion data from the sensor system 610. In addition, the control system 620 is configured to pre-process the sensor data to provide input data (e.g., digital image data) in a suitable form to the framework 200. The framework 200 is advantageously configured to provide semantic segmentation data so that objects shown in the sensor data can be detected and recognized.
[0082] In addition, the control system 620 is configured to generate actuator control data based at least on the output data (e.g., semantic segmentation data, object recognition data, etc.) of the framework 200 according to the computer vision application 680. In this regard, the control system 620 is configured to generate actuator control data that allows for safer and more accurate control of the actuator system 630 of the vehicle through the improved semantic segmentation provided by the framework 200. The actuator system 630 may include any number and combination of actuators of a braking system, a propulsion system, an engine, a transmission system, a steering system, or a vehicle. The actuator system 630 is configured to control the vehicle so that the vehicle follows road rules and avoids collisions based at least on the output data (e.g., semantic segmentation data) generated by the framework 200 in response to receiving one or more digital images based on the sensor data.
[0083] Figure 8 800 is a diagram of a system 600 according to an example embodiment with respect to a security technology 800. As a non-limiting example, the security technology 800 includes at least a monitoring system, a control access system, a surveillance system, or any suitable type of security device. For example, as an example, Figure 8 Involving security technology 800, it is configured to physically control the locked state and unlocked state of the lock of door 802 and display enhanced images / video on display technology 804. Security technology 800 can also trigger an alarm and / or provide electronic notifications to other communication devices / technologies. In this example, sensor system 610 includes at least an image sensor configured to provide image / video data. Sensor system 610 can also include other sensors, such as motion sensors, infrared sensors, etc.
[0084] The control system 620 is configured to obtain image / video data from the sensor system 610. The control system 620 is also configured to generate semantic segmentation data via the framework 200 based on the image / video data obtained from the sensor system 610. In addition, the control system 620 is configured to generate actuator control data by using the output data (e.g., semantic segmentation data) generated by the framework 200, which allows for safer and more accurate control of the actuator system 630. The control system 620 is configured to display the output data of the framework 200, or any data related to the computer vision application 680, or any number and combination thereof, on the display technology 804.
[0085] Fig. 9 is a diagram of system 600 with respect to imaging technology 900 according to an example embodiment. As non-limiting examples, imaging technology 900 includes a magnetic resonance imaging (MRI) device, an x-ray imaging device, an ultrasound device, a medical imaging device, any suitable type of imaging device, or any number and combination thereof. Fig. 9 , the sensor system 610 includes at least one image sensor. The control system 620 is configured to obtain image data from the sensor system 610. Via the framework 200, the control system 620 is also configured to generate semantic segmentation data based on the sensor data (e.g., image data). In addition, the control system 620 is configured to provide semantic segmentation data and object detection / recognition data about the image data of the sensor system 610. In addition, the control system 620 is configured to display any relevant data (e.g., sensor data, output data of the framework 200, relevant data from the computer vision application 680, or any number and combination thereof) on the display 902.
[0086] As described above, embodiments are advantageous and provide multiple benefits. For example, framework 200 provides a novel pixel representation learning method for semantic segmentation. Framework 200 combines self-supervised contrastive learning of machine learning model 220 and guidance from pre-trained visual language model 210 (e.g., CLIP) to learn consistent pixel embeddings about visual and conceptual semantics. Framework 200 extends self-supervised contrastive learning from images to pixels by attracting each pixel embedded to its positive pairs and rejecting its negative pairs.
[0087] In addition, the framework 200 facilitates the implementation of a new task, namely, category-free semantic segmentation. This new task does not require any manual annotations, and even assumes that no category names are given during training. This is more challenging than recent work (e.g., MaskCLIP+) that requires category names for both known and unknown categories. For example, compared to MaskCLIP+, for the new task, the framework 200 assumes that no unknown category names are given during training. In addition, the pixel embeddings of MaskCLIP+ are distorted by predefined categories, which limits the ability of MaskCLIP+ to recognize the unknown. In addition, MaskCLIP+ requires unknown category information during training, which hinders real-world applications.
[0088] In addition, the framework 200 exhibits consistent and significant gains over other unsupervised and language-driven semantic segmentation methods on at least four datasets: Pascal VOC 2012, Pascal Context and COCO-Stuff, and DAVIS 2017. In addition, the framework 200 significantly outperforms the prior art on segmentation of unknown categories. In addition, the framework 200 is configured to recognize unknown categories during inference time without knowing any unknown category information during training time, and also learns fine-grained and clearer pixel embeddings with self-supervision.
[0089] That is, the above description is intended to be illustrative and not restrictive, and is provided in the context of a specific application and its requirements. It will be appreciated by those skilled in the art from the foregoing description that the present invention can be implemented in various forms, and various embodiments can be implemented individually or in combination. Therefore, although embodiments of the present invention have been described in conjunction with specific examples of the present invention, the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and / or methods of the present invention is not limited to the embodiments shown and described, because various modifications will become apparent to skilled practitioners when studying the drawings, the specification, and the following claims. Additionally or alternatively, components and functions can be separated or combined in a manner different from the various described embodiments, and can be described using different terms. These and other variations, modifications, additions, and improvements may fall within the scope of the present disclosure as defined in the appended claims.
Claims
1. A computer-implemented method for language-guided self-supervised semantic segmentation, the method comprising: receiving a source image; generating a modified image by performing data augmentation on the source image, the modified image being a modified version of the source image; generating, via the machine learning model, a first pixel embedding based on the modified image; generating a first fragment embedding using the first pixel embedding; generating a second pixel embedding based on the source image via a pre-trained visual language model; generating a second segment embedding by applying segment contour data of the first segment embedding to corresponding clusters of the second pixel embedding after performing data augmentation on the second pixel embedding; generating embedding consistency loss data by at least comparing the first segment embedding to the second segment embedding; generating combined loss data including embedded consistent loss data; updating the parameters of the machine learning model based on the combined loss data; and A framework for language-guided self-supervised semantic segmentation is provided, which includes (i) a pre-trained visual-language model and (ii) a machine learning model after its parameters have been updated.
2. The computer-implemented method of claim 1 , further comprising: generating another modified image by performing another data augmentation on the source image, the another modified image being another modified version of the source image; generating, via a machine learning model, a third pixel embedding based on the other modified image; generating a third fragment embedding using the third pixel embedding; and generating contrastive loss data by calculating similarity data of the first pixel embedding relative to the first segment embedding and the third segment embedding, Wherein the combined loss data further comprises comparative loss data.
3. The computer-implemented method of claim 1 , further comprising: establishing a set of category prototypes, the set of category prototypes comprising at least (i) a first category prototype associated with a predetermined category in which a category name is given to the framework during training, and (ii) a second category prototype associated with a placeholder category; generating a pseudo label for a specific second segment embedding among the second segment embeddings, wherein the pseudo label is defined by a specific category prototype having a maximum similarity with the specific second segment embedding, wherein the specific category prototype is within the category prototype set; generating a class prediction for a particular first segment embedding, the class prediction selected from a set of class prototypes based on a maximum similarity measure; and The semantic consistency loss data is generated by calculating the cross entropy of the category prediction relative to the pseudo label, The combined loss data further includes semantic consistency loss data.
4. The computer-implemented method of claim 3, wherein: The first category prototype is an average of a collection of various segment embeddings determined to be most similar to the text embedding of the category name; and The second category prototype is (i) initialized by a random segment embedding selected from the second segment embeddings and not associated with a predetermined category, and (ii) updated by minimizing the distance data between the random segment embedding and a set of other segment embeddings classified as belonging to the placeholder category.
5. The computer-implemented method of claim 1 , wherein: The pre-trained visual language model includes another machine learning model trained to perform image classification using open vocabulary categories; and The machine learning model includes at least one encoder.
6. The computer-implemented method of claim 1, wherein each first segment embedding is generated by computing the average of a cluster of first pixel embeddings associated with the same visually coherent region.
7. The computer-implemented method of claim 1 , further comprising: receiving a digital image; The framework is employed in computer vision applications to identify objects displayed in digital images; and An actuator is controlled based at least on the identified object.
8. A system for language-guided self-supervised semantic segmentation, the system comprising: one or more processors; at least one non-transitory computer-readable medium in data communication with the one or more processors, the at least one non-transitory computer-readable medium having stored thereon computer-readable data including instructions that, when executed by the one or more processors, are configured to cause the one or more processors to perform a method comprising: receiving a source image; generating a modified image by performing data augmentation on the source image, the modified image being a modified version of the source image; generating, via the machine learning model, a first pixel embedding based on the modified image; generating a first fragment embedding using the first pixel embedding; generating a second pixel embedding based on the source image via a pre-trained visual language model; generating a second segment embedding by applying segment contour data of the first segment embedding to corresponding clusters of the second pixel embedding after performing data augmentation on the second pixel embedding; generating embedding consistency loss data by at least comparing the first segment embedding to the second segment embedding; generating combined loss data including embedded consistent loss data; updating the parameters of the machine learning model based on the combined loss data; and A framework for language-guided self-supervised semantic segmentation is provided, which includes (i) a pre-trained visual-language model and (ii) a machine learning model after its parameters have been updated.
9. The system of claim 8, wherein the method further comprises: generating another modified image by performing another data augmentation on the source image, the another modified image being another modified version of the source image; generating, via a machine learning model, a third pixel embedding based on the other modified image; generating a third fragment embedding using the third pixel embedding; and generating contrastive loss data by calculating similarity data of the first pixel embedding relative to the first segment embedding and the third segment embedding, Wherein the combined loss data further comprises comparative loss data.
10. The system of claim 8, wherein the method further comprises: establishing a set of category prototypes, the set of category prototypes comprising at least (i) a first category prototype associated with a predetermined category in which a category name is given to the framework during training, and (ii) a second category prototype associated with a placeholder category; generating a pseudo label for a specific second segment embedding among the second segment embeddings, wherein the pseudo label is defined by a specific category prototype having a maximum similarity with the specific second segment embedding, wherein the specific category prototype is within the category prototype set; generating a class prediction for a particular first segment embedding, the class prediction selected from a set of class prototypes based on a maximum similarity measure; and The semantic consistency loss data is generated by calculating the cross entropy of the category prediction relative to the pseudo label, The combined loss data further includes semantic consistency loss data.
11. The system of claim 10, wherein: The first category prototype is an average of a collection of various segment embeddings determined to be most similar to the text embedding of the category name; and The second category prototype is (i) initialized by a random segment embedding selected from the second segment embeddings and not associated with a predetermined category, and (ii) updated by minimizing the distance data between the random segment embedding and a set of other segment embeddings classified as belonging to the placeholder category.
12. The system of claim 8, wherein: The pre-trained visual language model includes another machine learning model trained to perform image classification using open vocabulary categories; and The machine learning model includes at least one encoder.
13. The system of claim 8, wherein each first segment embedding is generated by computing the average of a cluster of first pixel embeddings associated with the same visually coherent region.
14. The system of claim 8, further comprising: receiving a digital image; The framework is employed in computer vision applications to identify objects displayed in digital images; and An actuator is controlled based at least on the identified object.
15. A non-transitory computer readable medium having stored thereon computer readable data including instructions executable by one or more processors to perform a method comprising: receiving a source image; generating a modified image by performing data augmentation on the source image, the modified image being a modified version of the source image; generating, via the machine learning model, a first pixel embedding based on the modified image; generating a first fragment embedding using the first pixel embedding; generating a second pixel embedding based on the source image via a pre-trained visual language model; generating a second segment embedding by applying segment contour data of the first segment embedding to corresponding clusters of the second pixel embedding after performing data augmentation on the second pixel embedding; generating embedding consistency loss data by at least comparing the first segment embedding to the second segment embedding; generating combined loss data including embedded consistent loss data; updating the parameters of the machine learning model based on the combined loss data; and A framework for language-guided self-supervised semantic segmentation is provided, which includes (i) a pre-trained visual-language model and (ii) a machine learning model after its parameters have been updated.
16. The non-transitory computer readable medium of claim 15, wherein the method further comprises: generating another modified image by performing another data augmentation on the source image, the another modified image being another modified version of the source image; generating, via a machine learning model, a third pixel embedding based on the other modified image; generating a third fragment embedding using the third pixel embedding; and generating contrastive loss data by calculating similarity data of the first pixel embedding relative to the first segment embedding and the third segment embedding, Wherein the combined loss data further comprises comparative loss data.
17. The non-transitory computer readable medium of claim 15, wherein the method further comprises: establishing a set of category prototypes, the set of category prototypes comprising at least (i) a first category prototype associated with a predetermined category in which a category name is given to the framework during training, and (ii) a second category prototype associated with a placeholder category; generating a pseudo label for a specific second segment embedding among the second segment embeddings, wherein the pseudo label is defined by a specific category prototype having a maximum similarity with the specific second segment embedding, wherein the specific category prototype is within the category prototype set; generating a class prediction for a particular first segment embedding, the class prediction selected from a set of class prototypes based on a maximum similarity measure; and The semantic consistency loss data is generated by calculating the cross entropy of the category prediction relative to the pseudo label, The combined loss data further includes semantic consistency loss data.
18. The non-transitory computer readable medium of claim 17, wherein: The first category prototype is an average of a collection of various segment embeddings determined to be most similar to the text embedding of the category name; and The second category prototype is (i) initialized by a random segment embedding selected from the second segment embeddings and not associated with a predetermined category, and (ii) updated by minimizing the distance data between the random segment embedding and a set of other segment embeddings classified as belonging to the placeholder category.
19. The non-transitory computer readable medium of claim 15, wherein: The pre-trained visual language model includes another machine learning model trained to perform image classification using open vocabulary categories; and The machine learning model includes at least one encoder.
20. The non-transitory computer-readable medium of claim 15, wherein each first segment embedding is generated by computing an average of a cluster of first pixel embeddings associated with the same visually coherent region.
Citation Information
Cited By
Image classification method and system based on visual language prompt optimization
CN121280773A
