Feature adjustment output transformer for generalizable semantic segmentation

Abstract

Systems and techniques for processing data (e.g., image data) are provided. For example, according to some aspects of the present disclosure, a method may include receiving, at a transformer of a machine learning system, a learnable query, keys and values obtained from a feature map of a segmentation model of the machine learning system. The method may also include learning, via the transformer, a mapping between unsupervised and supervised outputs of the segmentation model based on the feature map.

Description

Technical Field

[0001] The present disclosure generally relates to image processing. For example, aspects of the present disclosure relate to systems and techniques for providing a feature-conditioned output transformer that learns a mapping between unsupervised and supervised outputs on top of existing segmentation models. Background Art

[0002] The increased versatility of digital camera products has allowed digital cameras to be integrated into a wide variety of devices and has expanded their use to different applications. For example, phones, drones, cars, computers, televisions, and many other devices today are often equipped with camera devices. Camera devices allow users to capture images and / or videos from any system. Images and / or videos can be captured for recreational use, professional photography, surveillance and automation, and other applications. In addition, camera devices are increasingly equipped with specific functions for modifying images or creating artistic effects on images. For example, many camera devices are equipped with image processing capabilities for generating different effects on captured images.

[0003] Many image processing techniques rely on image segmentation algorithms that divide an image into segments that can be analyzed or processed to identify objects, produce specific image effects, etc. Some example practical applications of image segmentation include chroma key synthesis, feature extraction, object detection, recognition tasks (e.g., object recognition, facial recognition, etc.), image stylization, machine vision, medical imaging, and depth of field (or "bokeh") effects, etc. However, camera devices and image segmentation techniques often produce poor and inconsistent results.

[0004] Some systems utilize machine learning models to perform semantic segmentation on images. Such systems may be referred to as adaptive semantic segmentation systems. Many adaptive semantic segmentation systems utilizing machine learning models require the model to be adapted to an online stream of unlabeled target domain data on the device, which requires a large amount of computing resources and memory. Many machine learning models are developed using powerful computing systems with limited amounts of training data and then deployed on user devices (e.g., personal computers, laptops, mobile devices, etc.) that do not have the same high-level computing resources. Such devices may not have the ability to perform on-device training (e.g., via backpropagation as part of the training process) for machine learning models. Summary of the invention

[0005] In some examples, systems and techniques are described for providing a transformer machine learning model on top of an existing semantic segmentation model. The transformer is a deep learning model that employs the concept of self-attention, which involves differentially weighting the importance of each part of the input data. The transformer is designed to process sequential input data. However, the attention mechanism provides context for any input in the input sequence, which enables the ability to perform parallel processing on the input data. The transformer can be trained to learn the mapping between unsupervised output and supervised output by conditioning on intermediate feature maps from the semantic segmentation model.

[0006] According to at least one example, a processor-implemented method for processing data (e.g., image data) is provided. The method may include: receiving a learnable query at a transformer of a machine learning system, a key and a value obtained from a feature map of a segmentation model of the machine learning system; and learning a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0007] In another example, an apparatus for processing data is provided. The apparatus may include: at least one memory configured to store features associated with a transformer of a machine learning system; and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory and configured to: receive a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system via the transformer of the machine learning system; and learn a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0008] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: receive, via a transformer of a machine learning system, a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system; and learn, via the transformer, a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map.

[0009] In another example, an apparatus is provided. The apparatus may include a component for receiving a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system via a transformer of the machine learning system; and a component for learning a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0010] In some aspects, one or more of the devices described herein are, may be part of, and / or may include a mobile device or wireless communication device (e.g., a mobile phone or so-called "smartphone" or other mobile device), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle or a computing system, device, or component of a vehicle, a wearable device (e.g., a network-connected watch or other wearable device), a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other system or device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the device described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the device, the state of the device (eg, tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0011] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0012] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0014] Figure 1A illustrates example implementations of a system on a chip (SoC) according to some examples;

[0015] Figure 1B An example framework for obtaining a pre-trained image semantic segmentation model and then using the model during test time according to some examples is illustrated;

[0016] Figure 2A An example framework for learning a transformer configured in connection with an existing segmentation model and learning a mapping between unsupervised and supervised outputs by conditioning on intermediate feature maps according to some examples is illustrated;

[0017] Figure 2B illustrates a method for test-time adaptation with photometric consistency loss and geometric consistency loss according to some examples;

[0018] Figure 3A The transformation between supervised output and unsupervised output according to some examples is illustrated;

[0019] Figure 3B Various objectives, such as supervised objectives and unsupervised objectives, according to some examples are illustrated;

[0020] Figure 4 is a flowchart illustrating an example of a process for training a transformer and generating a transfer matrix according to some examples; and

[0021] Figure 5 is a block diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0022] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0023] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0024] In consumer and professional environments, the demand and consumption of image and video data have increased significantly. Many devices and systems are equipped with the ability to capture and process image and video data. For example, a camera or a computing device including a camera (e.g., a mobile phone or smart phone including one or more cameras) can capture a video and / or image of a scene, a person, an object, etc. The image and / or video can be captured and processed and output (and / or stored) for consumption. The image and / or video can be further processed for certain effects, such as compression, frame rate up-conversion, sharpening, color space conversion, image enhancement, high dynamic range (HDR), denoising, low light compensation, etc., without restriction. The image and / or video can also be further processed for certain applications, such as computer vision, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), and autonomous driving, etc. In some examples, one or more image or video artificial intelligence (AI) models can be used to process images and / or videos, which models may include but are not limited to AI quality enhancement and AI augmentation models.

[0025] Image and video processing operations may be computationally intensive. In some cases, as the resolution of the input image or video data frame increases (e.g., as the number of pixels to be processed per input image or video data frame increases), the image and video processing operations may become increasingly computationally intensive. For example, a video data frame having a 4K resolution may include approximately four times as many individual pixels as a video data frame having a full HD (e.g., 1080p) resolution. In some examples, the image and video processing operations may be performed by processing each pixel individually. In some examples, one or more machine learning models may be used to perform the image and video processing operations to derive a mapping from input image data (e.g., raw image data captured by one or more image sensors) to a final output image.

[0026] For example, one or more machine learning models (e.g., deep learning systems) may be used to derive a mapping between raw image data including color values ​​for each pixel position and a final output image. The final output image may include processed image data derived from the raw image data (e.g., based on a mapping learned by one or more machine learning models). In some examples, the one or more machine learning models may include a neural network (e.g., a convolutional neural network (CNN)) of convolutional filters for image and / or video processing tasks. For example, an image processing neural network may include an input layer, a plurality of hidden layers, and an output layer. The input layer may include raw image data from one or more image sensors. The hidden layer may include a convolutional filter that may be applied to the input data, or to the output from a previous hidden layer, to generate a feature map. The filter of the hidden layer may include weights for indicating the importance of the nodes of the filter. In some cases, the neural network may have a series of many hidden layers, in which the early layers determine the simple and low-level characteristics of the raw image input data, and the later layers establish a hierarchical structure of more complex and abstract characteristics. The neural network may then generate the final output image (e.g., constituting the output layer) based on the determined high-level features.

[0027] When tested on data with a similar distribution to the training data used to train the deep learning system, the deep learning system can produce state-of-the-art results. However, whenever there is a large distribution shift between the training data and the test data, the performance of the deep learning system can begin to degrade. For example, this phenomenon is evident when the deep learning system is readily deployed on real-world environments, such as a deep learning system for an autonomous vehicle that is trained to operate in one city but then moved to a new city, or a deep learning system for an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device) that is trained to operate in one location and then moved to a new location.

[0028] Various devices (e.g., autonomous vehicles, XR devices, etc.) may perform the task of semantic segmentation to classify the pixels of an image into different classes (e.g., people, sky, buildings, streets, etc.). Semantic segmentation may perform dense prediction or labeling pixels for better scene understanding. Because semantic segmentation may be a dense prediction task, labeling a dataset for training and / or testing may be a laborious task. In some cases, a semantic segmentation model may be pre-trained on a synthetic dataset, where labels may be obtained (e.g., from a game engine). Once the model is pre-trained on a synthetic dataset, the model may be deployed in a real-world target environment, provided that the distribution shift is taken into account by using domain adaptation methods. These domain adaptation methods may assume that there are a large number of unlabeled images from the target environment. Therefore, the pre-trained semantic segmentation model may be fine-tuned multiple times on a large number of unlabeled target domain images and also on labeled source domain images (if present) to produce a domain invariant model. Such domain invariant segmentation models may produce much better pixel accuracy than models that do not perform any adaptation on unlabeled target domain data.

[0029] Although the above-mentioned domain adaptation methods can provide improvements in image segmentation performance, the performance of such methods may be overestimated relative to more realistic real-world adaptation scenarios. For example, in current domain adaptive segmentation methods, there may be a set-aside unlabeled target domain split for offline adaptation and a separate split for testing. In real-world situations, access to a large number of unlabeled images may not be possible before evaluation can begin. For example, an XR device, personal robot, or other device or system that attempts to segment a scene when deployed in a new environment is immediately exposed to an online unsupervised image stream. Semantic segmentation models deployed by such devices need to adapt to the scenes depicted in the online image stream and also need to provide accurate predictions for the online image stream.

[0030] Current domain adaptation methods may not be directly applicable to arrangements for online model adaptation of a single test image at a time. For example, the forward and backward passes of a single image through a pre-trained segmentation model may produce noise losses and gradients, and therefore unreliable predictions. In addition, adaptive semantic segmentation systems typically operate (e.g., are trained to operate) using batches of unlabeled target domain data as input, and cannot handle online streaming data. For example, test time adaptive (TTA) semantic segmentation considers the adaptation of a source pre-trained image semantic segmentation model to unlabeled test images from a new target distribution. However, most TTA methods for semantic segmentation consider adaptation to batches of target data distributions, which is contrary to the real-world situation where samples from novel environments arrive one by one in an online manner. In addition, adaptive semantic segmentation systems typically use pseudo-labels for supervision. For example, a machine learning system (e.g., a neural network) may process a batch of unlabeled images and output one or more segmentation masks for the batch of unlabeled images. Pseudo-labels may be generated based on one or more segmentation masks, which may be used to further train the machine learning system. However, pseudo-labels may be inherently noisy and may therefore provide incorrect supervision for training a machine learning system (e.g., when performing supervised learning or training). In addition, as described above, some devices do not have the ability to perform training (e.g., lack the ability to perform back-propagation) due to limited computing power or memory storage limitations.

[0031] In some aspects, described herein are systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for providing a transformer machine learning model on top of an existing semantic segmentation model (e.g., one or more neural networks). The transformer machine learning model can be trained to learn a mapping (e.g., a transformation) between an unsupervised output and a supervised output of a semantic segmentation model, such as by adjusting the transformer machine learning model based on an intermediate feature map from the semantic segmentation model. In some cases, during test time adaptation, the transformation maps the unsupervised segmentation to a more reliable supervised segmentation mask. The system can operate in an inference mode (also referred to as test time inference) in which the supervised output is used for prediction. The system can also operate in an adaptation mode (also referred to as test time adaptation) in which a loss is determined based on the unsupervised output. The loss can be used to perform backpropagation to update the semantic segmentation model, and then perform inference using the supervised output for prediction.

[0032] Generally speaking, a transformer machine learning model is a deep learning model that incorporates the concept of self-attention, which involves differentially weighting the importance of each part of the input data and / or also the recursive output data. The transformer machine learning model can process sequential input data, and can process the entire input or batches of inputs together. The self-attention mechanism provides context for any position in the input sequence. This approach unlocks the ability to perform more parallel processing and enables systems to perform training that otherwise may not have the computational resources to perform training in a traditional manner.

[0033] In some cases, the system and technology may include operations performed at both the pre-training stage and the adaptation stage. For example, in the pre-training stage, the system and technology may obtain a model that can serve as an initialization for adaptation and thus produce better generalization. As described above, the system described herein can learn a generalizable model by using a feature-adjusted transformer model to map from unsupervised prediction to supervised prediction. For example, the transformer model can perform cross-attention on the learnable query and the key and value obtained from the token of the intermediate feature map (e.g., the feature map output by one or more hidden layers of the semantic segmentation model). This operation generates a transfer weight matrix that is multiplied with the unsupervised head output to produce a supervised output. The transformer model can be trained by implementing a supervised loss target for the supervised head and an unsupervised loss for the unsupervised head. The supervised mask may depend at least in part on the unsupervised segmentation mask, which is different from the multi-task learning setting in which the head can be related only by a shared feature backbone and therefore competes during training. During the test time adaptation of the image, the unsupervised loss can be back-propagated through the unsupervised head. For prediction, a supervised head can be used.

[0034] The systems and techniques described herein can enable a semantic segmentation system to generate better predictions during inference. For example, adding a transformer model on top of an existing (e.g., pre-trained) segmentation model can make the segmentation model more generalizable. Better generalizable models can be critical for deployment on devices with hardware that do not support model adaptation to test data. For example, in some cases, a device that implements the system and techniques that may not have the resources to perform full backpropagation to train a full segmentation model can perform a certain degree of adaptation of the semantic segmentation model on the device. Transformer models can be used because transformer models can capture global context, which can be important for correlating different classes of unsupervised and supervised outputs. In addition to transformers, other types of models, such as fully connected layer networks, can also be used.

[0035] Various aspects of the disclosure will be described with respect to the figures.

[0036] Figure 1AAn example implementation of a system on a chip (SOC) 100 is illustrated, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed among multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0037] The SOC 100 may also include additional processing blocks tailored for specific functions, such as a GPU 104, a DSP 106, a connectivity block 110 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 112 that may, for example, detect and recognize gestures. In one specific implementation, the NPU is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SOC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system. In some examples, the sensor processor 114 may be associated with or connected to one or more sensors for providing sensor input to the sensor processor 114. For example, the one or more sensors and the sensor processor 114 may be provided in, coupled to, or otherwise associated with the same computing device.

[0038] SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, the instructions loaded into CPU 102 may include a code for searching for a stored multiplication result corresponding to the multiplication product of the input value and the filter weight in a lookup table (LUT). The instructions loaded into CPU 102 may also include a code for disabling a multiplier during the multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into CPU 102 may include a code for storing the calculated multiplication product of the input value and the filter weight when the lookup table of the multiplication product is missing. SOC 100 and / or its components may be configured to perform image processing using machine learning techniques according to various aspects of the present disclosure discussed herein. For example, SOC 100 and / or its components may be configured to perform semantic image segmentation and / or object detection according to various aspects of the present disclosure.

[0039] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without using explicit instructions. An example of an ML system is a neural network (also called an artificial neural network), which can include a group of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in various applications and / or devices, such as image and / or video decoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0040] Each node in a neural network can simulate a biological neuron by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​restrict the way in which the input data is related to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be added. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the result, thereby generating an output signal or "output activation" (sometimes called a feature map or activation map) of the node. The weight values ​​can initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during the training phase when the network learns how to identify a specific class by its typical input data characteristics).

[0041] There are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multi-layer perceptron (MLP) neural networks, transformer neural networks, and the like. For example, a convolutional neural network (CNN) is a feed-forward artificial neural network. A convolutional neural network may include a collection of artificial neurons that each have a receptive field (e.g., a spatially local region of an input space) and collectively tile an input space. RNNs work on the principle of saving the output of a layer and feeding that output back to the input to help predict the results of that layer. GANs are generative neural networks that learn patterns in input data so that a neural network model can generate new synthetic outputs that are reasonably likely to come from the original data set. GANs may include two neural networks operating together, including a generative neural network that generates a synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data may be fed to an input layer, and one or more hidden layers provide a level of abstraction for the data. Predictions may then be made to the output layer based on the abstract data.

[0042] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL methods are based on neural networks, such as RNN or CNN, and utilize multiple layers. The use of multiple layers in a deep neural network allows for the gradual extraction of higher-level features from a given raw data input. For example, the output of the first layer of artificial neurons becomes the input of the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of the third layer of artificial neurons, and so on. The layer located between the input and output of the entire deep neural network is generally referred to as a hidden layer. Hidden layer learning (e.g., being trained) transforms the intermediate input from the previous layer into a slightly more abstract and complex representation that can be provided to subsequent layers until the final or desired representation is obtained as the final output of the deep neural network.

[0043] As mentioned above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of the neural network may include a feature map or activation map, which may include artificial neurons (or nodes). The feature map may include filters, kernels, etc. A node may include one or more weights for indicating the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, where early layers are used to determine simple and low-level characteristics of the input, and later layers establish a hierarchical structure of more complex and abstract characteristics.

[0044] Deep learning architectures can learn hierarchies of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0045] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0046] Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are sequentially delivered to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of a high-level concept can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0047] Figure 1B1 is a diagram illustrating an example framework 130 including an initial set of training data 132 for training a convolutional encoder-decoder 134 to perform semantic segmentation (during a training time 138). For example, once trained, the convolutional encoder-decoder 134 may produce output classifications or output data 136. As shown, the output data 136 may include a segmentation map. The segmentation map may include values ​​(e.g., pixels or other data) that identify (e.g., classify) different objects or regions of an image (such as people, sky, vehicles, roads, buildings, etc.). The convolutional encoder-decoder 134 may be pre-trained and then used for testing. At a later point in time (after the convolutional encoder-decoder 134 is trained to perform semantic segmentation), a series of images 140 without annotations or labels (e.g., received at times T=1, T=2, and T=3) may be received at different times during a test time 146 by a pre-trained image semantic segmentation model 142 (e.g., a pre-trained convolutional encoder-decoder trained to perform semantic segmentation). The pre-trained image semantic segmentation model 142 may be adapted or further trained using the series of images 140 .

[0048] As described above, described herein are systems and techniques that can implement new pre-training strategies for an image semantic segmentation model (e.g., image semantic segmentation model 142) that can generalize or better adapt to an online unsupervised stream of test images and produce more accurate predictions without the need for using annotations in additional training data.

[0049] Figure 2A 200. As shown, an input image 202 may be provided to a pre-trained image semantic segmentation model or feature encoder 204. The input image 202 may include unlabeled image data (e.g., without labels for supervised learning). In some cases, other types of input data may be provided as input to the system 200.

[0050] The following is a description of an example task to be performed by the disclosed system. Consider labeling source domain data The availability of the input image And the corresponding segmentation map Here, H and W are the height and width of the input image. L may represent the number of class labels. This labeled source domain data may be used to pre-train the model M. For online test-time adaptation, the model M may be adapted to the incoming unlabeled image sequence from the target domain. The sequence of frames is not necessarily temporally related (e.g., the frames in the sequence may not be adjacent frames of the video). Figure 2AAs shown, the model can be further decomposed into sub-networks, such as a feature extractor F204 (also called a feature encoder), a classifier or prediction head H208, and an optional transformer T214.

[0051] Figure 2A The process of pre-training in the case of using a transformer model is illustrated. The segmentation network 203 includes a feature encoder 204 (also referred to as a feature extractor F) and a classifier / prediction head 208. The feature encoder 204 can generate a feature map 206. The feature map 206 is provided to the classifier / prediction head 208. The output of the classifier / prediction head 208 is an unsupervised predicted segmentation mask 209, from which an unsupervised loss can be generated.

[0052] The feature map 206 is also provided to a mapper 212, which generates data such as keys and values ​​associated with the feature map 206. For example, the mapper 212 may convert the feature map 206 into a data patch (which may be, for example, in an n×n matrix), and may flatten the data patch into a one-dimensional vector or other representation of the data that may be provided to a transformer 214. Operations may be performed on the one-dimensional vector to produce keys and values, which are passed to the transformer 214. A query 216 is also provided to the transformer 214. In some examples, the query 216 may also be generated by performing one or more operations on the one-dimensional vector generated by the mapper 212.

[0053] An example of a key and value may be associated with a feature map of size 56×56×256. These values ​​may represent a structure having a width of 56 pixels, a height of 56 pixels, and 256 channels. The system 200 may convert or transform this initial data into 4 patches, which may each have a size of 14×14×256 (4×14=56 for each of the width and height of the structure). The patches are then converted to a one-dimensional vector or other data structure.

[0054] Keys and values ​​are generated or mapped using different operations (e.g., projections). For example, a token multiplied by a linear projection configured for a key will produce a set of keys. Another token or the same token multiplied by a linear projection for a value will produce a set of values. The transformation of feature maps to keys and values ​​can be performed in the mapper 212.

[0055] like Figure 2A The query 216 shown may be a learnable object or parameter. In some cases, the query 216 may be input to a mapper (not shown), which is also processed via a projection configured for the query 216. The transformer 214 performs multiple operations on the query, key, and value as disclosed herein, such as a multi-head attention operation.

[0056] In some cases, a fully connected layer (not shown) may be used in addition to or in place of transformer 214. Transformer 214 may be advantageous because it provides global context in its operation. The use of the two heads (outputs) of segmentation network 203 in conjunction with transformer 214 enables system 200 to determine the relationship between supervised features (e.g., supervised log odds) and unsupervised features (e.g., unsupervised log odds). Log odds are defined more fully below. Using global context to generate transformations via transformer 214 improves the ability of system 200 to find relationships between classes of images.

[0057] Figure 2B An example system 250 is illustrated showing test time adaptation with photometric consistency loss 260 and geometric consistency loss 258 associated with photometric enhancement and geometric enhancement 252. System 250 illustrates Figure 2A An example of test time adaptation of the segmentation network 203 of the system 200. For example, Figure 2B The segmentation model 254 is Figure 2A Example of a segmentation network 203 of FIG. For adaptation on unlabeled image sequences, the system 250 may use two types of augmentation - photometric augmentation and geometric augmentation 252 to optimize the augmentation consistency loss. Such augmentations may be used in unsupervised segmentation to cluster features and produce a good initial representation. For photometric transformations, augmentations such as grayscale conversion, color jittering, Gaussian blurring, etc. may be used. When these types of transformations are applied to the input image, the segmentation map may be no different than when no transformation is applied to the input image. This property may be referred to as invariance with respect to photometric transformations. However, neural networks are highly sensitive to changes in the input and may produce different segmentation maps when presented with photometric transformations, and are therefore not naturally invariant to photometric transformations.

[0058] The systems and techniques described herein can minimize the output prediction difference between the original input and its photometric enhancement. For example, consider a test image x and a sampled photometric enhancement A p (·), when applying photometric enhancement to the test image x, the system 250 may produce an enhancement For the original input test image x and its enhancement Both, segmentation model 254 (e.g., Figure 2A The segmentation network 203) generates unsupervised output log probability o u =H(F(x)) and In order to minimize o u and The difference between the two, system 250 can use the difference loss or consistency loss term Possible differences may be the L1 distance, the L2 distance, the divergence between probabilities obtained from the log-odds (eg, the Kullback-Leibler divergence or the KL divergence), any combination thereof, and / or other metrics.

[0059] For geometric transformations, enhancements such as random cropping, rotation, pixel shuffling, etc. may be performed. When these enhancements are applied to the input image, the segmentation map may have the same transformation as the segmentation map of the original input image. This property of geometric transformations may be referred to as equivariance. The segmentation model 254 may not necessarily be equivariant to geometric transformations.

[0060] The systems and techniques described herein can enforce this equivariance by minimizing the difference between the transformed segmentation map of the input image and the output prediction of the segmentation map of the transformed image. For example, consider a test image x and a sampled geometric enhancement A g (·), when applying geometric enhancement to the test image, the system 250 may produce an enhanced image For the original input test image x and its enhancement Both, system 250 respectively produces unsupervised output log probability o u =H(F(x)) and To implement equivariance, system 250 can be implemented by using a difference loss or consistency loss term To minimize o u and the transformed log-odds The difference between . This difference can be the same as the difference used for the photometric enhancement consistency loss. To adapt the model to the test sample x, the system 250 uses a combination of the photometric enhancement consistency loss and the geometric enhancement consistency loss as follows:

[0061]

[0062] Once you use L TTA (x) to update the model, the system 250 can use equation (5) to reason about the output predictions of the supervised head. In some cases, during test-time adaptation, the back-propagated gradients through the unsupervised head do not affect the transformer module, and thus the transformer module remains frozen throughout. When the transformer module is not used, the model has a single head. Output log probability o u and then L TTA (x) Processing by a single head after adaptation, and performing inference by only a single head. In some aspects, the pre-training and adaptation process or framework can be summarized as follows:

[0063] Given: Source dataset and

[0064] Target dataset sequence

[0065] Step 1: Pre-trained model

[0066] For those from For each sampled batch:

[0067] For each sample

[0068] About parameters F,H p ,T takes the gradient descent step of equation (4);

[0069] Step 2: Adaptation and evaluation

[0070] For those from Each sample

[0071] About parameters F and H p Take the gradient descent step of equation (5);

[0072] Use equation (3) to predict Segmentation diagram.

[0073] In some aspects, for the segmentation network 203, a pre-trained checkpoint of DeepLab-V2 ResNet-101 can be used. For example, a cross entropy loss can be used to train the ResNet-101 network on the corresponding source dataset. For the disclosed transformer 214, a 1-layer transformer decoder (e.g., without positional encoding) can be used. In some cases, an example default intermediate feature for conditioning the transformer 214 can be the output of a block of the backbone. The transformer 214 can be trained on top of the segmentation network 203 using the loss defined in equation (4). Unless explicitly mentioned, λ=0.1 and maximum squares can be set as unsupervised losses. For the enhanced consistency loss in equation (5), the L2 distance metric can be used as L p (·) and L g (·) The default metric for both. For adaptation, stochastic gradient descent (SGD) can be used with a learning rate of 1e-4. In some cases, only the batch normalization parameters can be updated for one iteration per sample.

[0074] Figure 3A An example operation 300 of the transformer 214 is illustrated. The transformer 214 may receive the query 216 and the tokenized key and value, and may use a transformer decoder (TransDec) operation to generate an affine feature 302 (denoted as "F"). The "P" in the example operation 300 refers to Figure 2A Linear projection 218 (in Figure 2AEquations (3) and (4) generally involve TransDec operation 302. Affine features F and projections P (e.g., Figure 2A The linear projection 218 in ( ) can be operated by the Softmax operation 4 (or other operations) to generate a transfer matrix W, such as Figure 3A As shown. Linear projection P218 can be a linear projection or a linear weighting of the data. Generating transfer matrix W 304 corresponds to equation (5). Transfer matrix 220 is also Figure 2A Shown in.

[0075] The output network 210 of the prediction head 208 includes an unsupervised prediction segmentation mask 209 and may include an unsupervised loss. The transfer matrix 220 is calculated by Figure 3A Equation 306 in shows that in this equation there is a supervised log probability L sup Equal to the unsupervised log probability L unsup Multiply by the transfer matrix W 220. The log odds are the inverse of the standard logistic function and may be referred to as log odds because they are essentially equal to the logarithm of the odds p / (1-p), where p is the probability. The log odds refer to features (e.g., from one or more feature maps) output by one or more layers of the model before the normalization layer (e.g., to be input to a Softmax layer such as Figure 1B Softmax layer shown in ). Those skilled in the art will understand the use of log odds in machine learning and data transformation, and will be able to utilize the log odds associated with the unsupervised prediction segmentation mask 209 and the supervised segmentation mask 226. The use of log odds is one example of data that can be used in conjunction with the introduction of the transformer 214 and associated components for providing an inference mode to the segmentation network 203.

[0076] The segmentation network 203 can be a multi-layer network. For example, there can be 101 layers in the segmentation network 203. In some examples, the classifier / prediction head 208 can be the last two layers. Thus, the feature map 206 can represent data at the state of a particular layer of the segmentation network 203. In another example, the feature map 206 can represent data after layer 99 of the segmentation network 203, and the prediction head 208 can represent the last two layers of the network 203 before the network 203 produces an unsupervised prediction segmentation mask 209. The segmentation network 203 can also be characterized as a segmentation model or more generally any model that can include a segmentation model or any other machine learning model of a different type that can be used in the same manner as described herein.

[0077] Multiplication by Figure 2AThe multiplier 222 in 220 represents. The transfer matrix W 220 is related to determining how the classes in different data sets are related to each other between two different types of outputs. The benefit of using the transformer 214 is to obtain a class-to-class relationship to generate a supervised output. The general framework 224 of the output of the multiplier 222 includes a supervised segmentation mask 226 (also referred to as a supervised prediction segmentation mask 226) and a ground truth (GT) mask 228, and a supervised loss represents the comparison of these two masks. The transformer 214 takes a learnable query 216 and the key and value obtained from the feature map 206 processed by the mapper 212 of the segmentation model. The input query can be the number of classes x the number of channels, which is 256 in the example of this article. Other sizes of input queries or other data structures are also applicable. The transformer 214 output is also the type of query, which can be the same size, where the number of classes is the same, but different values ​​can exist. Keys and values ​​can also be obtained from feature maps of different depths and even from the output.

[0078] As described above, transformer 214 generates a transfer matrix 220, which is then multiplied by the log odds of output network 210 to generate a new set of log odds as described above.

[0079] The old set of log-likelihoods is trained using the unsupervised loss shown with respect to output network 210, while the new set of log-likelihoods is trained using the supervised loss shown with respect to architecture 224. This training strategy forces transformer 214 to learn the transformation between unsupervised prediction mask 209 and supervised segmentation mask 226.

[0080] In some aspects, at test time, the transformer 214 remains frozen. The transformer 214 may remain frozen while the unsupervised output or unsupervised loss shown in the output network 210 is back-propagated through the segmentation network 203 for further training of the segmentation network 203. The manner in which the transformer 214 receives data from the segmentation network 203 prevents the back-propagated data from reaching or affecting the transformer 214.

[0081] During the test time inference mode, the system 200 can use the supervised segmentation mask 226 for prediction. In this case, no adaptation is required. The use of the transformer 214 improves the performance of the system 200. In the inference mode, the image 202 is passed through the segmentation network 203 without adaptation, and the system 200 uses the supervised segmentation mask 226 as output. The system 200 does not use the unsupervised prediction mask 209 as a format output, but in this scenario, the unsupervised prediction mask 209 is used in conjunction with the transfer matrix 220 to generate the supervised segmentation mask 226. In this case, the supervised segmentation mask 226 is better than the unsupervised prediction mask 209.

[0082] The value of P in this example may be 256 x the number of classes. As shown in equation 304, the output of transformer 214 multiplied by P will be the number of classes multiplied by the number of classes. In this regard, transfer matrix 220 is structured to have an appropriate size for the relationship between supervised log odds and unsupervised log odds.

[0083] Another mode may be test time adaptation. In this case, the system 200 may use the unsupervised prediction mask 209 and loss shown in the output network 210 to update the segmentation network 203 by back propagation. After the segmentation network 203 is updated, the system 200 may transition back to the inference mode, in which the inference uses the supervised segmentation mask 226 and loss of the transformer 214. Figure 2A Again, in this scenario, the unsupervised prediction mask 209 is not used as a preferred output, but is used in conjunction with the transfer matrix 220 to generate a supervised segmentation mask 226.

[0084] In adaptation mode, when the unsupervised loss shown in the output network 210 is used to further refine or train the segmentation network 203, the second head (output) associated with the transformer 214 is configured via the use of a forward pass so that back propagation is not propagated to the transformer 214. In other words, the transformer 214 is frozen in one aspect because the back propagation of the unsupervised loss only affects or further trains the segmentation network 203 and not the transformer 214.

[0085] Once the parameters on the segmentation network 203 have been updated using the unsupervised loss, the inference process may proceed as outlined above, where the predicted output of the system is selected as the supervised segmentation mask 226 via use of the unsupervised output used in conjunction with the transfer matrix 220 .

[0086] As described herein, systems and techniques are provided for performing image processing using one or more image processing machine learning networks that enforce consistency across at least one variable of unlabeled data applied to a pre-trained model. Figure 3B Various objectives 310 used in the system and technology are illustrated. For example, a supervised objective 312 can be used to minimize the cross entropy loss. The supervised objective 312 is used in conjunction with the supervised segmentation mask 226 (which may also be referred to as a supervised head). Several unsupervised objectives 314 can be applied with respect to the unsupervised prediction mask 209 to minimize entropy or maximize probability squared. Various options for unsupervised objectives can be applied to the unsupervised prediction mask 209. Equation (6) below in the present disclosure illustrates the use of these loss equations. The methods disclosed herein can be used with any pre-trained segmentation network and backbone, which allows the system and technology to evolve with the existing technology.

[0087] As described above, the systems and techniques described herein can determine the relationship between supervised semantic segmentation maps and unsupervised semantic segmentation maps. For example, an input image can be fed forward through the feature extractor F (e.g., Figure 2A feature encoder 204) to generate a feature map So that f i =F(x i ). The feature map f can be fed into a classifier or prediction head (denoted as H P ) (e.g., prediction head 208) to generate output log odds Make o i =H P (f i ). These log odds may then be passed through a Softmax operation (or other similar operation) to obtain a probability map for the segmentation (eg, the segmentation prediction mask 209)

[0088] For F and H P , a system (e.g., system 200) may use a prediction probability map p i with the ground truth segmentation label y i The cross entropy loss between However, according to the systems and techniques described herein, the system can be designed to learn a relationship between supervised and unsupervised predictions that can facilitate test-time adaptation from unsupervised image sequences. The annotation of the output log-odds can be obtained from i Change to The system can then use the feature map f i As used for (for example, Figure 2A The transformer decoder module of the transformer 214 adjusts the input to construct the key and value (e.g., about Figure 2A The transformer decoder can use learnable queries (For example, Figure 2A 216), the learnable query is a C-dimensional vector representation of the category for the supervised output identification. As described in this article, in order to generate keys and values, from the feature map f i Get the patch. Flatten the patch to produce n tokens These tokens t i It is then fed into the multi-head attention stage of the transformer decoder, followed by the feed-forward network. To understand the multi-head attention scheme, the single-head attention mechanism can be defined as follows:

[0089] q= q s W q ,k i = t i Wk ,v i = t i W v (1)

[0090] Where W q , W k , is a weight matrix to produce a linear representation of the original token. These processed tokens can then be used to produce the attention operation Att(·) such that:

[0091]

[0092] For a multi-head attention operation MHAtt(·) with M heads, q s and t i Split into M parts and t i,1 ,...,t i,M , where the size of each split is C′=C / M. The attention operation is applied to all such splits to produce:

[0093]

[0094] where LN is the layer normalization operation, DO is the dropout operation, and is a linear mapping. The output of the multi-head attention mechanism can be passed through a two-layer feed-forward network, where each layer includes a rectified linear unit (ReLU) activation, followed by dropout, residual connection, and layer normalization (e.g., similar to equation (4) above). The alternating multi-head attention and feed-forward network can produce multiple layers of transformer 214. The output of transformer 214 can be directed to the transformer decoder input q s Generate representation Thus, a C-dimensional vector representation of each of the classes is included, adjusted according to the feature map of the input image. This representation can then be mapped to an L-dimensional space via the linear prediction 218 of the system 200 to produce a transfer matrix 220 (e.g., a weight matrix) associated with the supervised log-odds and the unsupervised log-odds. Thus, the following operations can be applied:

[0095]

[0096] in is the projection layer and These log-probabilities can then be passed through a Softmax operation (or other similar operations) to obtain a segmentation probability map For F and H P Traditional end-to-end training, the system can use the prediction probability map with the ground truth segmentation label y i The cross entropy loss between To train the unsupervised log-odds The system can use unsupervised loss The unsupervised loss can be one of the losses used for test-time adaptation, such as minimum entropy, maximum square, or the enhanced consistency loss described herein (e.g., L1 loss, L2 loss, KL divergence, etc.). Therefore, the system can use a total loss, such as the following total loss, to train the feature extractor F (e.g., Figure 2A Feature encoder 204), classifier head H P (For example, Figure 2A The prediction head 208) and the transformer module T (eg, Figure 2A The entire semantic segmentation network of transformer 214) is:

[0097]

[0098] By minimizing this training loss with source domain data, the system can learn the mapping between unsupervised and supervised segmentation predictions. During online test-time adaptation of sample x, the transformer module can remain frozen and the system can use the output o of the unsupervised head u To obtain the unsupervised loss L to be used to update the model parameters TTA (o u ). After adaptation is complete, the system can use the supervised head to output o s to achieve the purpose of evaluation.

[0099] As described herein, the framework of the disclosed systems and techniques provides online test-time adaptive semantic segmentation. The technique includes training a transformer 214 to map unsupervised predictions to supervised predictions. The system and technique also proposes enhanced consistency as a fine-tuning goal to adapt the model to unlabeled target domain data. The proposed system and technique outperform other competing methods both quantitatively and qualitatively. The design and ablation of the framework improve segmentation performance.

[0100] Figure 4 A process 400 for generating output from a machine learning model is illustrated in which an inference mode and an adaptation mode may be used. At block 402, process 400 may include: in a transformer (e.g., Figure 2A The system 200 may receive a learnable query, a key and a value obtained from a feature map of a segmentation model of a machine learning system at a transformer 214 of the system 200. For example, the learnable query may include query 216, and the key and the value may include information about Figure 2A those discussed.

[0101] At block 404, process 400 may include: Figure 2A The transformer 214 of the system 200 learns the unsupervised output of the segmentation model based on the feature map (e.g., Figure 2A The unsupervised predicted segmentation mask 209 of Figure 2A The mapping between the supervised segmentation mask 226).

[0102] In some aspects, in order to learn the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map, the process 400 may include generating a transfer matrix (e.g., the transfer matrix 220) via a transformer. The process 400 may also include multiplying the transfer matrix with the unsupervised output of the segmentation model (e.g., via multiplier 222) to produce a supervised output. In some cases, the transformer is configured to apply affine features and linear predictions (e.g., from linear projection 218) to generate a transfer matrix (e.g., the transfer matrix 220). For example, the affine features may be based on a query, a key, a value, a linear projection, or any combination thereof. In some examples, as previously described, a token multiplied by a linear projection configured for a key may produce a set of keys, another token or the same token multiplied by a linear projection for a value will produce a set of values, etc. In some examples, the keys and values ​​are tokenized.

[0103] In some aspects, process 400 may include training a first feature set associated with an unsupervised output of a segmentation model using an unsupervised loss. Process 400 may include training a second feature set associated with a supervised output using a supervised loss. In some cases, the first feature set includes a first log-odds set and the second feature set includes a second log-odds set. In some examples, during an inference mode (or test time inference) of a machine learning system, process 400 may use supervised outputs based on supervised losses for prediction. In some examples, during an adaptation mode (or test time adaptation) of a machine learning system, process 400 may use unsupervised outputs and unsupervised losses to update a segmentation model to produce an adapted segmentation model. In some cases, process 400 may include freezing a transformer at test time in the adaptation mode. In some aspects, process 400 may include receiving additional data; and processing the additional data using an adapted segmentation model to generate additional supervised outputs.

[0104] As described herein, the systems and techniques described herein can learn a transformation between two heads (e.g., two different outputs of the segmentation network 203) that can give more insight into how supervised and unsupervised learning occurs. In addition, as described above, there is flexibility as to whether the system needs to adapt the segmentation network 203, depending on the capabilities of the device on which the segmentation network 203 is deployed.

[0105] In another example, a device for processing data is provided. The device (e.g., Figure 5 The computing device architecture 500 shown in FIG. 5 may include at least one memory (e.g., Figure 5 515, 520, 535 and / or 512), wherein at least one memory is configured to store a transformer (e.g., Figure 2A and at least one processor (e.g., implemented in circuitry) coupled to at least one memory and configured to: receive a learnable query via a transformer of a machine learning system, extract a segmentation model (e.g., Figure 2A and learning a mapping between an unsupervised output and a supervised output of a segmentation model based on the feature map via a transformer.

[0106] In another example, a non-transitory computer-readable medium having instructions stored thereon (eg, Figure 5 515, 520, 535 and / or 512 in memory), which instructions, when executed by one or more processors, configure the one or more processors to: via a transformer of a machine learning system (e.g., Figure 2A The transformer 214 of the system 200 receives a learnable query, a segmentation model from a machine learning system (e.g., Figure 2A and learning a mapping between an unsupervised output and a supervised output of a segmentation model based on the feature map via a transformer.

[0107] In another example, an apparatus (eg, Figure 5 ). The apparatus may include components for receiving, via a transformer of a machine learning system, a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system; and components for learning, via the transformer, a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map.

[0108] Examples of uses of the disclosed solution may include self-driving vehicles. Most city segmentation models are trained on clean real-world datasets or simulated game datasets. The solution disclosed herein can be used for online adaptation to real-world scenarios, where accurate segmentation masks along with depth will allow the vehicle to take appropriate control actions, such as speed control, steering, braking.

[0109] For extended reality use cases, indoor segmentation is required for use cases such as human occlusion rendering and semantic reconstruction. The indoor environment on which the segmentation model is trained may have different characteristics from the environment where it is deployed. There may be changes in layout, brightness, etc. In such cases, online adaptation is important to encounter domain shifts of the pre-training dataset and the test dataset.

[0110] Robotics is another field where this technology can be useful. Accurate semantic segmentation will enable multiple capabilities in robots, such as navigation, localization, and interaction with physical objects in the environment. In order to address domain shift characteristics between the source and target datasets, such as changes in object characteristics, environmental changes like lighting, etc.

[0111] In some examples, the processes described herein (e.g., process 400 and / or any other process described herein) can be performed by a computing device, apparatus, or system. In some examples, process 400 can be performed by a computer having Figure 5 The computing device or system of the computing device architecture 500 of the present invention may be executed. The computing device, apparatus or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a laptop computer, a smart TV, a camera and / or any other computing device with resource capabilities to perform the processes described herein (including process 400 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to convey and / or receive data, any combination thereof and / or other components. The network interface may be configured to convey and / or receive data based on an Internet Protocol (IP) or other types of data.

[0112] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0113] Process 400 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0114] Additionally, process 400 and / or any other process described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0115] Figure 5An example computing device architecture 500 of an example computing device that can implement the various techniques described herein is illustrated. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other device. The components of the computing device architecture 500 are shown as being in electrical communication with each other using a connection 505 (such as a bus). The example computing device architecture 500 includes a processing unit (CPU or processor) 510 and a computing device connection 505 that couples various computing device components including a computing device memory 515 (such as a read-only memory (ROM) 520 and a random access memory (RAM) 525) to the processor 510.

[0116] The computing device architecture 500 may include a cache of high-speed memory directly connected to the processor 510, very close to the processor, or integrated as part of the processor. The computing device architecture 500 may copy data from the memory 515 and / or the storage device 530 to the cache 512 for fast access by the processor 510. In this way, the cache can provide a performance improvement that avoids the processor 510 from delaying when waiting for data. These engines and other engines can control or be configured to control the processor 510 to perform various actions. Other computing device memories 515 can also be used. The memory 515 may include a variety of different types of memory with different performance characteristics. The processor 510 may include any general-purpose processor and hardware or software services (such as service 1 532, service 2 534, and service 3 536 stored in the storage device 530) configured to control the processor 510, as well as a special-purpose processor in which software instructions are incorporated into the processor design. The processor 510 may be a self-contained system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0117] To enable user interaction with the computing device architecture 500, the input device 545 may represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 535 may also be one or more of a plurality of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some cases, a multi-modal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 500. The communication interface 540 may generally govern and manage user input and computing device output. There is no restriction on operating on any particular hardware arrangement, and thus the underlying features herein may be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0118] The storage device 530 is a non-volatile memory and may be a hard disk or other type of computer readable medium that can store computer accessible data, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette tape, a random access memory (RAM) 525, a read-only memory (ROM) 520, and hybrids thereof. The storage device 530 may include services 532, 534, 536 for controlling the processor 510. Other hardware or software modules or engines are contemplated. The storage device 530 may be connected to the computing device connection 505. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer readable medium connected to the necessary hardware components (such as the processor 510, the connection 505, the output device 535, etc.) to perform the function.

[0119] Aspects of the present disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. Although described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to a particular device.

[0120] The term "device" is not limited to one or a specific number of physical objects (such as a smart phone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. In addition, the term "system" is not limited to multiple components or specific aspects. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static components. Although the following description and examples use the term "system" to describe various aspects of the present disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

[0121] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks including devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0122] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many of the operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0123] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that cause a general purpose computer, a special purpose computer, or a processing device or otherwise configure them to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary files, intermediate format instructions (such as assembly language), firmware, source code, etc.

[0124] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as flash memory), memory or memory devices, disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, compact discs (CDs) or digital versatile discs (DVDs), any suitable combination thereof, etc. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, engine, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0125] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0126] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0127] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0128] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be embodied and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in an order different from the described order.

[0129] One of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0130] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0131] The phrase "coupled to" refers to any component being physically connected directly or indirectly to another component, and / or any component being in direct or indirect communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0132] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0133] Claim language or other language reciting "at least one processor configured to," "at least one processor configured to," etc., indicates that one processor or multiple processors (in any combination) may perform the associated operations. For example, claim language reciting "at least one processor configured to: X, Y, and Z" means that a single processor may be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a particular subset of operations X, Y, and Z such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting "at least one processor configured to: X, Y, and Z" may mean that any single processor may perform only at least a subset of operations X, Y, and Z.

[0134] Various exemplary logic blocks, modules, engines, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0135] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0136] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0137] The following provides exemplary aspects of the present disclosure:

[0138] Aspect 1. A processor-implemented method for processing data, comprising: receiving a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system at a transformer of the machine learning system; and learning a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0139] Aspect 2. A processor-implemented method according to Aspect 1, wherein learning the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map includes: generating a transfer matrix via the transformer; and multiplying the transfer matrix with the unsupervised output of the segmentation model to produce the supervised output.

[0140] Aspect 3. The processor-implemented method according to aspect 2, wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix.

[0141] Aspect 4. The processor-implemented method of aspect 3, wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

[0142] Aspect 5. The processor-implemented method of aspect 4, wherein the key and the value are tokenized.

[0143] Aspect 6. The processor-implemented method according to any one of Aspects 1 to 5 further includes: using an unsupervised loss to train a first set of features associated with the unsupervised output of the segmentation model; and using a supervised loss to train a second set of features associated with the supervised output.

[0144] Aspect 7. The processor-implemented method of aspect 6, wherein the first feature set comprises a first set of log-odds, and wherein the second feature set comprises a second set of log-odds.

[0145] Aspect 8. The processor-implemented method according to any one of Aspects 6 or 7 further includes: during an inference mode of the machine learning system, using the supervised output based on the supervised loss to make predictions.

[0146] Aspect 9. The processor-implemented method according to any one of Aspects 6 or 7 further includes: during an adaptation mode of the machine learning system, using the unsupervised output and the unsupervised loss to update the segmentation model to produce an adapted segmentation model.

[0147] Aspect 10. The processor-implemented method of aspect 9, further comprising: freezing the converter at test time in the adaptation mode.

[0148] Aspect 11. The processor-implemented method according to any one of Aspects 9 or 10, further comprising: receiving additional data; and processing the additional data using the adapted segmentation model to generate additional supervised outputs.

[0149] Aspect 12. A device for processing data, the device comprising: at least one memory, the at least one memory being configured to store features associated with a transformer of a machine learning system; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: receive a learnable query, keys and values ​​obtained from a feature map of a segmentation model of the machine learning system via the transformer of the machine learning system; and learn a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0150] Aspect 13. An apparatus according to Aspect 12, wherein learning the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map includes: generating a transfer matrix via the transformer; and multiplying the transfer matrix with the unsupervised output of the segmentation model to produce the supervised output.

[0151] Aspect 14. The apparatus according to aspect 13, wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix.

[0152] Clause 15. The apparatus according to clause 14, wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

[0153] Aspect 16. The apparatus according to aspect 15, wherein the key and the value are tokenized.

[0154] Aspect 17. The apparatus according to any one of Aspects 12 to 16 further includes: using an unsupervised loss to train a first set of features associated with the unsupervised output of the segmentation model; and using a supervised loss to train a second set of features associated with the supervised output.

[0155] Aspect 18. An apparatus according to aspect 17, wherein the first feature set comprises a first set of log-odds, and wherein the second feature set comprises a second set of log-odds.

[0156] Aspect 19. An apparatus according to any one of Aspects 18 or 19, further comprising: during an inference mode of the machine learning system, using the supervised output based on the supervised loss to make predictions.

[0157] Aspect 20. The apparatus according to any one of Aspects 18 or 19 further comprises: during an adaptation mode of the machine learning system, using the unsupervised output and the unsupervised loss to update the segmentation model to produce an adapted segmentation model.

[0158] Aspect 21. The apparatus according to aspect 20, further comprising: freezing the converter at a test time in the adaptation mode.

[0159] Aspect 22. The apparatus according to any one of Aspects 20 or 21, further comprising: receiving additional data; and processing the additional data using the adapted segmentation model to generate additional supervised output.

[0160] Aspect 23. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor, configure the at least one processor to: receive a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system via a transformer of the machine learning system; and learn a mapping between an unsupervised output and a supervised output of the segmentation model based on the feature map via the transformer.

[0161] Aspect 24. A non-transitory computer-readable medium according to Aspect 23, wherein in order to learn the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map, the non-transitory computer-readable medium stores additional instructions, and the additional instructions cause the at least one processor to be further configured to: generate a transfer matrix via the transformer; and multiply the transfer matrix with the unsupervised output of the segmentation model to produce the supervised output.

[0162] Aspect 25. The non-transitory computer readable medium of aspect 24, wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix.

[0163] Aspect 26. The non-transitory computer-readable medium of aspect 25, wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

[0164] Aspect 27. The non-transitory computer-readable medium of aspect 26, wherein the key and the value are tokenized.

[0165] Aspect 28. A non-transitory computer-readable medium according to any one of Aspects 23 to 27, wherein the non-transitory computer-readable medium stores additional instructions that cause the at least one processor to be further configured to: train a first set of features associated with the unsupervised output of the segmentation model using an unsupervised loss; and train a second set of features associated with the supervised output using a supervised loss.

[0166] Aspect 29. The non-transitory computer-readable medium of aspect 28, wherein the first feature set comprises a first set of log-odds, and wherein the second feature set comprises a second set of log-odds.

[0167] Aspect 30. A non-transitory computer-readable medium according to any one of Aspects 28 or 29, wherein the non-transitory computer-readable medium stores additional instructions, wherein the additional instructions cause the at least one processor to be further configured to: make predictions using the supervised output based on the supervised loss during the inference mode of the machine learning system.

[0168] Aspect 31. A non-transitory computer-readable medium according to any one of Aspects 29 or 30, wherein the non-transitory computer-readable medium stores additional instructions, wherein the additional instructions cause the at least one processor to be further configured to: during an adaptation mode of the machine learning system, use the unsupervised output and the unsupervised loss to update the segmentation model to produce an adapted segmentation model.

[0169] Aspect 32. The non-transitory computer readable medium of aspect 31, wherein the non-transitory computer readable medium stores additional instructions that cause the at least one processor to be further configured to: freeze the converter at test time in the adaptation mode.

[0170] Aspect 33. A non-transitory computer-readable medium according to any one of Aspects 31 or 32, wherein the non-transitory computer-readable medium stores additional instructions, wherein the additional instructions cause the at least one processor to be further configured to: receive additional data; and process the additional data using the adapted segmentation model to generate additional supervised outputs.

[0171] Aspect 34. An apparatus comprising one or more components for performing the operations or steps according to any one of aspects 1 to 12.

Claims

1. A method implemented by a processor for processing data, comprising: receiving, at a transformer of a machine learning system, a learnable query, a key and a value obtained from a feature graph of a segmentation model of the machine learning system; as well as A mapping between an unsupervised output and a supervised output of the segmentation model is learned based on the feature map via the transformer.

2. The processor-implemented method of claim 1 , wherein learning the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map comprises: generating a transfer matrix via the transformer; as well as The transfer matrix is ​​multiplied by the unsupervised output of the segmentation model to produce the supervised output. 3 . The processor-implemented method of claim 2 , wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix. 4 . The processor-implemented method of claim 3 , wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

5. The processor-implemented method of claim 4, wherein the key and the value are tokenized.

6. The processor-implemented method of claim 1 , further comprising: training a first set of features associated with the unsupervised output of the segmentation model using an unsupervised loss; as well as A second set of features associated with the supervised output is trained using a supervised loss.

7. The processor-implemented method of claim 6, wherein the first feature set comprises a first log-odds set, and wherein the second feature set comprises a second log-odds set.

8. The processor-implemented method of claim 6, further comprising: During an inference mode of the machine learning system, predictions are made using the supervised output based on the supervised loss.

9. The processor-implemented method of claim 6, further comprising: During an adaptation mode of the machine learning system, the segmentation model is updated using the unsupervised output and the unsupervised loss to produce an adapted segmentation model.

10. The processor-implemented method of claim 9, further comprising: In the adaptation mode the converter is frozen at test time.

11. The processor-implemented method of claim 9, further comprising: receiving additional data; as well as The additional data is processed using the adapted segmentation model to generate additional supervised outputs.

12. A device for processing data, the device comprising: at least one memory configured to store features associated with a transformer of a machine learning system; and at least one processor coupled to the at least one memory and configured to: receiving, via the transformer of the machine learning system, a learnable query, a key and a value obtained from a feature graph of a segmentation model of the machine learning system; as well as A mapping between an unsupervised output and a supervised output of the segmentation model is learned based on the feature map via the transformer.

13. The apparatus of claim 12, wherein to learn the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map, the at least one processor is configured to: generating a transfer matrix via the transformer; and The transfer matrix is ​​multiplied by the unsupervised output of the segmentation model to produce the supervised output.

14. The apparatus of claim 13, wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix.

15. The device of claim 14, wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

16. The apparatus of claim 15, wherein the key and the value are tokenized.

17. The apparatus of claim 12, wherein the at least one processor is configured to: training a first set of features associated with the unsupervised output of the segmentation model using an unsupervised loss; and A second set of features associated with the supervised output is trained using a supervised loss.

18. The apparatus of claim 17, wherein the first feature set comprises a first log-odds set, and wherein the second feature set comprises a second log-odds set.

19. The apparatus of claim 17, wherein the at least one processor is configured to: During an inference mode of the machine learning system, predictions are made using the supervised output based on the supervised loss.

20. The apparatus of claim 18, wherein the at least one processor is configured to: During an adaptation mode of the machine learning system, the segmentation model is updated using the unsupervised output and the unsupervised loss to produce an adapted segmentation model.

21. The apparatus of claim 20, wherein the at least one processor is configured to: In the adaptation mode the converter is frozen at test time.

22. The apparatus of claim 20, wherein the at least one processor is configured to: receiving additional data; and The additional data is processed using the adapted segmentation model to generate additional supervised outputs.

23. A non-transitory computer readable medium having stored thereon instructions which, when executed by at least one processor, cause the at least one processor to be configured to: receiving, via a transformer of a machine learning system, a learnable query, a key and a value obtained from a feature map of a segmentation model of the machine learning system; and A mapping between an unsupervised output and a supervised output of the segmentation model is learned based on the feature map via the transformer.

24. The non-transitory computer-readable medium of claim 23, wherein to learn the mapping between the unsupervised output and the supervised output of the segmentation model based on the feature map, the non-transitory computer-readable medium stores further instructions that cause the at least one processor to be further configured to: generating a transfer matrix via the transformer; and The transfer matrix is ​​multiplied by the unsupervised output of the segmentation model to produce the supervised output.

25. The non-transitory computer readable medium of claim 24, wherein the transformer is configured to apply affine features and linear prediction to generate the transfer matrix.

26. The non-transitory computer readable medium of claim 25, wherein the affine feature is based on at least one of a query, a key, a value, or a linear projection.

27. The non-transitory computer-readable medium of claim 26, wherein the key and the value are tokenized.

28. The non-transitory computer readable medium of claim 23, wherein the non-transitory computer readable medium stores additional instructions that cause the at least one processor to be further configured to: training a first set of features associated with the unsupervised output of the segmentation model using an unsupervised loss; and A second set of features associated with the supervised output is trained using a supervised loss.

29. The non-transitory computer readable medium of claim 28, wherein the first feature set comprises a first log-odds set, and wherein the second feature set comprises a second log-odds set.

30. The non-transitory computer readable medium of claim 28, wherein the non-transitory computer readable medium stores additional instructions that cause the at least one processor to be further configured to: During an inference mode of the machine learning system, predictions are made using the supervised output based on the supervised loss.

Images