Robotic system and method for determing relationships between items in an image

Abstract

Implementations process an image using an image encoder to generate encoder output tokens that can each represent a corresponding candidate item / object in the image. The encoder output tokens that are generated using the image encoder can be processed to determine both: (i) classifications of the items in the image and (ii) relationships between the items in the image. The encoder output tokens can also optionally be utilized to determine bounding boxes and / or other localizations that define a corresponding region of the image for each of the items. The determined classifications of the candidate items and the relationships between the items can be utilized for various purposes, such as constructing a scene graph where the items and their classifications are represented as nodes and the relationships between the items are represented as edges.

Description

ROBOTIC SYSTEM AND METHOD FOR DETERMINING RELATIONSHIPS BETWEEN ITEMS IN AN IMAGEBackground

[0001] A goal of computer vision is to decompose visual scenes into structured semantic representations. A task towards this goal is object detection, in which items in an image are localized by bounding boxes and classified into semantic categories. For example, a first item can be classified as a book and localized by a first bounding box, a second item can be classified as a table and localized by a second bounding box, and a third item can be classified as a shelf and localized by a third bounding box.

[0002] However, a full scene description also includes the semantic relationships between items. For example, and continuing with the prior example, a full scene description would need to define whether the book is on the table or the shelf, whether the shelf is on the table, etc.

[0003] This gives rise to the task of visual relationship detection (VRD). In VRD, the model performs object detection and additionally predicts pairwise relationships between detected items in the form of <subject-predicate-object> triplets. Detecting both items and their relationships allows, for example, the construction of a scene graph in which objects are represented as nodes and their relationships as edges. Scene graphs have wide-ranging applications, such as applications in robotics, image retrieval, and providing grounding and explainability to multimodal large language models.

[0004] Prior work with VRD typically draws a distinction between object detection and relationship prediction. In such prior work, object detection is performed either as a wholly separate step before relationship prediction, or relationship prediction is performed by completely separate model parts such as relationship decoders that are responsible for modeling the interactions between objects. This separation between object detection and relationship prediction stages makes it difficult to optimize such models end-to-end for VRD, which can require an extended duration of training of relationship prediction model(s) and / orcan negatively impact robustness and / or accuracy of such models. Additionally or alternatively, this separation can require excess utilization of memory, processor, and / or other computational resources. For example, this separation can require that an image be fully processed twice (once in object detection and again in relationship prediction) and / or can require utilization of memory and / or processor intensive decoder model(s) in relationship prediction.

[0005] As opposed to separation of object detection and relationship prediction, some implementations disclosed herein utilize an encoder-only architecture that models objects and relationships jointly, directly in the image encoder. Such an architecture can be used to perform open-vocabulary relationship detection and can be trained end-to-end on mixtures of object detection and relationship annotations.Summary

[0006] Implementations disclosed herein can process an image using an image encoder to generate encoder output tokens. Each of the encoder output tokens can represent a corresponding candidate item / object in the image. Put another way, each of the encoder output tokens can indicate a detection of an item in the image. The image encoder can be, for example, one initially trained for object detection, such as one that is a transformer-based encoder-only object detector. The encoder output tokens that are generated using the image encoder can be processed to determine both: (i) classifications of the items in the image and (ii) relationships between the items in the image. The encoder output tokens can also optionally be utilized to determine bounding boxes and / or other localizations that define a corresponding region of the image for each of the items. The determined classifications of the candidate items and the relationships between the items can be utilized for various purposes, such as constructing a scene graph where the items and their classifications are represented as nodes and the relationships between the items are represented as edges.

[0007] Accordingly, some implementations enable utilization of a shared image encoder to process an image to generate encoder output tokens, and utilization of those tokens in determining both (i) classifications and (ii) relationships and, optionally, bounding boxes and / orother localizations. Further, some of those implementations enable determining the classifications, the relationships, and optionally the localizations without utilization of any decoder networks.

[0008] As a non-limiting example, assume that a given image is obtained (e.g., actively captured or received from a device). The given image can be an image captured by a camera or other vision component of a device, such as a vision component of a robot, an autonomous vehicle, a mobile phone, or other device. The given image can be processed using an image encoder to generate encoder output tokens that each represent a corresponding candidate item of the image.

[0009] Continuing with the example, in determining a relationship between two candidate items of the image, a first encoder output token can be processed using a subject subnetwork to generate a subject embedding representing a first candidate item as a subject. An "embedding" can refer to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values. The subject subnetwork can be a neural network model, such as a multi-layer perceptron (MLP). A second encoder output token can be processed using an object subnetwork to generate an object embedding representing a second candidate item as an object. The object subnetwork can be a neural network model, such as an MLP. A relationship embedding can then be generated based on combining the subject embedding and the object embedding. For example, element-wise addition can be used in combining the subject embedding and the object embedding.

[0010] The relationship embedding can then be compared (e.g., dot product comparison) to predicate embeddings to generate a score for each of the predicate embeddings. Each of the predicate embeddings represents a corresponding relationship between items (e.g., "sitting on", "near", "below", etc.) and each of the predicate embeddings can be generated (e.g., pregenerated), for example, based on processing corresponding natural language text, that describes the relationship, using a text encoder. A described relationship can include, for example, a spatial relationship, such as a spatial relationship that characterizes the relative positions of two items. Each of the generated scores defines a likelihood that the first candidate item (as a subject) has the corresponding relationship, represented by the predicateembedding, to the second candidate item (as an object). Put another way, each of the generated scores is specific to the relationship embedding, which is generated based on combing the subject and object embeddings, and each of the generated scores is for a corresponding one of the predicate embeddings.

[0011] The generated scores can then be used in selecting at least one relationship for the first candidate item (as a subject) to the second candidate item (as an object). For instance, the relationship that corresponds to the predicate embedding having the best score can be selected as the relationship. Optionally, a relationship of the first candidate item to the second candidate item is only selected when the best score also satisfies a threshold. Otherwise, no relationship is selected for the first candidate item to the second candidate item.

[0012] This general process can be performed for each of multiple item pairs represented by multiple encoder output tokens, optionally including other pairs where the first candidate item is a subject or object, but the second candidate item is not the subject or object (or vice versa). In various implementations, it can be computationally infeasible to determine relationship embeddings and / or to determine relationship scores for all possible combinations of items of an image. Accordingly, some implementations determine relationship embeddings and / or determine relationship scores for only a computationally feasible subset of combinations of items. For example, some implementations utilize a relationship attention subnetwork, which can be a hard attention mechanism, to select the subset. Utilization of such relationship attention subnetwork to select a subset can be more computationally efficient than determining relationship embeddings and / or determining relationship scores for the full set. For instance, the relationship attention subnetwork can have a computational cost that is comparable to a single self-attention layer.

[0013] Continuing with the example, in various implementations the first encoder output token can also be used in selecting a classification for the first candidate item and / or a localization for the first candidate item in the image. Likewise, the second encoder output token can also be used in selecting a classification for the second candidate item and / or a localization for the second candidate item in the image. For example, in selecting a classification based on the first encoder output token, the first encoder output token (or thesubject embedding or alternative embedding of the first encoder output token) can be compared to classification embeddings. For instance, a dot product comparison cam be performed to generate classification scores for the classification and the classification that corresponds to the best score can be selected as the classification. Each of the classification embeddings can represent a corresponding classification (e.g., "table", "bookshelf", "book", etc.) and each of the classification embeddings can be generated (e.g., pre-generated), for example, based on processing corresponding natural language text, that describes the classification, using a text encoder.

[0014] In these and other manners, generated encoder output tokens can be used to (i) both select classifications for a subject, object pair of items of an image and (ii) select a relationship of the subject to the object. The classifications and the relationship can be used to generate a <subject, predicate, object> triplet that can optionally form part of a scene graph.

[0015] A relationship selected according to implementations disclosed herein can be utilized for one or more of various purposes. As one example, the relationship can be processed using one or more generative models (e.g., a sequence-to-sequence and / or transformer-based large language model) to generate output indicative of one or more automated actions, and the one or more automated actions can be caused to be implemented (e.g., transmitted or otherwise communicated to one or more component(s)). For instance, a generated scene graph can be processed using a generative model, optionally along with user instructions and / or other content, to generate output indicative of one or more automated actions, such as one or more robotic actions.

[0016] As another example, prior to obtaining an image based on which a <subject, predicate, object> triplet is generated, a robotic control policy can be used to generate robotic control output that is used in controlling a real physical robot. For example, the robotic control policy can process preceding image(s) using the robotic control policy to generate the robotic control output. In such an example, the image can be one captured by a camera of the robot after implementation of the robotic control output and the <subject, predicate, object> triplet can be used to determine a measure of success of implementation of the one or more automated robotic control actions and / or to determine one or more next automated robotic controlactions to implement. For instance, the robotic control policy can process preceding image(s) and natural language instructions of "put the book on the bookshelf" to generate robotic control output that is intended to cause the robot to pick up a book and place it on a bookshelf. The <subject, predicate, object> triplet can be for a subject having a classification of "book" and an object having a classification of "bookshelf" and can be used to determine whether the book was successfully placed on the bookshelf (e.g., does the predicate correspond to a relationship "on" as opposed to e.g., "near"). Optionally, the score for the relationship represented by the predicate of the triplet can be used in determining the measure of success. For example, the success measure can match the score and / or whether success is determined can depend on a magnitude of the score.

[0017] As yet another example, a task to be performed can be processed to generate a <subject, predicate, object> triplet that reflects a goal state of the task. Further, a corresponding probability for a determined relationship, that is determined based on an image captured during an attempted performance of the task, and that is between the subject and the object of the triplet and conforms to the predicate of the triplet, can be used in determining a reward signal that reflects probability that the task has been successfully performed. The reward signal can be utilized in determining whether to deem the attempted performance of the task as successful and / or as a supervision signal for a corresponding training instance generated based on the attempted performance of the task (e.g., a training instance that includes input corresponding to action(s) taken during the attempted performance and that includes output corresponding to the reward signal). As a particular example, for a robotic task specified by natural language of "place the sponge next to the bowl", it can be processed to generate a triplet of <sponge, next to, bowl> that reflects the goal state of the robotic task. An image, captured during performance of the robotic task (e.g., by a camera of a robot performing the task), can be processed to determine a measure that reflects a strength of the "next to" relationship between a "sponge" and "bowl" object and that measure can be used as the reward signal.

[0018] Some implementations include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of themethods described herein. Some implementations include a robot and / or a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described herein.

[0019] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. As another example, the paper included in this disclosure is contemplated as being part of the subject matter disclosed herein.Brief Description of the Drawings

[0020] FIG. 1A illustrates an example of processing an image to detect objects in the image and to determine relationships between the objects in the image.

[0021] FIG. IB illustrates an example of processing that can be performed using the relationship attention layer of FIG. 1A.

[0022] FIG. 2 is a flowchart illustrating an example method according to various implementations disclosed herein.

[0023] FIG. 3 is a flowchart illustrating another example method according to various implementations disclosed herein.

[0024] FIG. 4 depicts an example robot, in accordance with various implementations.

[0025] FIG. 5 schematically depicts an example architecture of a computer system.

[0026] FIGS. 6A and 6B illustrate various tables.

[0027] FIG. 7 illustrates example objects that can be detected, and example relationships between the objects, that can be determined according to implementations disclosed herein.Description

[0028] A fundamental goal of computer vision is to decompose visual scenes into structured semantic representations. A commonly studied task is object detection, where objects in an image are localized by bounding boxes and classified into semantic categories. However, ascene description also includes the semantic relationships between objects. This gives rise to the task of visual relationship detection (VRD). In VRD, a model detects objects and infers pairwise relationships between them in the form of <subject-predicate-object> triplets.

[0029] Detecting both objects and their relationships allows the construction of a scene graph, where objects are represented as nodes and their relationships as edges. Scene graph generation (SGG) has wide-ranging applications in various technical fields such as robotics and image retrieval. Structured scene representations are also increasingly used to provide grounding and explainability to multimodal large language models.

[0030] Prior work typically distinguishes between object detection and relationship prediction. Detection is performed either as a separate step before relationship prediction, or by separate model parts, such as relationship decoders, that model interactions between objects. This distinction between object detection and relationship prediction hinders end-to-end optimization for VRD. In contrast, implementations disclosed herein propose an encoder-only architecture that models objects and relationships jointly, directly in the image encoder. This architecture performs open-vocabulary relationship detection and can be trained end-to-end on arbitrary mixtures of object detection and relationship annotations.

[0031] Implementations build on a transformer-based encoder-only object detector, where the image encoder's output tokens directly represent object proposals. Class embeddings and bounding boxes are decoded from these tokens with lightweight heads. This architecture is well-suited to learning relationships directly in the image encoder, without additional relationship-specific stages, because the encoder's self-attention already models a ll-to-a II pairwise interactions between object proposal tokens.

[0032] To access information about the relationship between two tokens, the embeddings corresponding to the <subject> and <object> tokens can be combined using a relationship attention layer. Obtaining relationship embeddings for all possible pairwise combinations of object proposal tokens would be computationally infeasible. To reduce the number of combinations, a self-supervised hard attention mechanism can be utilized that selects the highest-confidence <subject-object> pairs at a computational cost comparable to a single self-attention layer. Some implementations disclosed herein directly supervise the attention scores of this mechanism without propagating gradients through the hard selection.

[0033] An additional benefit is that object names are disentangled from relationship predicates during inference. Unlike prior open-vocabulary techniques, implementations disclosed herein can embed object and predicate texts separately and efficiently generate confidence scores for all possible <subject-predicate-object> combinations.

[0034] In summary, implementations disclosed herein describe: an efficient architecture for open-vocabulary visual relationship detection; a single-stage method for joint object and relationship detection training; efficient, disentangled object and relationship inference; and analysis of inference speed, ablations, and qualitative examples. Implementations achieve state-of-the-art visual relationship detection performance on the Visual Genome dataset and on the large-vocabulary GQA benchmark in open-vocabulary and zero-shot settings, while being significantly simpler than prior approaches.

[0035] Some implementations disclosed herein propose an architecture for open-vocabulary relationship prediction, where both objects and their relationships are handled as first-class citizens in a single-stage process within the model backbone. This builds upon an encoder-only architecture for object detection, adapted for relationship prediction by adding a specialized attention layer. This specialized attention layer leverages the pairwise structure between existing object embeddings to derive relationship embeddings.

[0036] An encoder-only open-vocabulary object detection architecture forms the basis of the model according to various implementations disclosed herein. This architecture can include transformer-based image and text encoders, contrastively pre-trained on numerous image-text pairs. The image encoder can be adapted for detection by removing the final pooling layer and adding heads that predict bounding boxes and class embeddings directly from the image encoder's output tokens. For open-vocabulary object classification, the embeddings from the class prediction head are compared to text encoder embeddings of object descriptions. This architecture exhibits robust open-vocabulary object detection performance and avoids training instabilities seen in some encoder-decoder detection models.

[0037] Referring to FIGS. 1A and IB, an image 101 can be processed using a vision transformer encoder 151 to generate image tokens 101A. For relationship selection, image tokens 101A are projected using two lightweight multi-layer perceptrons (MLPs) 152A and 152B, of a relationship attention layer 152, to generate <subject> embeddings 102A and <object> embeddings 103A. A relationship score is then computed as the inner product of all <subject> and <object> embeddings 102A and 103A. Relationships are filtered by first selecting top object instances, using the diagonal scores to represent instance likelihood. Among the remaining instances, top <subject-object> pairs are selected using off-diagonal scores. This yields relationship triplets, each including a <subject> index 102, an <object> index 103, and a relationship embedding 104 computed by summing the respective <subject> and <object> embeddings. For classification to generated predicted object / predicates classes 106, relationship embeddings 104 are compared against text embeddings 105 of object class or predicate text descriptions. Image tokens, for the top <subject-object> pairs are selected e.g., using select layer(s) 153, and processed using MLP 154 to generate predicted boxes, or other bounding shape(s), for the subjects and objects of the pairs.

[0038] Each of the image encoder tokens 101A represents an object proposal. The selfattention layers of the vision transformer encoder 151 model all pairwise interactions between the image encoder tokens 101A. To extract information as <subject-predicate-object> triplets, two MLPs 152A and 152B transform vision encoder output tokens 101A into <subject> and <object> embeddings 102A and 103A. Using distinct MLPs, 152A and 152B, for <subject> and <object> breaks symmetry in subsequent processing.

[0039] A relationship embedding, 104, representing the relationship between two object proposals is obtained by element-wise addition of their <subject> and <object> embeddings 102 and 103. Since obtaining relationship embeddings for all <subject-object> pairs is computationally prohibitive, a relationship attention layer 152 performs hard attention to select the most likely <subject-object> pairs 102, 103. This relationship attention layer 152 is used to compute a score based on the inner product of the <subject> and <object> embedding vectors 102A and 103A, representing the likelihood of a relationship between them. For example, the relationship attention layer 152 can be used to generate an attention-like scorePij = s^ , where Sj and Oj are the embedding vectors of <subject> / and <object> j, and ptj represents the likelihood that a relationship between <subject> / and <object> j exists. This generates an N x N matrix (where N is the number of object proposals), from which the top k pairs, 102 and 103, are selected. For these top k pairs, relationship embeddings 104 are computed and layer normalized. For example, relationship embeddings 104 can be computed as rtj= LayerNorm(Sj + o . To classify a pair's relationship, its relationship embedding 104 is compared to text embeddings 105 of relationship predicates. Embeddings where <subject> and <object> are identical (e.g., (ri=7)) represent object instances, and object classes are predicted from them. Bounding boxes 107 are predicted from corresponding image encoder output tokens.

[0040] The relationship attention layer 152 identifies object pairs for relationship classification using hard attention, with <subject> embeddings 102A as queries and <object> embeddings 103A as keys. Since hard selection is non-differentiable, gradients from relationship prediction cannot directly train this layer. Instead, the relationship score is self-supervised to predict the maximum predicate class probability that will be predicted for the relationship at the classification stage.

[0041] The image and text encoders can be initialized from a vision-language model contrastively pre-trained on numerous image-text pairs. After adding the relationship attention layer 152 and object bounding box and class prediction heads, the model is jointly trained on a mixture of object and relationship detection datasets in a single stage.

[0042] Bipartite matching between object predictions and ground-truth annotations is performed based on a cost including object classification and box prediction losses. This matching establishes a matching of predicted to ground-truth relationship predicates. Unmatched predictions are trained to predict low scores for all classes and incur no box prediction loss. Bounding box regression can use LI and / or generalized intersection-over- union (gloU) losses. Object and predicate classification can use a sigmoid cross-entropy loss computed between ground truth classes and logits obtained from the inner product of relationship embeddings (selected by the Relationship Attention layer) with text embeddingsof class names. For individual object embeddings, the class name is the object category or description. For relationship embeddings, the predicate text is used.

[0043] The relationship attention layer 152 is used to predict a score representing the likelihood that a <subject-object> pair forms a relationship (if i j) or that an object exists in the image (if i = j). This score can be trained with a sigmoid cross-entropy loss against targets provided by the model itself — the maximum probability predicted for any class for the corresponding embedding. This loss is only computed for selected objects and relationships.

[0044] The final loss can be an equally weighted sum of classification loss, LI box loss, gloU box loss, and relationship score loss.

[0045] FIG. 2 is a flowchart illustrating an example method 200 according to various implementations disclosed herein. For convenience, the operations of the method 200 are described with reference to a system that performs the operations. This system can include one or more components of a computer system, such as computer system 510. Moreover, while operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added.

[0046] At block 202, the system can obtain an image and obtains predicate embeddings. Each of the predicate embeddings represents a corresponding relationship between items.

[0047] At block 204, the system can process the image using an image encoder to generate encoder output tokens, each representing a corresponding candidate item of the image.

[0048] At block 206, the system can process a first encoder output token using a subject subnetwork to generate a subject embedding representing a first candidate item, of the corresponding candidate items of the image, as a subject. At block 206 the system can also process a second encoder output token using an object subnetwork to generate an object embedding representing a second candidate item, of the corresponding candidate items of the image, as an object.

[0049] At block 208, the system can generate a relationship embedding based on combining the subject embedding and the object embedding. For example, element-wise addition can be used in combining the subject embedding and the object embedding.

[0050] At block 210, the system can compare the relationship embedding to the predicate embeddings.

[0051] At block 212, the system can determine, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding, where each of the corresponding scores defines a likelihood that the first candidate item has the corresponding relationship, represented by the predicate embedding, to the second candidate item.

[0052] At block 214, the system can select, based on the corresponding scores, a given relationship of the corresponding relationships.

[0053] In some implementations, method 200 can further include: obtaining classification embeddings, where each of the classification embeddings represents a corresponding classification of an item; selecting a first classification for the first candidate item based on comparing the first encoder output token to the classification embeddings; and selecting a second classification for the second candidate item based on comparing the second encoder output token to the classification embeddings.

[0054] In some implementations, method 200 can further include generating a subject- predicate-object triplet that includes the first classification as the subject, the given relationship as the predicate, and the second classification as the object.

[0055] In some implementations, method 200 can further include: processing the first encoder output token, using a localization subnetwork, to generate first localization data defining a corresponding region of the image for the first candidate item; processing the second encoder output token, using the localization subnetwork, to generate second localization data defining a corresponding region of the image for the second candidate item; storing the first localization data in association with the subject of the subject-predicate-object triplet; and storing the second localization data in association with the object of the subject-predicate-object triplet. In some of those implementations, method 200 can further include: processing a third encoder output token using the subject subnetwork to generate an additional subject embedding representing a third candidate item, of the corresponding candidate items of the image, as a subject; processing a fourth encoder output token using the object subnetwork to generate anadditional object embedding representing a fourth candidate item, of the corresponding candidate items of the image, as an object; and determining to not generate an additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining to not generate a corresponding additional score for each of the predicate embeddings and for the additional relationship embedding. In some versions of those implementations, determining to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding can include processing the additional subject embedding and the additional object embedding using the relationship attention subnetwork to generate an additional predicted score (e.g., that defines a likelihood that a relationship exists between the third candidate item represented by the additional subject embedding and the fourth candidate item represented by the additional object embedding); and determining, based on the additional predicted score, to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding.

[0056] In some implementations, the relationship attention subnetwork may be previously trained based on: processing a prior subject embedding and a prior object embedding using the relationship attention subnetwork to generate a prior predicted score; generating a prior relationship embedding based on combining the prior subject embedding and the prior object embedding; comparing the prior relationship embedding to the predicate embeddings; determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the prior relationship embedding; selecting a maximum score of the corresponding scores; generating a loss based on comparing the prior predicted score to the maximum score; and updating the relationship attention subnetwork based on the loss.

[0057] In some implementations, method 200 can further include causing the given relationship to be rendered at a user interface output device of a client device associated with the image, such as via a client device that includes a camera via which the image is captured.

[0058] In some implementations, method 200 can further include processing the given relationship, using one or more generative models, to generate output indicative of one or more automated actions; and causing the one or more automated actions to be implemented. The one or more automated actions can include, for example, one or more automated robotic control actions.

[0059] In some implementations, method 200 can further include: generating a scene graph that includes the relationship and that also additional relationships generated based on other subject embedding, object embedding pairs that are generated based on the image; and processing the scene graph, using one or more generative models, to generate output indicative of one or more automated actions; and causing the one or more automated actions to be implemented.

[0060] In some implementations, method 200 can further include: processing, prior to obtaining the image and using a robot control policy, one or more preceding images, that are captured prior to the image, to generate robot control policy output that is indicative of one or more automated robotic control actions; causing the one or more automated robotic control actions to be implemented; and determining, based on the selected given relationship, a measure of success of implementation of the one or more automated robotic control actions, and / or determining, based on the selected given relationship, one or more next automated robotic control actions to implement.

[0061] In some implementations, block 208 and / or block 212 can be preceded by determining whether to generate the relationship embedding and / or the corresponding score. Determining whether to generate the relationship embedding and / or the corresponding score can include processing the subject embedding and the object embedding using a relationship attention subnetwork to generate a predicted score, where the predicted score defines a likelihood that a relationship exists between the first candidate item represented by the subject embedding and the second candidate item represented by the object embedding; and determining, basedon the predicted score, to generate the relationship embedding and / or the corresponding score. In some of those implementations, determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score can include determining that the predicted score satisfies a threshold that is relative to other predicted scores for other subject embedding, object embedding pairs, or determining that the predicted score satisfies a fixed threshold.

[0062] FIG. 3 is a flowchart illustrating another example method 300 according to various implementations disclosed herein. For convenience, the operations of the method 300 are described with reference to a system that performs the operations. This system can include one or more components of a computer system, such as computer system 510. Moreover, while operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted or added.

[0063] At block 302, the system can obtain a training data instance that includes a training image and one or more ground truth labels for the training image. The ground truth labels include a given ground truth label that includes relationship text that describes a relationship, of a first item characterized in the image to a second item characterized in the image.

[0064] At block 304, the system can process the training image using an image encoder to generate encoder output tokens each representing a corresponding candidate item of the training image.

[0065] At block 306, the system can process a first encoder output token using a subject subnetwork to generate a subject embedding representing the first item; and process a second encoder output token using an object subnetwork to generate an object embedding representing the second item.

[0066] At block 308, the system can generate a relationship embedding based on combining the subject embedding and the object embedding and generate a predicate embedding based on processing the relationship text using a text encoder.

[0067] At block 310, the system can generate a predicate loss based on comparing the relationship embedding to the predicate embedding.

[0068] At block 312, the system can update the subject network and the object network based on a loss that is based on the predicate loss. In some implementations, block 312 can further include updating the image encoder and the text encoder based on the loss.

[0069] In some implementations, a ground truth label further includes subject text that describes a first classification of the first item and object text that describes a second classification for the second item. In some of those implementations, method 300 can further include: generating a first classification embedding based on processing the subject text using the text encoder; generating a second classification embedding based on processing the object text using the text encoder; generating a first classification loss based on comparing the first classification embedding to the subject embedding or an additional embedding generated based on the first encoder output token; and generating a second classification loss based on comparing the second classification embedding to the object embedding or a further embedding generated based on the second encoder output token. In some of those implementations, the loss is further based on the first classification loss and the second classification loss.

[0070] FIG. 4 schematically depicts an example architecture of a robot 420. The robot 420 includes a robot control system 460, one or more operational components 440a-440n, and one or more sensors 442a-442m. The sensors 442a-442m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 442a-m are depicted as being integral with robot 420, this is not meant to be limiting. In some implementations, sensors 442a-m may be located external to robot 420, e.g., as standalone units.

[0071] Operational components 440a-440n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot 420 may have multiple degrees of freedom and each of the actuators may control the actuation of the robot 420 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., amotor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

[0072] The robot control system 460 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 420. In some implementations, the robot 420 may comprise a "brain box" that may include all or aspects of the control system 460. For example, the brain box may provide real time bursts of data to the operational components 440a-n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 440a-n. In some implementations, the robot control system 460 may perform one or more aspects of method(s) described herein, such as method 200 of FIG. 2.

[0073] As described herein, in some implementations all or aspects of the control commands generated by control system 460, in controlling a robot during performance of a robotic task, can be generated based on robotic skill(s) determined to be relevant for the robotic task based on world-grounding and task-grounding measures as described herein. Although control system 460 is illustrated in FIG. 4 as an integral part of the robot 420, in some implementations, all or aspects of the control system 460 may be implemented in a component that is separate from, but in communication with, robot 420. For example, all or aspects of control system 460 may be implemented on one or more computing devices that are in wired and / or wireless communication with the robot 420, such as computing device 510.

[0074] FIG. 5 is a block diagram of an example computer system 510. Computer system 510 typically includes at least one processor 514 which communicates with a number of peripheral devices via bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, user interface output devices 520, user interface input devices 522, and a network interface subsystem 516. The input and output devices allow user interaction with computer system510. Network interface subsystem 516 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0075] User interface input devices 522 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 510 or onto a communication network.

[0076] User interface output devices 520 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 510 to the user or to another machine or computer system.

[0077] Storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include the logic to perform selected aspects of method 200, method 300, and / or to implement one or more aspects of robot 400. Memory 525 used in the storage subsystem 524 can include a number of memories including a main random-access memory (RAM) 530 for storage of instructions and data during program execution and a read only memory (ROM) 532 in which fixed instructions are stored. A file storage subsystem 526 can provide persistent storage for program and data files, and may include a hard disk drive, a CD- ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 526 in the storage subsystem 524, or in other machines accessible by the processor(s) 514.

[0078] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computer system 510 communicate with each other as intended. Although bussubsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0079] Computer system 510 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 510 depicted in FIG. 5 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 510 are possible having more or fewer components than the computer system depicted in FIG. 5.

[0080] With reference to FIGS. 6A and 6B, various experiments based on implementations disclosed herein are described. The following datasets were used for training or evaluation: Visual Genome, the largest VRD dataset, is labeled with 2.3M triplet relationships across 108K images. However, due to noisy annotations, the community commonly evaluates VRD on a cleaned version reduced to the 150 most frequent object classes and 50 most frequent predicate classes. This is referred to as Visual Genome 150 (VG150). GQA uses the same image corpus as Visual Genome but has more diverse labels, with 1703 object classes (1704 including background) and 310 predicates. Spatial relationships (e.g., "to the left of") are automatically labeled based on bounding box positions. GQA200 is a reduced, cleaned version of GQA with 200 object classes and 100 predicate classes. HICO-DET focuses on human-object interactions. It contains 50k images exhaustively annotated for 600 human-object interactions (HOIs). Objects365 (0365) is a large-scale object detection dataset with 365 object categories across two million images. Open X-Embodiment (OXE) is a dataset for learning vision-language-action policies in robotics. The lack of bounding box annotations precludes quantitative evaluation, but its visual differences from standard VRD benchmarks and its application potential make it suitable for qualitative evaluation.

[0081] Models were trained on a mixture of VG, VG150, GQA200, HICO, and 0365, with proportions shown in Table 1 of FIG. 6A. The B / 32 model was trained for 200,000 steps at batch size 256. To avoid using evaluation images for training, all training datasets were rigorously filtered to remove images present in any test split (VG150, GQA, GQA200, and HICO)using an image similarity filter that detects near matches. Table 1 shows the number of images removed. This deduplication is consistent with prior work.

[0082] Prior work uses diverse evaluation metrics, often dataset-specific. For non-exhaustively labeled datasets (Visual Genome and GQA), precision-based metrics are inconclusive. A Recall@K metric is used, where K is a fixed budget of <subject-predicate-object> triplets. However, this metric is biased towards frequent predicates. Therefore, mean Recall@K, which averages Recall@K across each predicate class, is used. For exhaustively labeled datasets like HICO, mean Average Precision (mAP), a standard object detection metric, is used. The mean is taken across all 600 possible HOI triplets. Results are also reported separately for rare (< 10 occurrences) and non-rare (> 10 occurrences) HOIs as mAPr and mAPn. Evaluation can be "graph-constrained" (one prediction per object pair) or "graph-unconstrained" (multiple predictions per pair). To ensure fair comparison, the PyTorch evaluation procedure was replicated in JAX for recall-based metrics, and the original Matlab procedure was numerically reproduced in JAX for HICO. Exhaustive Relationship Evaluation. Previous methods entangle objects and relationships in a single representation, scoring embeddings against the full <subject-predicate-object> triplet. This is computationally intractable for large vocabularies (e.g., VG, GQA). Prior works evaluate only on triplets present in the test split, potentially inflating metrics. In contrast, the described method disentangles objects and predicates, enabling exhaustive evaluation across all object and predicate combinations.

[0083] Table 2 of FIG. 6A shows results for the models and prior work. The method significantly improves on the prior best method, DT2-ACBS, by 5.1 points mR@100 (29.5 vs. 24.4). A large performance difference is also observed compared to UniVRD, which uses a similar detection architecture but adds a Transformer decoder. Training the B / 32 model on the UniVRD data mixture still shows a large improvement (23.9% VG mR@100 vs. 12.1% for UniVRD; Table 6 of FIG. 6B), suggesting the encoder-only architecture is superior to decoder-based architectures

[0084] Improving long-tail performance is crucial in VRD due to the skewed distribution of relationship triplets. The method, without special treatment of rare classes, was evaluated on the GQA dataset, which has a larger vocabulary than VG. The method surpasses prior results on the GQA200 test split (Table 3 of FIG. 6A). The full GQA dataset, with 1703 object and 311predicate classes, was also evaluated. The model achieves higher performance on scene graph generation than prior methods on scene graph classification (Table 4 of FIG. 6A). Zero-shot generalization to unseen classes (least frequent 1503 object and 211 predicate classes) was also assessed (Table 4, bottom). Performance is lower than on seen classes but still higher than prior scene graph classification methods. Two factors contribute to the strong open-vocabulary performance: the simple design allows for knowledge transfer from the pre-trained VLM backbone, and the open-vocabulary design allows end-to-end training on a mix of datasets.

[0085] The model was evaluated on the HICO benchmark for human-object interaction. Performance is comparable to the most similar prior method but does not show the same improvements seen for VG or GQA200 (Table 5 of FIG. 6B). HICO's narrower vocabulary and limited training data may limit the benefit of pre-trained representations. This is supported by the state-of-the-art HICO method, which uses large amounts of person-focused pseudo-labels. 4.5 Ablations Ablation studies were conducted to analyze the interplay of training data and architecture. Inference Speed and Number of Predicted Relationships. The primary hyperparameter is k, the number of relationships selected by the Relationship Attention layer. Figure 3 shows the relationship between VRD performance, inference speed, and k. For training, a value of k was used, resulting in 71% (B / 32) to 81% (L / 14) of the speed of a pure object detection model. Without top-k selection, the model would be much slower. Since most <subject-object> pairs do not form relationships, VRD performance is unaffected by top-k selection at inference. A value of k can be used at inference to achieve 90% of the object detector's speed (for B / 32 and L / 14), achieving 52.8 FPS at batch size 1 on an NVIDIA V100 GPU. Dataset Mixture. Models were trained on various dataset mixtures (Table 6 of FIG. 6B). The full mixture (VG, VG150, GQA200, HICO, and 0365) was used for all experiments except where noted. Training on the UniVRD mixture (VG150, HICO, 0365, COCO) still showed superior performance on VG150 (23.9 mR@100 vs 12.1 for UniVRD for B / 32 models). This suggests the encoder-only model with Relationship Attention is superior to decoder-based architectures. Including both object detection (0365) and large-vocabulary VRD data (VG) improves all metrics. Object Detection Performance. Accurate object detection is crucial for good relationship detection. The architecture treats object detection as a special case ofrelationship detection where <subject> equals <object>. Training the model only on detection datasets (Objects365 and Visual Genome, same mixture as OWL-ViT) resulted in similar object detection performance to OWL-ViT (21.9% vs. 22.1% mAP and 18.3% vs. 18.9% mAPr on LVIS), indicating the Relationship Attention design does not negatively impact object detection.

[0086] FIG. 7 illustrates example objects that can be detected, and example relationships between the objects, that can be determined according to implementations disclosed herein. It is noted that FIG. 7 shows only some of the example relationships that can be determined, such as the top relationships by confidence score. Detailed description of each of the objects and the relationships is omitted in this detailed description for conciseness, but they are illustrated in FIG. 7. For example, FIG. 7 illustrates detected objects of books and bookcase, and a determined relationship of the books being "on" the bookcase. As another example, FIG. 7 illustrates detected objects of a mirror and a wall and a detected relationship of the mirror "hanging on" the wall. As yet another example, FIG. 7 illustrates detected objects of a mirror and a couch and a detected relationship of the mirror "hanging above" the couch.

[0087] In some implementations, a method implemented by processor(s) is provided and includes obtaining an image and predicate embeddings. Each of the predicate embeddings represents a corresponding relationship between items. The method further includes processing the image using an image encoder to generate encoder output tokens each representing a corresponding candidate item of the image. The method further includes processing a first encoder output token using a subject subnetwork to generate a subject embedding representing a first candidate item, of the corresponding candidate items of the image, as a subject. The method further includes processing a second encoder output token using an object subnetwork to generate an object embedding representing a second candidate item, of the corresponding candidate items of the image, as an object. The method further includes generating a relationship embedding based on combining the subject embedding and the object embedding. The method further includes comparing the relationship embedding to the predicate embeddings. The method further includes determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding. Each of the corresponding scores defines a likelihood that the first candidate itemhas the corresponding relationship, represented by the predicate embedding, to the second candidate item. The method further includes selecting, based on the corresponding scores, a given relationship of the corresponding relationships.

[0088] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0089] In some implementations, the method further includes obtaining classification embeddings that each represent a corresponding classification of an item, selecting a first classification for the first candidate item based on comparing the first encoder output token to the classification embeddings, and selecting a second classification for the second candidate item based on comparing the second encoder output token to the classification embeddings. In some versions of those implementations, the method further includes generating a subject- predicate-object triplet that includes the first classification as the subject, the given relationship as the predicate, and the second classification as the object. In some of those versions, the method further includes processing the first encoder output token, using a localization subnetwork, to generate first localization data defining a corresponding region of the image for the first candidate item and processing the second encoder output token, using the localization subnetwork, to generate second localization data defining a corresponding region of the image for the second candidate item. The first localization data can be stored in association with the subject of the subject-predicate-object triplet and the second localization data can be stored in association with the object of the subject-predicate-object triplet.

[0090] In some implementations, the method further includes, prior to generating the relationship embedding based on combining the subject embeddings and the object embedding and / or prior to determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding, determining whether to generate the relationship embedding and / or the corresponding score. In some versions of those implementations, determining whether to generate the relationship embedding and / or the corresponding score includes processing the subject embedding and the object embedding using a relationship attention subnetwork to generate a predicted score and determining, based on the predicted score, to generate the relationship embedding and / or thecorresponding score. The predicted score can define a likelihood that a relationship exists between the first candidate item represented by the subject embedding and the second candidate item represented by the object embedding. In some versions of those implementations, determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score includes determining that the predicted score satisfies a threshold that is relative to other predicted scores for other subject embedding, object embedding pairs. In some additional or alternative versions of those implementations, determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score includes determining that the predicted score satisfies a fixed threshold. In yet further additional or alternative versions of those implementations, the method further includes: processing a third encoder output token using the subject subnetwork to generate an additional subject embedding representing a third candidate item, of the corresponding candidate items of the image, as a subject; processing a fourth encoder output token using the object subnetwork to generate an additional object embedding representing a fourth candidate item, of the corresponding candidate items of the image, as an object; and determining to not generate an additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining to not generate a corresponding additional score for each of the predicate embeddings and for the additional relationship embedding. In some of those yet further additional or alternative versions, determining to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding can include processing the additional subject embedding and the additional object embedding using the relationship attention subnetwork to generate an additional predicted score and determining, based on the additional predicted score, to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding. The additional predicted score can define a likelihoodthat a relationship exists between the third candidate item represented by the additional subject embedding and the fourth candidate item represented by the additional object embedding. Optionally, the relationship attention subnetwork is previously trained based on: processing a prior subject embedding and a prior object embedding using the relationship attention subnetwork to generate a prior predicted score; generating a prior relationship embedding based on combining the prior subject embedding and the prior object embedding; comparing the prior relationship embedding to the predicate embeddings; and determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the prior relationship embedding; selecting a maximum score of the corresponding scores; generating a loss based on comparing the prior predicted score to the maximum score; and updating the relationship attention subnetwork based on the loss.

[0091] In some implementations, the method further includes causing the given relationship to be rendered at a user interface output device of a client device associated with the image. In some of those implementations, the client device includes a camera via which the image is captured.

[0092] In some implementations, the method further includes processing the given relationship, using one or more generative models, to generate output indicative of one or more automated actions, and causing the one or more automated actions to be implemented. In some of those implementations, the one or more automated actions include one or more automated robotic control actions.

[0093] In some implementations, the method further includes generating a scene graph that includes the relationship and that also additional relationships generated based on other subject embedding, object embedding pairs that are generated based on the image. In some versions of those implementations, the method further includes processing the scene graph, using one or more generative models, to generate output indicative of one or more automated actions, and causing the one or more automated actions to be implemented.

[0094] In some implementations, the method further includes: processing, prior to obtaining the image and using a robot control policy, one or more preceding images, that are captured prior to the image, to generate robot control policy output that is indicative of one or moreautomated robotic control actions; causing the one or more automated robotic control actions to be implemented; and determining, based on the selected given relationship, a measure of success of implementation of the one or more automated robotic control actions, and / or determining, based on the selected given relationship, one or more next automated robotic control actions to implement.

[0095] In some implementations, a method implemented by processor(s) is provided and includes processing an image using an image encoder to generate encoder output tokens and processing the encoder output tokens using a subject subnetwork to generate subject embeddings. Each of the subject embeddings is for a corresponding one of the encoder output tokens and represents a corresponding candidate item, of the image, as a corresponding subject of a subject-predicate-object triplet. The method further includes processing the encoder output tokens using an object subnetwork to generate object embeddings. Each of the object embeddings is for a corresponding one of the encoder output tokens and representing a corresponding candidate item, of the image, as a corresponding object of a subject-predicate-object triplet. The method further includes generating relationship embeddings. Generating each of the relationship embeddings includes combining a corresponding unique pair of a corresponding one of the subject embeddings and a corresponding one of the object embeddings. The method further includes, for each of the relationship embeddings: comparing the relationship embedding to one or more predicate embeddings, wherein each of the predicate embeddings represents a corresponding relationship between items; and determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding. Each of the corresponding scores defines a likelihood that the corresponding subject, represented by the corresponding one of the subject embeddings used in generating the relationship embedding, has the corresponding relationship, represented by the predicate embedding, to the corresponding object represented by the corresponding one of the object embeddings used in generating the relationship embedding.

[0096] These and other implementations of the technology disclosed herein can include one or more of the following features.T1

[0097] In some implementations, generating the relationship embeddings is performed for only a subset of the corresponding unique pairs and the method further includes selecting, from the corresponding unique pairs, the subset of the corresponding unique pairs. In some versions of those implementations, selecting the subset of the corresponding unique pairs includes, for each of the corresponding unique pairs, processing the corresponding one of the subject embeddings and the corresponding one of the object embeddings, of the corresponding unique pair, using a relationship attention subnetwork to generate a corresponding predicted score and selecting the subset of the corresponding unique pairs based on the corresponding predicted scores. The corresponding predicted score defines a likelihood that a relationship exists between the corresponding one of the subject embeddings and the corresponding one of the object embeddings. In some of those versions selecting the subset of the corresponding unique pairs based on the corresponding predicted scores includes selecting a given quantity of the corresponding unique pairs that have the corresponding predicted scores most indicative of likelihood that a relationship exists. The subset of the corresponding unique pairs can be, for example, less than fifty percent, less than forty percent, less than thirty percent, less than twenty percent, less than ten percent, less than five percent, less than one percent, or less than two-tenths of a percent of the corresponding unique pairs.

[0098] In some implementations, combining a corresponding unique pair of a corresponding one of the subject embeddings and a corresponding one of the object embeddings includes performing an element-wise addition of the corresponding one of the subject embeddings and the corresponding one of the object embeddings.

[0099] In some implementations, comparing the relationship embedding to the one or more predicate embeddings includes performing a dot-product comparison of the relationship embedding to each of the one or more predicate embeddings. In some of those implementations, determining, based on the comparing, the corresponding score for each of the predicate embeddings and for the relationship embedding includes determining the corresponding score based on the result of the dot-product comparison.

[0100] In some implementations, the one or more predicate embeddings are each generated based on processing, using a text encoder, a natural language text sequence that describes the corresponding relationship between items that is represented by the predicate embedding.

[0101] In some implementations, the image encoder and the text encoder are pre-trained jointly with one another. In some of those implementations, the image encoder and the text encoder are pre-trained in conjunction with one another by repeatedly performing operations including: obtaining: (i) a training image, (ii) a positive text sequence, wherein the positive text sequence characterizes the training image, and (iii) one or more negative text sequences, where the negative text sequences do not characterize the training image; generating an embedding of the training image using the image encoding subnetwork; generating respective embeddings of the positive text sequence and each of the negative text sequences using the text encoding subnetwork; and jointly training the image encoding subnetwork and the text encoding subnetwork to encourage: (i) greater similarity between the embedding of the training image and the embedding of the positive text sequence, (ii) lesser similarity between the embedding of the training image and the embeddings of the negative text sequences.

[0102] In some implementations, the method further includes, for each of the relationship embeddings: comparing the corresponding one of the subject embeddings, used in generating the relationship embedding, to one or more classification embeddings that each represents a corresponding category of an item; determining, based on the comparing, a corresponding subject classification score for each of the classification embeddings and for the corresponding one of the subject embeddings; comparing the corresponding one of the object embeddings, used in generating the relationship embedding, to the one or more classification embeddings; determining, based on the comparing, a corresponding object classification score for each of the classification embeddings and for the corresponding one of the object embeddings. In some of those implementations, the method further includes determining a plurality of subject-predicate-object triplets based on the corresponding scores, the corresponding subject classification scores, and the corresponding object classification scores.

[0103] In some implementations, a method implemented by processor(s) is provided and includes obtaining a training data instance that includes a training image and one or moreground truth labels for the training image. The ground truth labels including a given ground truth label that includes relationship text that describes a relationship, of a first item characterized in the in the image to a second item characterized in the image. The method further includes processing the training image using an image encoder to generate encoder output tokens each representing a corresponding candidate item of the training image. The method further includes processing a first encoder output token using a subject subnetwork to generate a subject embedding representing the first item. The method further includes processing a second encoder output token using an object subnetwork to generate an object embedding representing the second item. The method further includes generating a relationship embedding based on combining the subject embedding and the object embedding. The method further includes generating a predicate embedding based on processing the relationship text using a text encoder. The method further includes generating a predicate loss based on comparing the relationship embedding to the predicate embedding. The method further includes updating the subject network and the object network based on a loss that is based on the predicate loss.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors, the method comprising: obtaining an image and predicate embeddings, wherein each of the predicate embeddings represents a corresponding relationship between items; processing the image using an image encoder to generate encoder output tokens each representing a corresponding candidate item of the image; processing a first encoder output token using a subject subnetwork to generate a subject embedding representing a first candidate item, of the corresponding candidate items of the image, as a subject; processing a second encoder output token using an object subnetwork to generate an object embedding representing a second candidate item, of the corresponding candidate items of the image, as an object; generating a relationship embedding based on combining the subject embedding and the object embedding; comparing the relationship embedding to the predicate embeddings; and determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding, wherein each of the corresponding scores defines a likelihood that the first candidate item has the corresponding relationship, represented by the predicate embedding, to the second candidate item; and selecting, based on the corresponding scores, a given relationship of the corresponding relationships.

2. The method of claim 1, further comprising: obtaining classification embeddings, wherein each of the classification embeddings represents a corresponding classification of an item;selecting a first classification for the first candidate item based on comparing the first encoder output token to the classification embeddings; and selecting a second classification for the second candidate item based on comparing the second encoder output token to the classification embeddings.

3. The method of claim 2, further comprising: generating a subject-predicate-object triplet that includes the first classification as the subject, the given relationship as the predicate, and the second classification as the object.

4. The method of claim 3, further comprising: processing the first encoder output token, using a localization subnetwork, to generate first localization data defining a corresponding region of the image for the first candidate item; and processing the second encoder output token, using the localization subnetwork, to generate second localization data defining a corresponding region of the image for the second candidate item.

5. The method of claim 4, further comprising: storing the first localization data in association with the subject of the subject- predicate-object triplet; and storing the second localization data in association with the object of the subject- predicate-object triplet.

6. The method of any preceding claim, further comprising, prior to generating the relationship embedding based on combining the subject embeddings and the object embedding and / or prior to determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding: determining whether to generate the relationship embedding and / or the corresponding score.

7. The method of claim 6, wherein determining whether to generate the relationship embedding and / or the corresponding score comprises: processing the subject embedding and the object embedding using a relationship attention subnetwork to generate a predicted score, wherein the predicted score defines a likelihood that a relationship exists between the first candidate item represented by the subject embedding and the second candidate item represented by the object embedding; and determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score.

8. The method of claim 7, wherein determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score comprises: determining that the predicted score satisfies a threshold that is relative to other predicted scores for other subject embedding, object embedding pairs.

9. The method of claim 7 or claim 8, wherein determining, based on the predicted score, to generate the relationship embedding and / or the corresponding score comprises: determining that the predicted score satisfies a fixed threshold.

10. The method of any one of claims 6 to 9, further comprising: processing a third encoder output token using the subject subnetwork to generate an additional subject embedding representing a third candidate item, of the corresponding candidate items of the image, as a subject; processing a fourth encoder output token using the object subnetwork to generate an additional object embedding representing a fourth candidate item, of the corresponding candidate items of the image, as an object; and determining to not generate an additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining tonot generate a corresponding additional score for each of the predicate embeddings and for the additional relationship embedding.

11. The method of claim 10, wherein determining to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or determining to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding comprises: processing the additional subject embedding and the additional object embedding using the relationship attention subnetwork to generate an additional predicted score, wherein the additional predicted score defines a likelihood that a relationship exists between the third candidate item represented by the additional subject embedding and the fourth candidate item represented by the additional object embedding; and determining, based on the additional predicted score, to not generate the additional relationship embedding based on combining the additional subject embeddings and the additional object embedding and / or to not generate the corresponding additional score for each of the predicate embeddings and for the additional relationship embedding.

12. The method of any one of any one of claims 7 to 11, wherein the relationship attention subnetwork is previously trained based on: processing a prior subject embedding and a prior object embedding using the relationship attention subnetwork to generate a prior predicted score; generating a prior relationship embedding based on combining the prior subject embedding and the prior object embedding; comparing the prior relationship embedding to the predicate embeddings; determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the prior relationship embedding; selecting a maximum score of the corresponding scores;generating a loss based on comparing the prior predicted score to the maximum score; and updating the relationship attention subnetwork based on the loss.

13. The method of any preceding claim, further comprising: causing the given relationship to be rendered at a user interface output device of a client device associated with the image.

14. The method of claim 13, wherein the client device includes a camera via which the image is captured.

15. The method of any preceding claim, further comprising: processing the given relationship, using one or more generative models, to generate output indicative of one or more automated actions; and causing the one or more automated actions to be implemented.

16. The method of claim 15, wherein the one or more automated actions include one or more automated robotic control actions.

17. The method of any preceding claim, further comprising generating a scene graph that includes the relationship and that also additional relationships generated based on other subject embedding, object embedding pairs that are generated based on the image.

18. The method of claim 17, further comprising: processing the scene graph, using one or more generative models, to generate output indicative of one or more automated actions; and causing the one or more automated actions to be implemented.

19. The method of any one of claims 1 to 17, further comprising:processing, prior to obtaining the image and using a robot control policy, one or more preceding images, that are captured prior to the image, to generate robot control policy output that is indicative of one or more automated robotic control actions; causing the one or more automated robotic control actions to be implemented; and determining, based on the selected given relationship, a measure of success of implementation of the one or more automated robotic control actions, and / or determining, based on the selected given relationship, one or more next automated robotic control actions to implement.

20. A method implemented using one or more processors, the method comprising: processing an image using an image encoder to generate encoder output tokens; processing the encoder output tokens using a subject subnetwork to generate subject embeddings, each of the subject embeddings being for a corresponding one of the encoder output tokens and representing a corresponding candidate item, of the image, as a corresponding subject of a subject-predicate-object triplet; processing the encoder output tokens using an object subnetwork to generate object embeddings, each of the object embeddings being for a corresponding one of the encoder output tokens and representing a corresponding candidate item, of the image, as a corresponding object of a subject-predicate-object triplet; generating relationship embeddings, wherein generating each of the relationship embeddings comprises combining a corresponding unique pair of a corresponding one of the subject embeddings and a corresponding one of the object embeddings; for each of the relationship embeddings: comparing the relationship embedding to one or more predicate embeddings, wherein each of the predicate embeddings represents a corresponding relationship between items; and determining, based on the comparing, a corresponding score for each of the predicate embeddings and for the relationship embedding,wherein each of the corresponding scores defines a likelihood that the corresponding subject, represented by the corresponding one of the subject embeddings used in generating the relationship embedding, has the corresponding relationship, represented by the predicate embedding, to the corresponding object represented by the corresponding one of the object embeddings used in generating the relationship embedding.

21. The method of claim 20, wherein generating the relationship embeddings is performed for only a subset of the corresponding unique pairs and further comprising: selecting, from the corresponding unique pairs, the subset of the corresponding unique pairs.

22. The method of claim 21, wherein selecting the subset of the corresponding unique pairs comprises: for each of the corresponding unique pairs: processing the corresponding one of the subject embeddings and the corresponding one of the object embeddings, of the corresponding unique pair, using a relationship attention subnetwork to generate a corresponding predicted score, wherein the corresponding predicted score defines a likelihood that a relationship exists between the corresponding one of the subject embeddings and the corresponding one of the object embeddings; and selecting the subset of the corresponding unique pairs based on the corresponding predicted scores.

23. The method of claim 22, wherein selecting the subset of the corresponding unique pairs based on the corresponding predicted scores comprises: selecting a given quantity of the corresponding unique pairs that have the corresponding predicted scores most indicative of likelihood that a relationship exists.

24. The method of any one of claims 20 to 23, wherein the subset of the corresponding unique pairs is less than fifty percent, less than forty percent, less than thirty percent, less than twenty percent, less than ten percent, less than five percent, less than one percent, or less than two-tenths of a percent of the corresponding unique pairs.

25. The method of any one of claims 20 to 24, wherein combining a corresponding unique pair of a corresponding one of the subject embeddings and a corresponding one of the object embeddings comprises: performing an element-wise addition of the corresponding one of the subject embeddings and the corresponding one of the object embeddings.

26. The method of any one of claims 20 to 25, wherein comparing the relationship embedding to the one or more predicate embeddings comprises: performing a dot-product comparison of the relationship embedding to each of the one or more predicate embeddings; and wherein determining, based on the comparing, the corresponding score for each of the predicate embeddings and for the relationship embedding comprises: determining the corresponding score based on the result of the dot-product comparison.

27. The method of any one of claims 20 to 26, wherein the one or more predicate embeddings are each generated based on processing, using a text encoder, a natural language text sequence that describes the corresponding relationship between items that is represented by the predicate embedding.

28. The method of claim 27, wherein the image encoder and the text encoder are pre-trained jointly with one another.

29. The method of claim 28, wherein the image encoder and the text encoder are pre-trained in conjunction with one another by repeatedly performing operations comprising: obtaining: (i) a training image, (ii) a positive text sequence, wherein the positive text sequence characterizes the training image, and (iii) one or more negative text sequences, wherein the negative text sequences do not characterize the training image; generating an embedding of the training image using the image encoding subnetwork; generating respective embeddings of the positive text sequence and each of the negative text sequences using the text encoding subnetwork; and jointly training the image encoding subnetwork and the text encoding subnetwork to encourage: (i) greater similarity between the embedding of the training image and the embedding of the positive text sequence, (ii) lesser similarity between the embedding of the training image and the embeddings of the negative text sequences.

30. The method of any one of claims 20 to 29, further comprising: for each of the relationship embeddings: comparing the corresponding one of the subject embeddings, used in generating the relationship embedding, to one or more query embeddings that each represents a corresponding category of an item; determining, based on the comparing, a corresponding subject classification score for each of the query embeddings and for the corresponding one of the subject embeddings; comparing the corresponding one of the object embeddings, used in generating the relationship embedding, to the one or more query embeddings; determining, based on the comparing, a corresponding object classification score for each of the query embeddings and for the corresponding one of the object embeddings.

31. The method of claim 30, further comprising: determining a plurality of subject-predicate-object triplets based on:the corresponding scores, the corresponding subject classification scores, and the corresponding object classification scores.

32. A method implemented using one or more processors, the method comprising: obtaining a training data instance that includes a training image and one or more ground truth labels for the training image, the ground truth labels including a given ground truth label that includes: relationship text that describes a relationship, of a first item characterized in the in the image to a second item characterized in the image; processing the training image using an image encoder to generate encoder output tokens each representing a corresponding candidate item of the training image; processing a first encoder output token using a subject subnetwork to generate a subject embedding representing the first item; processing a second encoder output token using an object subnetwork to generate an object embedding representing the second item; generating a relationship embedding based on combining the subject embedding and the object embedding; generating a predicate embedding based on processing the relationship text using a text encoder; generating a predicate loss based on comparing the relationship embedding to the predicate embedding; and updating the subject network and the object network based on a loss that is based on the predicate loss.

33. The method of claim 32, further comprising updating the image encoder and the text encoder based on the loss.

34. The method of claim 32 of 33, wherein the given ground truth label further includes subject text that describes a first classification of the first item and object text that describes a second classification for the second item, and further comprising: generating a first classification embedding based on processing the subject text using the text encoder; generating a second classification embedding based on processing the object text using the text encoder; generating a first classification loss based on comparing the first classification embedding to the subject embedding or an additional embedding generated based on the first encoder output token; and generating a second classification loss based on comparing the second classification embedding to the object embedding or a further embedding generated based on the second encoder output token; wherein the loss is further based on the first classification loss and the second classification loss.

35. A system comprising memory storing instructions and one or more processors operable to execute the instructions to perform the method of any preceding claim.

36. A robot comprising a vision component, actuators, memory storing instructions, and one or more processors operable to execute the instructions to perform the method of any one of claims 1 to 31.

Images