Method for generating structured text describing image

Abstract

To provide a computer-implemented method for generating a structured text describing an image.SOLUTION: A method includes: a step of extracting an item from an image; a step of encoding the extracted item; a step of generating a domain embedding from a predicted domain of the image; a step of predicting a relationship between two items in the image by decoding the encoded extracted item and the domain embedding; and a step of classifying the two items and the predicted relationship to form a structured text as a triplet.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method, a computer program product and an information processing device for generating structured text that describes an image. [Background technology]

[0002] In the field of computer vision, image recognition has made incredible progress in recent years, and neural networks in particular have become popular for recognizing objects in images and generating structured text, such as scene graphs, from images.

[0003] Scene graph generation (SGG) can be used for tasks such as visual question and answer (VQA), image editing, and action recognition. However, scene graph recognition models are often trained with labeled images that are biased toward general object relations. That is, the labeled images often focus on simple descriptors such as "on," "in," and "by" to describe subject-object relations. The number of simple descriptors in large databases often far exceeds the number of meaningful, contextually rich predicates for describing object-subject pairs. The predictions of SGG models trained on such an imbalanced data distribution are biased toward simple, "coarse-grained" descriptions, resulting in less informative scene graphs.

[0004] While simple descriptors can correctly label object relationships, they provide little or no context to the image. This results in simple scene graph triplets that lack meaningful relationships between objects. These simple scene graphs are therefore of limited use in downstream tasks such as VQA, which require rich semantic representations.

[0005] Common research attempts to debias scene graphs by focusing on long-tail predicate distributions, i.e., more informative, 'fine-grained' descriptions such as 'riding on', 'selling', and 'standing on'. However, we have found that these approaches can disrupt overall model performance. For example, simple leading predicates in a distribution are often also present in more meaningful trailing predicates, such as 'on' in 'standing on'. Therefore, a scene graph model can be adversely affected by focusing on debiasing long-tail predicates or by overfitting or underfitting leading predicates. Therefore, it is desirable to generate contextually meaningful predicates using SGG methods. Summary of the Invention

[0006] The invention is defined in the independent claims, to which reference should now be made. Further features are set out in the dependent claims.

[0007] According to one aspect of the present invention, there is provided a computer-implemented method for generating structured text describing an image, comprising the steps of: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; The method may further include outputting the structured text. [Brief explanation of the drawings]

[0008] By way of example only, reference is made to the accompanying drawings in which: [Figure 1] 1 shows an example of a scene graph generated using methods known in the art. [Figure 2] 1 shows a block diagram of an exemplary method for generating a scene graph as known in the art. [Figure 3] 2 illustrates another method known in the art for generating a scene graph from an input image. [Figure 4] 1 shows a block diagram of a scene graph generation method disclosed herein; [Figure 5] 1 shows a flowchart of method steps for generating a scene graph. [Figure 6] Here is an example of the distribution of words in a domain: [Figure 7] 1 illustrates an example of a domain prediction module that may be used in a scene graph generation method. [Figure 8] 10 illustrates another example of a domain prediction module that may be used in the scene graph generation method. [Figure 9] 10 illustrates an example of a domain-conditional decoder that can be used in a scene graph generation method. [Figure 10] 10 shows an example of another domain-conditional decoder that can be used in the scene graph generation method. [Figure 11] 1 illustrates an example of Transformer domain coding that can occur in a domain-conditional decoder. [Figure 12] 1 illustrates an overall training approach for training the scene graph generation method disclosed herein. [Figure 13] An example of domain division and grouping is shown below. [Figure 14] 1 shows an example of a training method for training a domain predictor. [Figure 15] We present an example of an alternative training approach for training the domain prediction module. [Figure 16] 1 shows the overall processing flow of a scene graph generation method. [Figure 17] 1 illustrates the overall testing phase of the scene graph generation method disclosed herein. [Figure 18A] 1 shows an example of a semantically meaningful scene graph triplet generated using the method disclosed herein in contrast to existing methods. [Figure 18B] 1 shows an example of a semantically meaningful scene graph triplet generated using the method disclosed herein in contrast to existing methods. [Figure 19] 10 illustrates another example of generating a scene graph triplet using the methods disclosed herein. [Figure 20] 1 illustrates an example of a graphical user interface (GUI) for performing the methods disclosed herein. [Figure 21] FIG. 1 is a block diagram of an information processing device or computer system. DETAILED DESCRIPTION OF THE INVENTION

[0009] In recent years, machine learning has contributed greatly to the advancement of computer vision (CV). There is a demand for improving visual scene understanding in computer vision models for recognition and application tasks. A particular computer vision task is scene graph generation, where neural networks can receive input images and generate object relationships or attributes within the images.

[0010] In general, a scene graph is a graph that represents a particular combination of at least one object (or item) and at least one relationship and / or action (or "predicate" or "attribute"). For example, a scene graph may represent the objects in an image and their relationships to one another. That is, a scene graph may correspond to a logical / spatial representation of a graphical scene. Objects in a scene, such as dogs, people (i.e., common nouns), are formed from object nodes, and relationships, actions, or descriptors between objects, e.g., predicates such as "looking," "sitting," "playing," "black," and "tall," form predicate nodes that connect the object nodes or are connected to a single object node. In some scene graphs, nodes are assigned only to objects, and predicates are labels along the edges connecting the objects.

[0011] A scene graph may be a directed graph where nodes represent objects and edges represent pairwise relationships between the objects. An example of two objects and a pairwise relationship is "a man playing with a dog", where man and dog are object nodes and the pairwise relationship or predicate is "playing with". Another example is "a man looking at a dog", where the predicate is the term "looking at". To indicate this relationship in a scene graph, an arrow can be drawn from the man to the dog (or in any other way used in other notations) to indicate that the man is doing the "looking", and not the other way around.

[0012] Additionally or alternatively, a scene graph can be decomposed into structured text, such as triplets, that describe object integration. For example, an image containing two items may be given as input. One item may form the subject of the image, and another item may form the object of the image. Triplets that describe subject-object interactions may be of the form <subject, predicate, object>. That is, the predicate may indicate a pairwise relationship between objects, as described above. For example, a triplet may describe a human-object interaction (HOI):

[0013] Thus, a scene graph is a structural representation that captures detailed semantics that model relationships (referring to actions or global predicates) between objects ("man", "dog", "car"), object attributes or local predicates ("man is tall", "car is red"), and paired objects ("man sitting in car"). The elements of a scene graph are thus object nodes and predicate nodes (or in some cases predicate edges).

[0014] Objects or subjects of a scene graph can be surrounded by a bounding box that delimits the object nodes. Each object can have zero, one, or more attributes or local predicates such as color, state (e.g., sitting, looking), material (e.g., metallic, plastic), and / or emotion (e.g., smiling, crying). Additionally or alternatively, objects may have relations (or actions) such as "jumping over" or "sitting on" and these relations may be spatial ("in front of," "behind"), or may involve the preposition "with," the comparative "taller than," the descriptive verb "strolled," etc. Thus, a scene graph can be organized as a set of "object, relation, object" (<object, predicate, subject> can be thought of as a set of visual relation triplets of

[0015] It is desirable to design a scene graph generation method that can enable more meaningful contextual relationship prediction between objects, and the inventors have surprisingly found that this can be achieved with little computational and training overhead using domain embedding knowledge specific to each domain.

[0016] Figure 1 shows an example scene graph 100 generated using methods known in the art. For example, a transformer-based image encoder can be provided with an input image 105 to encode the image based on the output of an object detection unit. In this example, a pre-trained decoder is used to predict objects and their relationships.

[0017] We found that known methods train decoders on large datasets containing biased data. For example, model predictions are biased toward common relationships (like, on, in, has) that have little meaning. Transformer-based neural networks have shown promise for visual scene understanding, but are computationally intensive and cumbersome to train. For example, we found that the performance of transformer models is limited and requires computationally intensive retraining to include new data distributions.

[0018] Figure 1 shows an input image 105 of a man and motorcycle in a parking lot. The input image is fed into a Transformer model, which can recognize the bike and the motorcycle. The model uses a convolutional neural network to recognize objects in the image. The output image 110 shows bounding boxes 115, 120 around the man and motorcycle. That is, in this simplified overview, the decoder unit can recognize the man and motorcycle in the image and generate the bounding boxes. The figure also shows an example output triplet 125 of the Transformer-based model. The determined prediction given by the model is<human, on, bike> This prediction is correct, but it makes no sense given the image context. The desired output 135 or detected target is instead<man, parking, bike> The inventors have discovered that conditioning a neural network model on domain-specific context-based information can lead to meaningful relationship predictions between objects. That is, some additional information can be used to explain different scene types (or meanings) to generate meaningful relationship predictions.

[0019] FIG. 2 shows a block diagram of an exemplary method 200 known in the art for generating a scene graph. The exemplary method is described, for example, in Yang, J., Ang, Y.Z., Guo, Z., Zhou, K., Zhang, W., and Liu, Z., 2022, October. Generating Panoptic Scene Graphs. European Conference on Computer Vision (pp. 178-196). Cham: Springer Nature Switzerland. An image 205 is input into a scene graph generation (SGG) model, and features are extracted by a feature extractor. For example, image features, such as object words, are extracted from a CNN backbone such as the FASTER-RCNN 'off-the-shelf' detector.

[0020] The extracted features are input to a Transformer Encoder-Decoder model. As shown in Figure 2, the extracted features are input to a Transformer Encoder 215, and the output of the encoder is input to a decoder. The diagram shows a predicate decoder 220. The general model also includes an object decoder (not shown). The extracted features are fed to a Transformer model (called the query learning block), which uses queries to represent triplet information. The output from the decoder is input to a feedforward network trained using triplet matching.

[0021] This known method uses a transformer-based image encoder and separate object and predicate decoder blocks. However, this method does not use an additional context inference-based module. We found that this method cannot consider context predicates, and the model is biased toward general predicates, e.g., prediction: Man on bike, ground truth: Man riding on bike. This model picks up data bias due to the large imbalance in the label triplets used to train the model, and requires some additional information to correctly identify predicates. Furthermore, a new domain distribution requires retraining the entire network.

[0022] 3 shows another method 300 known in the art for generating a scene graph from an input image. This method is described in Zheng, C., Gao, L., Lyu, X., Zeng, P., El Saddik, A. and Shen, HT, 2023. Dual-branch hybrid learning network for unbiased scene graph generation. IEEE Transactions on Circuits and Systems for Video Technology also uses a quasi-feature extractor 310 (such as Faster R-CNN), a transformer encoder 315, and a predicate decoder 1320.

[0023] This known method further uses a context module 316, which receives detected object and predicate representations as input and combines local and global semantic representations of predicted objects and predicates to capture correlations between relational triplets. The context module has an object decoder 317 that receives input from the transformer encoder 315 and outputs all predicted object representations. The context module transformer encoder 318 receives as input all predicted object representations along with all predicted predicate representations from predicate decoder 1. Predicate decoder 2 319 generates context representations based on the input from the context module transformer encoder. The context representations are used to correct "out-of-context" predictions. Triplet matching 325 is used to generate scene graph triplets with refined predicates.

[0024] The inventors discovered that while the context module captures existing relational correlations, it does not capture domain-discriminating features. That is, the model is limited by the information in the image and does not use additional information (such as text or other labels) to train the context module. The context module extracts context using only information from a single input image. For example, to find global information about the "context," they encode all 10 objects in the image along with their predicted predicates. The method then concatenates the context prediction with the initial image prediction to determine a confidence level. Because only information from the image is used, the model does not learn new information and effectively uses the same information twice: in the predicate decoder 1 and the context module. Therefore, the inventors discovered that this method fails to generate a meaningful scene graph describing the input image.

[0025] This known method does not predict the domain of the image, but simply reuses the predicted predicates of the context, ignoring differences in sample distributions, and extracts context information from triplet relational correlations. In other words, the context module is trained to consider long-tail predicate distributions. Therefore, this method does not have a predicted domain. Instead, the context is derived from the predicted scene graph of the image, which does not solve the problem of SGG methods of capturing simple predicates. The context module does not capture domain-specific predicates. When generating a "semantic representation," the problem of determining semantically meaningful predicates remains unsolved because the context module continues to consider predicates that are unrelated to the context (or domain).

[0026] Furthermore, the entire network needs to be retrained for new domain data, and is therefore computationally intensive.

[0027] 4 shows a block diagram of the scene graph generation method disclosed herein. An input image 405 is input into the scene graph generation model. In this example, the image is of a man and his motorcycle in a parking lot.

[0028] Image features, such as items within an image, can be extracted. Image features can be extracted using a feature extractor 410. For example, the feature extractor used by the present inventors was a ResNet-50 feature extraction module (He, K., Zhang, X., Ren, S. and Sun, J., 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778)). The feature extractor can output an embedding, or feature map, of objects detected in the image. For example, the feature extractor can provide embeddings with dimensions such as (number of output channels, output height, output width). Of course, other feature extractors may be used. For example, any suitable convolutional neural network or residual neural network may be used. For example, Faster R-CNN, known in the art, may be used.

[0029] The extracted features, which may be the output of the feature extractor, may be input to the image encoder 415. The feature extractor may output each feature as an embedding of dimensions (number of output channels, output height, output width). The embedding may have a total number of (height x width) feature vectors / tokens fed to the encoder. The output shape of the encoder may change the number of output channels to match the embedding dimensions.

[0030] The image encoder may take the extracted features, e.g., the embedding of the feature extractor, and encode the input to generate another embedding (called an encoder embedding). That is, the image encoder may take the extracted items from the feature extractor as input and encode the extracted items. The image encoder we used was a pre-trained image encoder. Any suitable image encoder suitable for generating a scene graph may be used. We used a QPIC transformer model (image encoder and decoder) and pre-trained weights (first discussed in Tamura, M., Ohashi, H. and Yoshinaga, T., 2021. Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 10410-10419)). The transformer encoder and decoder described in the QPIC model were adapted from the known DETR discussed in (Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A. and Zagoruyko, S., 2020, August. End-to-end object detection with transformers. In European conference on computer vision (pp. 213-229). Cham: Springer International Publishing).

[0031] The image encoder may output the encoded items as encoder embeddings. The embeddings may be provided as input to a domain predictor 420 (described in connection with Figures 7 and 8 below). The domain predictor may be a neural network. For example, the domain predictor may be a multi-layer perceptron neural network. The domain predictor may take input from the image encoder and generate (or compute) a domain embedding. Alternatively, a large-scale language model (LLM) 422 may be used to generate the domain embedding. For example, the LLM may take domain text (e.g., shopping) as input and generate a domain embedding from that text. An example of an LLM that can be used is the "Bidirectional Encoder Representations from Transformers" or BERT, LLM, known in the art.

[0032] If the domain (or scenario) of the image is known before generating the scene graph, the embeddings can be generated using a large-scale language model. For example, if the scene graph is used for airport surveillance, the domain text for the LLM can be input (or generated) as general text describing an airport.

[0033] In examples where the domain (or scenario) of an image is unknown before generating a scene graph, embeddings can be generated directly from the image by capturing contextual information using domain prediction. For example, if a user takes a random photo with a mobile device and inputs it into a scene graph generation model, the contextual information or description may be unknown. The context of an image can be derived from items within the image. The predicted domain of an image can be generated from global information of the image. For example, the global information may include the image's head token (sometimes called a class token) generated by the image encoder. As described in connection with Figure 13, the context or domain can include a predicate distribution specific to that domain. Therefore, by capturing contextual information, more meaningful predicates can be selected or derived from the predicates in the predicted domain.

[0034] The domain or domain name can include contextual information for the input image. For example, the domain can include the setting of the event, a statement describing the image, the scene, the type of image, the environment captured by the image, the style of the image, etc. The domain can capture the location where the image was taken, for example, a shopping mall, an airport, a parking lot.

[0035] The domain embedding generated by the domain prediction or LLM and the encoded image generated by the image encoder can be input or provided to a domain-conditional decoder 425. For example, the domain-conditional decoder can be based on a QPIC decoder but can have weights (described below) adjusted from training. That is, the domain-conditional decoder can be a transform decoder that specifically learns features to focus on according to a given task (here, triplet prediction) and provides more appropriate features (prediction predicates).

[0036] The domain-conditional decoder may predict all object and predicate embeddings based on the conditioning provided by the domain embedding. The domain-conditional decoder may generate predicate embeddings from the image encoder embeddings and combine them with the domain embeddings. For example, the predicate embeddings and the domain embeddings may be combined by concatenation. Alternatively, the domain-conditional decoder may use the domain embeddings with an attention mechanism to generate extended predicate embeddings (see Figures 9, 10, and 11 below).

[0037] The extended predicate embeddings may be input to a classifier 430 to generate structured text in the form of triplets. The structured text triplets may be scene graph triplets. The extended predicate embeddings may be extracted from the decoder output and classified using a classifier. The classifier may include a separate unit or head for each of the box coordinates, i.e., the bounding box around the object, and a class label for each object, predicate, and subject. The class labels may be combined to form a triplet prediction, resulting in a final prediction.

[0038] The classifier may include a feedforward network. For example, we used a three-layer multilayer perceptron (MLP) NN with a ReLU activation function and hidden dimension d, along with a linear projection layer. The FFN may predict bounding box coordinates by calculating the normalized center coordinates, height, and width of the box relative to the input image. Furthermore, the linear layer may predict each class label (subject, predicate, object) by using a softmax function. Thus, each output of the classifier can be calculated as, for example, the object coordinate, as well as the object coordinate.<subject, predicate, object> In the form of a triplet, one combination of subject, object, and predicate class can be given. We used five individual classifiers (i.e., five FFNs) for the bounding boxes, subject and object class labels, and predicates. Each of the "individual classifiers" can be considered as a classifier of a (single) classifier unit that encompasses all classifiers. As mentioned above, each output of the five classifiers can give one or more combinations of triplet pairs (because one subject can be paired with two different objects).

[0039] As shown in the figure, the proposed method may use a triplet matching unit 435. For example, the triplet matching unit may be used in the training phase. The triplet matching unit may not be present in the implementation phase. The training phase is described in more detail with reference to FIG. 12. In the training phase, the classifier may generate multiple predictions for a triplet using multiple output embeddings from the decoder. In the triplet matching step, the predicted triplets generated by the classifier may be matched to the ground truth triplet because there may be multiple correct triplets in the ground truth, one of which is selected for each predicted triplet. Known methods, such as the Hungarian algorithm, may be used for triplet matching with the ground truth and then used to train the classifier. The Hungarian algorithm may be used to match the predicted triplet to the ground truth triplet. The loss between the two matched triplets may then be calculated, and the calculated loss may be used to adjust the classifier weights and the decoder weights in the training phase.

[0040] Therefore, in the method disclosed herein, given an image, the SGG method can first predict the domain from the image features using the neural network block. Then, the method can train the predicate decoder block on the domain representation to predict meaningful predicates belonging to that domain. Therefore, our method can capture semantically meaningful scene graph relationships using unique domain embeddings for images belonging to different domains. Furthermore, this model requires little overhead computation to learn the domain representation using the domain prediction module.

[0041] As mentioned above, a scene graph can represent items in an image and their interrelationships (or attributes). That is, a scene graph can correspond to a logical / spatial representation of a graphical scene. In this example, the scene graph generation method generates a triplet<human, parking, bike> Given an input, this method generates a semantically meaningful scene graph, which can improve downstream tasks such as visual question and answer (VQA), action recognition, and image editing.

[0042] Although the output from the scene graph method is shown as triplets in this example, the method can additionally or alternatively output a labeled scene graph in the form of a bounding box around each object in the image, with connecting nodes with labels (sometimes called labeled links) between the objects. An image may be given in the output with the scene graph overlaid on it, or the scene graph may be generated without an image in the output.

[0043] The proposed method for generating scene graphs disclosed herein can provide more semantically meaningful predicates. Our method uses additional domain knowledge to refine predicate prediction. This domain knowledge is trained to be unique for each domain predicate, thus overcoming the bias towards common predicates.

[0044] FIG. 5 shows a flowchart of the steps of a method 500 for generating a scene graph.

[0045] In the extraction step s510, items can be extracted from the input image. For example, an image capturing an item can be input into the scene graph generative model disclosed herein, and a feature extractor can be used to extract the item. We used a ResNet-50 feature extractor. ResNet-50 is a 50-layer convolutional neural network (48 convolutional layers, 1 MaxPool layer, 1 average pooling layer). Of course, any suitable feature extractor may be used.

[0046] Input image, e.g. RGB color input image:

[0047]

number

[0048]

number

[0049] In the encoding step s520, the extracted items may be input to an image encoder. The image encoder may form the encoder block of a transformer encoder-decoder module. The image encoder may receive the extracted items, for example, in the form of feature maps, and encode the extracted items. The items may be encoded by generating augmented feature maps using context information. For example, the image encoder may augment the feature maps using a self-attention model. As mentioned above, in one example, we used pre-trained weights from a QPIC image encoder. The image encoder may output the encoded items as embeddings (which may be referred to as encoder embeddings).

[0050] In the generating step s530, a domain embedding may be generated from the predicted domain of the image. The encoder embedding may be provided as input to domain prediction (described in connection with Figures 7 and 8 below). The domain prediction may be a neural network such as a multi-layer perceptron neural network. The predicted domain of the image may be generated from global information of the image. For example, the global information may include head tokens (sometimes called class tokens) of the image generated by the image encoder.

[0051] Alternatively, the generation step may use a large-scale language model (LLM) to generate the domain embeddings. If the scenario (or domain) of the images is known before generating the scene graph, the embeddings can be generated using a large-scale language model.

[0052] In the providing step s540, the encoded extracted items and domain embeddings may be input to a decoder configured to predict a relationship between two items in an image. The domain-conditional decoder may form the decoder block of a Transformer Encoder-Decoder. As described above, the inventors used weights from the QPIC decoder and then further trained the decoder to take the domain embedding as input. The decoder may be configured to predict a relationship between two items in an image by generating a predicate embedding from the encoded image, concatenating the domain embedding and predicate embedding to generate an extended predicate embedding, and predicting attributes from the extended predicate embedding. That is, the QPIC decoder architecture can be used to predict an initial predicate embedding that can be combined with the domain embedding to generate an extended predicate embedding.

[0053] Alternatively, the decoder may be configured to predict relationships between two items in an image by using a self-attention mechanism on a conditioned learnable query on a domain embedding, inputting the conditioned learnable query and the encoded items into a mutual attention mechanism to generate an extended predicate embedding, and predicting attributes (relationships) from the extended predicate embedding. That is, the domain-conditioned decoder can convert a set of learnable query vectors into a set of embeddings conditioned on the domain embedding (see Figures 10 and 11). In this example, the extended predicate (or domain) embedding can be generated directly without first generating an initial predicate embedding.

[0054] In the classification step s550, the two items and the predicted attributes may be classified to form structured text as triplets. For example, extended predicate embeddings may be input to a classifier to generate scene graph triplets. That is, extended predicate embeddings may be extracted from the decoder output and classified using the classifier. The classifier may include a separate unit, or head, for each box coordinate. That is, both the subject and object have a class label and a bounding box prediction. The class labels may be combined to form a triplet prediction, resulting in a final prediction. The classifier may further generate bounding box coordinates for the two items and / or each item in the image.

[0055] FIG. 6 shows an example of the distribution of words in a domain. For example, the original distribution graph 605 illustrates the long-tail problem experienced in the art. A training set containing images with all different contexts can be used to train a scene graph model. For example, the HICO-DET dataset known in the art can be used. However, the dataset containing labeled triplets (used in training) is biased toward simple relationships such as "on," as shown in the figure. Therefore, when using a scene graph model trained with biased data, the predicted triplets are also biased, leading to predicates that provide little context. Furthermore, including more information, such as more images, in the dataset requires retraining the entire model. This is a computationally intensive process, resulting in large computational overhead. Methods known in the art (as described above in connection with FIGS. 2 and 3) use distributions like the original distribution, thus resulting in predicates that provide no context for the image or the scene within the image.

[0056] Context domain 610 (labeled "street domain" in the figure) shows an example of a domain distribution determined by the inventors. The inventors identified a need to identify an underlying distributional representation without affecting the overall performance of the model. The domain distribution shown in the figure is for the "street domain." The graph shows the distribution of words in the street domain, such as "parked on," "standing on," and "walking on." Words can be manually classified into domains. For example, each domain may correspond more to certain words than others. Words can be identified by a domain module after training. Thus, a neural network, such as an LLM, can be trained to identify words and classify them into domains.

[0057] FIG. 6 also illustrates an example of a unique domain embedding, generated, for example, by domain prediction. As shown, an image of a man riding a motorcycle 615a may be provided as input to the scene graph generative model. Alternatively, an image of a man shopping on a motorcycle 615b may be provided as input. Predicate embeddings 620a, 620b for each of the images may be generated by a predicate decoder (e.g., a predicate decoder that is part of a domain-conditional decoder). The predicate embeddings for each image may be the same. That is, the predicate decoder may recognize the man riding a motorcycle in both images and therefore use this information to generate the embeddings. The embeddings may represent words or concepts associated with the images. Because both sample images include a man riding a motorcycle, the generated embeddings may be the same for both images. Of course, if other sample images not related to motorcycles were input, the generated predicate embeddings may be different.

[0058] For each image, a unique learned domain embedding 625a, 625b can be generated. For example, for an image of a man riding a motorcycle, the domain may be related to roads. Domain-conditional prediction may generate or predict unique domain embeddings for different domains based on each input image. The domain embedding for an image of a man riding a motorcycle shopping may be related to "shopping." Thus, a domain embedding can be generated from the domain text "shopping."

[0059] The predicate embedding and the uniquely learned domain embedding can be adjusted to generate domain-conditional predicate embeddings 630a, 630b. That is, the predicate information can be updated using the predicted domain embedding (and requires little computational overhead). For example, the predicate embedding and the unique domain embedding can be concatenated to generate a domain-conditional predicate embedding (which can be called an extended predicate embedding). Domain learning is shown here as predicates added to provide an extended range of predicates that can be selected according to probability.

[0060] 7 shows an example domain prediction module 700 that may be used in the scene graph generation method. The domain prediction module may be referred to as "Domain Prediction Module 1" or "Domain Prediction Block 1."

[0061] In this example, a user may input generic text that describes a use case scenario for an input image 705. The image encoder 715, domain conditional decoder 725, and predicate classifier 730 may be the same as, or substantially the same as, the image encoder, domain conditional decoder, and predicate classifier described above in connection with Figure 4. This example scene graph generation method may include additional functional blocks, such as a feature extractor and triplet matching (not shown).

[0062] The general text may be, for example, a domain name 723 input by a user into a pre-trained large-scale language model (LLM). The domain name may represent the context of the image, such as "shopping," "driving," "airport," etc. For example, the user may interact with a graphical user interface (GUI) to input the domain of the image. Alternatively, the user may be presented with a fixed list of domains to choose from, and / or the LLM may detect the domain from the image, and the user may confirm that the detected domain is correct.

[0063] The large-scale language model may receive an input domain name and generate a domain embedding. The domain embedding may be a dimension-256 embedding. That is, the embedding may be a dimension-256 vector. The output domain embedding may be input to a domain-conditional decoder block 725 to generate a domain-conditional predicate embedding (which may also be called an extended domain embedding or an extended predicate embedding).

[0064] 8 shows an example 800 of a domain prediction module 820 that may be used in the scene graph generation method. The domain prediction module may be referred to as "domain prediction module 2" or "domain prediction block 2."

[0065] The image encoder 815, the domain conditional decoder 825, and the predicate classifier 830 may be the same as, or substantially the same as, the image encoder, the domain conditional decoder, and the predicate classifier described above. As mentioned above, the scene graph generation method of this example may include further functional blocks such as a feature extractor and triplet matching.

[0066] In this example, the image encoder can output a transformer-coded embedding. That is, the image encoder may be a transform encoder that generates an embedding from an input image or input from a feature extractor. The transformer-coded embedding may include a patch token and a head token (sometimes otherwise referred to as a class token). The head token, which is commonly used for classification purposes, can be extracted by a domain prediction module.

[0067] The head token, which represents global / contextual information, can be fed into a domain prediction module, which outputs an embedding with domain knowledge. Generally, a head token (or class token) is a randomly initialized token that is prepended to an embedding (or prepended to a patch token). Because the head token may be randomly initialized, it may not initially contain any useful information. However, an image encoder receives the randomized head token as input and accumulates information from other tokens in the embedding. The head token can then collect information from all patches, for example, using multihead self-attention (MSA). The head token can then be used for classification.

[0068] In this example, the height domain prediction may be a neural network, such as a multi-layer perceptron (MLP) neural network. The inventors have found that a three-layer MLP module is effective for generating domain embeddings. Of course, any other suitable neural network may be used. For example, a different feed-forward neural network may be used. Furthermore, the MLP may have fewer or more than three layers.

[0069] The domain prediction disclosed herein may take a head token of dimension 256 as input and generate a domain embedding of dimension 256. That is, the input dimension size and the output dimension size may be equal. Furthermore, the hidden layer size of the neural network may be equal to the input dimension and the output dimension (thus, in this example, may be 256). The domain prediction may generate a single domain embedding from the input. For example, there may be a single word / sentence corresponding to a domain name. Each domain may have multiple predicates (which may be different from the domain name). For example, the domain name may be "driving" / "busy street driving", and the predicates within that domain may be "riding, parking, wearing, etc."

[0070] 9 shows an example 900 of a domain-conditional decoder 925 (sometimes referred to as domain-conditional decoder 1) that can be used in the scene graph generation method. The domain-conditional decoder can receive input from an image encoder. Furthermore, the domain-conditional decoder can receive input from either domain prediction module 1 or domain prediction module 2.

[0071] The domain-conditional decoder may include a predicate decoder and an object decoder (individual decoder blocks not shown). The predicate decoder and object decoder may be grouped to form the decoder 927 shown in the figure. The decoder may receive input from an image encoder and output all object and predicate embeddings from the encoded image.

[0072] The domain embedding obtained from the domain prediction can be added to the predicate embedding to condition them on the predicted domain. In this example, the predicate embedding can be concatenated with the domain embedding. That is, as shown in Figure 6, the predicate embedding and the domain embedding can be concatenated to form a single extended predicate embedding. In this example, the image decoder output can be a 256-dimensional embedding space vector, and the domain prediction output can be a 256-dimensional embedding space vector. Therefore, the domain-conditional decoder can output a 512-dimensional embedding space vector by concatenating the image decoder output and the domain prediction output. The attributes of two items in the input image can be predicted from the extended predicate embedding.

[0073] Figure 10 shows an example 1000 of a domain-conditional decoder 1025 (sometimes referred to as domain-conditional decoder 2) that can be used in the scene graph generation method. The domain-conditional decoder can receive input from an image encoder. Furthermore, the domain-conditional decoder can receive input from either domain prediction module 1 or domain prediction module 2. The domain-conditional decoder may form the decoder block of a transformer encoder-decoder. The image encoder can, for example, form the encoder block.

[0074] In this example, an attention mechanism, such as a combination of self-attention and cross-attention, can be used to generate the extended predicate embedding (the attention mechanism can form part of the Transformer architecture). The domain embeddings generated by the domain prediction 1020 can be input (or provided) to the decoder module 1027 as an additional query (domain query) in addition to the randomly initialized query (entity query). The embeddings generated by the image encoder can be input as key and value vectors.

[0075] The entity query (vector) can be conditioned on the domain query (vector). The entity query corresponds to the encoded image key and can be used to calculate an attention score. The attention score can be calculated using known methods such as determining a softmax function. The entity query can be updated by adding the encoded image value based on the attention score. The decoder module can output the final entity query (i.e., the extended predicate embedding) as a triplet embedding or all object and predicate embeddings. Details of the attention mechanism are provided in connection with Figure 11 below.

[0076] FIG. 11 shows an example of a transformer architecture that can be configured within the image encoder and domain-conditional decoder 2. The image encoder can generate an embedding 1105 from an input image. For example, the image encoder can receive an input image or a feature map generated by a feature extractor and generate 18x20 image patch tokens. Each image patch token can have a dimension of 256. Each token can be input to a self-attention mechanism block 1120 (sometimes referred to as the image encoder self-attention block), and a self-attention score can be calculated using known methods. For example, in the self-attention mechanism, every input vector is assigned three representations: a key, a query, and a value. The weights of the key, query, and value representations can be randomly initialized using an appropriate random distribution, such as a Gaussian distribution, a Xavier distribution, or a Chiming distribution. The initialization can be performed once before training the self-attention mechanism.

[0077] A known method for calculating a self-attention score can include the following steps: A dot-product attention is calculated as a score function; a softmax is taken for each attention score; a weight value can be calculated by multiplying the softmax attention score with a value representation; the weight values ​​can then be summed element-wise to generate an output; if there are multiple input vectors, the process is repeated using each input vector to generate the output.

[0078] The self-attention mechanism can also be used for domain embedding, for example, in the domain-conditional decoder block. Domain prediction can generate a 256-dimensional domain embedding. We input the domain embedding along with 100 random query vectors, each of 256 dimensions, into the self-attention block 1125 (which can be called the domain prediction self-attention block). The domain query was used to condition the entity query. That is, the self-attention block can condition the entity query on the domain query. Self-attention between the domain embedding and the query vector ensures that domain knowledge is induced in the query.

[0079] The outputs from the image encoder self-attention block and the domain prediction self-attention block can be input to the cross-attention block 1130. The cross-attention block can be configured within the domain-conditional decoder. Similar to the self-attention mechanism described above, the cross-attention mechanism uses key (K), query (Q), and value (V) matrices. However, unlike the self-attention mechanism in which K, Q, and V are derived from a single sequence, the cross-attention mechanism uses multiple sequences. The output from the image encoder self-attention block is projected onto the key (K) matrix and the value (V) matrix, and the output from the domain prediction is projected onto the query (Q) matrix. The attention score can be calculated by applying a softmax function to Q, K, and V using a normalization constant. We found that cross-attention between the image encoder embedding and the query captures contextual image features. In this example, the cross-attention block is shown to output a domain-conditional decoder embedding (extended predicate embedding). The extended embedding can have the same dimension as the query matrix. In this example, the expanded embedding has dimension 256.

[0080] 12 shows the overall training approach for training the scene graph generative model disclosed herein. The training approach can include two phases 1201, 1202 for training the method.

[0081] In the first phase 1201, each of the feature extraction unit, image encoder unit, decoder unit, and classifier unit is loaded with or obtained with training data (S1210). The image encoder unit and decoder unit can, for example, form a transformer encoder and decoder unit. The feature extraction unit and image encoder can be pre-trained models. For example, the feature extraction unit and image encoder can use pre-trained weights from the QPIC model by Masato Tamura et al. The feature extractor used by the inventors was the ResNet-50 feature extractor. Of course, other feature extractors can be used. In this first phase, the decoder can be loaded with pre-trained weights from the QPIC model. The inventors later modified the decoder weights in the training phase (see below). The training data can be obtained from the HICO-DET dataset. Of course, other suitable training data, such as the COCO dataset, can also be used. For example, any image with labeled ground truth triplets can be used in the training set.

[0082] In an encoding step s1220, the output from the feature extractor may be encoded using an image encoder. The image encoder may generate an embedding for the input image. For example, the image encoder may use a self-attention mechanism to generate a feature map for the image. The feature extractor may output a feature map, and the image encoder may take this feature map as input and generate another feature map with richer contextual information (i.e., using self-attention).

[0083] In the prediction step S1230, a domain prediction block may be used to generate predicted domain embeddings from the input by the image encoder. The predicted domain embeddings may be used to train a domain prediction module. For example, the domain prediction module may include a multi-layer perceptron (MLP). While the inventors used a three-layer MLP, other suitable neural networks, such as other feed-forward neural networks, may of course be used. The domain prediction module may be trained using ground truth domain labels of the input image. The domain prediction module may be domain prediction module 2 described in connection with FIG. 8. Methods for training domain prediction are described in connection with FIGS. 14 and 15 below. For example, the predicted domain embeddings may be passed through a linear layer to classify the domain (s1240) (see FIG. 14). Alternatively, the domain name may be passed to a large-scale language model (LLM), and a similarity score may be calculated together with the predicted domain embedding.

[0084] Alternatively, the domain prediction module may include a large scale language model LLM, such as domain prediction module 1 described in connection with Figure 7. The LLM may be a pre-trained LLM that takes a domain name as input and generates a domain embedding.

[0085] In a calculating step s1260, the error between the predicted domain embedding and the ground truth may be calculated, for example, a loss function between the predicted domain embedding and the ground truth may be calculated, as described in connection with Figures 14 and 15.

[0086] In an optimization step s1270, the weights of the domain prediction module may be updated. The training method may use the AdamW optimizer (Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, May 2019) to train the weights. The training method may include gradient descent to minimize the loss function. We used a batch size of 16 and an initial learning rate of 10 for the backbone network. -5 , and other initial learning rates of 10 -4 , and weight decay 10 -4 The model was trained for 50 epochs using the AdamW optimizer. Thus, the weights of the domain prediction module can be set when the error of the loss function is merged for a fixed number of epochs.

[0087] In a second phase 1202, the predicate embeddings from the decoder can be used by a predicate classifier to classify the predicate output. The weights of the predicate classifier can be trained using a small number of image and scene graph training samples taken from the domain. For example, similar to the first phase, training data and a pre-trained feature extractor, image encoder, decoder, and classifier can be loaded along with a domain prediction module (s1205). Further, similar to the first phase, input images can be encoded using an image encoder (s1215), and domain embeddings can be generated using the domain prediction module (s1225).

[0088] In generating step s1235, the domain-conditional decoder may generate an extended predicate embedding conditioned on the domain embedding. For example, the domain-conditional decoder may generate an extended predicate embedding (which may be referred to as an extended domain embedding) by combining the predicate embedding generated from the image encoder and the domain embedding generated from the domain prediction. The generated extended predicate embedding may be input to a predicate classifier (which may also be input to an object classifier and a subject classifier, not shown; the object and subject classifiers may take input directly from the domain-conditional embedding).

[0089] In classification step s1245, a predicate classifier can classify predicates from the expanded predicate embeddings. For example, the classifier can take the decoder output embeddings and classify them into object bounding box coordinates, object classes, and predicate classes. The classifier may include a feedforward neural network. Thus, each output of the classifier can be a predicate, e.g., a scalar, as well as an object coordinate.<subject, predicate, object> One combination of subject, object and predicate class can be given in triplet form.

[0090] In triplet matching step s1255, the predicted triplets generated by the classifier may be matched to the ground truth triplets, since there may be multiple correct triplets in the ground truth, one of which is selected for each predicted triplet. Known methods, such as the Hungarian algorithm, may be used for triplet matching with the ground truth.

[0091] In a calculating step s1265, the error between the predicted triplet and the ground truth may be calculated. For example, a Hungarian algorithm may be used to determine the matching cost, and a loss function may be calculated for each pair of predicted triplet and ground truth triplet.

[0092] In an optimization step s1275, the weights of the domain-conditioned decoder and classifier may be updated. The training method may use backpropagation and the AdamW optimizer (as described above) to train the weights. The training method may include gradient descent to minimize a loss function. For example, after Hungarian matching, some loss between the prediction and the matched ground truth may be estimated. The loss could be the cross-entropy between the predicted verbs or a focal loss. We used a batch size of 16 and an initial learning rate of 10 for the backbone network. -5 , other learning rate 10 -4 , and weight decay 10 -4 The model was trained for 50 epochs using the AdamW optimizer. Thus, the weights of the domain-conditional decoder prediction module and classifier can be set when the loss function errors are merged for a certain number of epochs. The classifiers can be trained separately for each domain (see Figure 13). That is, for each domain, e.g., kitchen, driving, shopping, etc., the classifier weights can be trained only on training samples of images and scene graphs taken from that domain.

[0093] FIG. 13 shows an example of domain division and grouping 1300. Domain division can be determined manually by a user and / or can be determined using a trained neural network. In this example, all samples 1305 can be divided into domains, the contents of which collectively describe a scenario. "All samples" are shown as being categorized into the domains "samples with objects from the driving scenario" 1307, "samples with objects from the kitchen scenario" 1308, and "samples from other scenarios" 1309.

[0094] A sample with objects from a driving scenario may include predicates related to the driving scenario, such as "man riding motorcycle," "man walking dog," and "man washing cycle." Thus, predicates in the driving-related domain may be "walk," "race," "park," "wash," "repair," "operate," "turn," "push," and "hop on." Objects in the driving domain may be "motorcycle," "bicycle," "horse," "train," "traffic signal," etc. Of course, other objects and predicates may be classified in the driving domain.

[0095] Similarly, a sample containing objects from a kitchen scenario may contain predicates related to kitchen scenes, such as "man holding knife," "man feeding dog," "man cleaning bowl," etc. Thus, predicates in the kitchen domain are "eat," "drink," "clean," "lick," "squeeze," "repair," "fill," "cut," "open," "peel," "cook," and objects are "bowl," "knife," "pizza," "refrigerator," etc. Of course, other objects and predicates can be classified in the kitchen domain.

[0096] Of course, other domains may exist, and objects and predicates may be classified into these other domains. Particular predicates and objects may exist in multiple domains. For example, general objects such as "man" and "bike" may exist in multiple domains. Similarly, a predicate such as "push" may occupy multiple domains.

[0097] Each domain may be used separately to train the domain prediction module 1315. That is, a neural network or transformer model may be trained separately for each domain. The domain prediction module may receive input from the image encoder described above and generate a domain embedding for the given input. In methods known in the art, the decoder used to generate the embedding is trained with all samples, leading to issues such as the long tail problem described above. In contrast, in the method disclosed herein, the domain prediction may be trained separately for each domain. Thus, the generated domain embedding can be generated using only objects and predicates from the predicted domain. That is, the domain prediction module can predict the domain of the image and generate a domain embedding specific to the predicted domain. Thus, the predicted predicates from the input image can be narrowed down to a pool of predicates within the specific domain, thereby leading to more semantically meaningful predictions. Furthermore, new domain distributions can be learned with few new samples from the distribution, thus reducing the computation required when training the model.

[0098] 14 shows an example training technique 1400 for training a domain prediction 1420. The domain prediction module may be, for example, the domain prediction module 2 described in connection with FIG. 8. The image encoder, feature extraction module (not shown), domain-conditional decoder, and predicate classifier may be the same or substantially the same as those described above. The domain-conditional decoder may be, for example, domain-conditional decoder 1 or domain-conditional decoder 2.

[0099] As mentioned above, the domain prediction module can take input global information, such as head tokens, from the image encoder and convert it into domain embeddings. The domain embeddings can be fed into a linear layer neural network, which outputs domain class predictions 1410. The weights of the domain predictions and linear layer blocks can be updated using a loss calculation between the ground truth domain classes 1415 and the predicted domain classes. The predicted and ground truth domain classes can be compared using a one-hot coding format and the calculated loss. The ground truth domain classes can be generated / determined by grouping similar samples into classes. For example, a group of objects such as bike, cycle, and helmet can form one domain labeled as 1 (representing the street domain). Another group of objects such as pizza, spoon, and refrigerator can be grouped into another domain labeled as 2 (representing the kitchen domain).

[0100] In this example, the height domain prediction may be a neural network such as a multi-layer perceptron (MLP) neural network. We have found that a three-layer MLP module is effective for generating domain embeddings. The domain prediction and linear layers can be trained using backpropagation with an Adam optimizer.

[0101] A simple linear layer, also known as a "fully connected layer" or "dense layer," can perform a linear transformation on a domain embedding. For example, a linear layer can take an input vector and map it to an output vector via a linear function. In a neural network, a linear layer can take an input tensor, such as a domain embedding, and transform it into an output tensor by performing a linear operation involving weights and biases.

[0102] When training a neural network with a linear layer, the goal is to find weights and biases that minimize a loss function. The loss function measures the difference between the network's predicted output and the true output (or ground truth). In this example, we used the usual multi-class cross-entropy loss function to train the linear layer. Of course, other loss functions may be used. We trained the linear layer and domain prediction together by minimizing the loss function between the domain class prediction of the linear layer and the domain class of the ground truth. In the implementation phase, the linear layer may be removed (or even not used) so that the output for domain embedding is directly input to the domain-conditional decoder.

[0103] Figure 15 shows another example training technique 1500 for training a domain prediction module 1520. The domain prediction module may be, for example, the domain prediction module 2 described in connection with Figure 8. As mentioned above, the domain prediction module may take input global information, such as head tokens, from an image encoder and convert it into a domain embedding.

[0104] The pre-trained large-scale language model 1505 can take a domain name 1507 as input and output a domain embedding. That is, a user can enter a domain name associated with an input image, and the LLM can generate a domain embedding from the input. In this method, loss can be calculated directly by comparing the "ground truth" domain embedding of the class generated by the LLM with the predicted domain embedding. That is, a similar metric can be used to compare the ground truth domain embedding with the predicted domain embedding. A loss function can be used to train the domain predictor, for example, a distance loss function. A distance loss function, such as a Euclidean distance loss function, can be calculated to bring the domain embedding predicted by the domain predictor closer to the LLM embedding. In one example, the domain predictor is a neural network, such as an MLP, and backpropagation can be used to adjust the neural network weights according to a minimizing loss function.

[0105] In this example, the domain embedding is shown to include three tokens (the domain-predictive embedding is shown with values ​​0.4, 0.8, and 0.3, and the LLM embedding is shown with values ​​0.5, 0.7, and 0.1). These values ​​may be normalized values; that is, each token in the embedding may vary between 0 and 1. Although three tokens are shown here, of course, the embedding may have more or fewer tokens.

[0106] FIG. 16 shows the overall processing flow of a scene graph generation method 1600. An input image 1605 can be input to a scene graph generator, and features can be extracted using a feature extractor. For example, the feature extractor can be a pre-trained feature extractor such as a ResNet-50 module. Of course, other feature extractors such as Faster R-CNN can also be used. The feature extractor can extract image features. That is, the feature extractor can generate regular CNN-based general features from the image. The feature extractor can generate an embedding. The embedding has dimensions (number of output channels, output height, output width) and can be output as a feature map.

[0107] The extracted features can be input to an image encoder, for example, a transformer encoder 1615. The image encoder may be a pre-trained model, for example, the QPIC model in Masato Tamura et al.'s paper. The QPIC model is a module adapted from the DETR module and trained for further scene graph generation tasks. The image encoder unit may obtain the embeddings from the feature extractor and encode the input image into a feature vector (or feature embedding). The image encoder may be part of a transformer encoder and may encode the input embeddings using a self-attention mechanism. The image encoder may generate head tokens 1617 (or class tokens) for the input image, which represent global image features.

[0108] The output embedding from the image encoder may be input to a domain prediction module (or unit). The domain prediction module may include a multi-layer perceptron layer (MLP). For example, the MLP may include three layers. The domain prediction may output a domain embedding 1622 using global image features. That is, the domain prediction module may output a domain embedding. The domain prediction module may be the domain prediction module 1 described above.

[0109] Alternatively, the domain prediction module may be a Domain Prediction Module 2 that is an LLM model, and may instead generate a domain embedding from user input.

[0110] The domain embedding from the domain prediction and the feature embedding from the image encoder may be input to the domain-conditional predicate decoder 1625. The domain-conditional predicate decoder may be domain-conditional decoder 1 or domain-conditional decoder 2 described above. For example, the domain-conditional predicate decoder may receive as input the domain embedding, along with, for example, 100 random query vectors, and use a self-attention mechanism to condition the random queries on the domain embedding. The domain-conditional decoder may receive the feature embeddings from the image encoder (which may be transformed using the self-attention mechanism as described above), along with the conditional random queries, and generate an extended domain (or predicate) embedding. For example, if 100 random queries are used, the domain-conditional predicate decoder may generate 100 extended predicate embeddings.

[0111] The image encoder and the domain-conditional predicate decoder may form a transformer encoder-decoder architecture. A self-attention mechanism on the encoder side may be used to mutually learn which information is important for each token (the input image is divided into multiple tokens). On the decoder side, typically, many random queries (100 random query vectors in this example) may be obtained with a cross-attention mechanism along with the image encoding, with each query representing a triplet prediction. The queries are conditioned on the domain embedding using self-attention to incorporate domain knowledge from the domain embedding into the queries. Queries can be designed so that one query captures at most one object-subject pair and the interaction between them. Therefore, the number of queries can be set large, so that it is larger than the number of actual subject-object pairs (or interactions) in the input image.

[0112] The expanded predicate embeddings can be input to the predicate classifier 1630. The predicate classifier can receive the input embeddings and output predicate classes based on the input. For example, in the figure, the predicate classifier is shown outputting the predicates "sit" and "on." The scene graph generation method can also include an object classifier and a subject classifier (not shown). Known methods can be used to generate the object and subject classes. For example, the object and subject classifier can receive input from a known scene graph generation transformer encoder / decoder (such as the QPIC model) and generate the object and subject classes. Thus, in this example using 100 queries, the 100 embeddings generated by the domain-conditional decoder can be input to the predicate classifier and classified into individual predicates (or attributes). For each sample image using 100 queries, each query can have a (subject, object, predicate) pair associated with it. Thus, there can be a total of 100 (subject, object, predicate) pairs. The same subject-object pair can be output with different predicates. For example,<man, on, bike> but also<man, riding, bike> may also be output.

[0113] The object classifier, subject classifier, and predicate classifier may be collectively referred to as a classifier unit or classifier module. The classifier unit receives the decoder output embeddings and classifies them into object bounding box coordinates, object classes, and predicate classes.

[0114] The output from the classifier may be input to a triplet matching module 1630. Each output of the classifier may provide one combination of subject, object, and predicate class, as well as an object coordinate (i.e., a predicate that matches a triplet of two objects). The triplet matching module may match 1632 the predicted triplet to a triplet in the ground truth, since there may be multiple correct triplets in the ground truth, one of which is selected for each predicted triplet. Known methods, such as the Hungarian algorithm, may be used for triplet matching with the ground truth. For example, the Hungarian algorithm disclosed in H.W. Kuhn and Bryn Yaw. The Hungarian method for the assignment problem. Naval Res. Logist. Quart, pages 83-97, 1955, may be used.

[0115] The Hungarian algorithm can include padding the ground truth set of object-subject pairs with a parameter φ (unpaired) so that the size of the ground truth set is the same as the number of queries. The algorithm can then be used to determine the best pair or assignment among all possible permutations of predicted predicates (queries). Matching costs can be determined, which have four types of costs: box regression cost, intersection-over-union (IoU) cost, object class cost, and predicate class cost (sometimes called behavior class cost). A loss function, the Hungarian loss, can then be calculated for all pairs matched in the previous step.

[0116] As shown in the figure, the classifier is<man, sit, cycle> and<cycle, on, white lines> The triplet matching module uses the Hungarian algorithm to match triplets and generate the ground truth.<man, parking, bike> and<bike, on, white lines> The loss can be determined using the algorithm. The ground truth and predicted triplets are matched based on the "best match" and the loss between the matches is calculated. In this example, the best match (closest) is shown in the figure; that is, it is the output of the matcher. The figure shows an example where the prediction may not be exactly the same as the ground truth. A relatively high loss can be calculated between the inaccurate prediction (shown in italics in the output) and the ground truth and back-propagated through the system being trained.

[0117] FIG. 17 provides an overall testing phase 1700 of the scene graph generation method disclosed herein.

[0118] In a loading step s1710, an input image may be loaded, for example a photograph taken from a camera or cell phone camera, or a still image from CCTV camera footage or a movie.

[0119] In the encoding step s1720, the input image may be encoded. For example, it may be input to a feature extractor, such as a ResNet-50 feature extractor, to generate a feature map. That is, the feature extractor may extract image features (or objects) from the input image. Any suitable convolutional neural network or residual neural network may be used. The output from the feature extractor may be input to an image encoder. Thus, the input image may be encoded by the image encoder. The image encoder may be the encoder side of a transformer encoder-decoder model. The output of the image encoder may be an embedding (encoder embedding). The embedding may be in the form of an augmented feature map.

[0120] In the generating step s1730, the encoder embedding may be input to the decoder side of a transformer encoder-decoder. In this example, the decoder is a domain-conditional decoder as described herein. The decoder may generate a predicate embedding from the input image encoder embedding. Furthermore, the image encoder embedding may be input to a domain prediction module. The domain prediction module may generate a domain embedding from the input embedding. The domain embedding may be input to a domain-conditional decoder, where the domain embedding may be combined with the predicate embedding to generate an extended predicate embedding. Alternatively, an attention mechanism may be used to generate the extended predicate (or domain) embedding, as discussed in connection with FIG. 11.

[0121] In a prediction step s1740, the extended predicate embeddings may be input to a classifier. The classifier may include a predicate classifier, an object classifier, and a subject classifier. The classifier may predict object, subject, and predicate categories (or classes). For each input embedding, the classifier may generate an <object, predicate, subject> triplet.

[0122] In a matching step s1750, the generated <object, predicate, subject> triplet may be compared with the ground truth triplet of the input image. The error may be calculated by determining the loss using the Hungarian algorithm.

[0123] 18A and 18B show example triplets 1800a, 1800b of semantically meaningful scene graphs generated using the methods disclosed herein, compared to the output of existing methods.

[0124] The first input image 1805a shows a man, a motorcycle, and a road. Using methods known in the art, the inventors determined that the generated triplets are<man, on, bike> We found that this triplet accurately describes the scene, but lacks contextual information and is ambiguous. We found that due to the long-tail distribution problem, known methods generally assign simple predicates, such as on, in, and next to.

[0125] In contrast, in FIG. 18a, the method disclosed herein can generate triplet 1820a.<man, riding on, bike> The predicate "riding on" is contextually more relevant and captures the relationship between the man and the bike in a more meaningful way. As shown, the domain 1815a "street" can be input into the model. For example, a domain prediction module can predict the domain and generate a domain embedding for the street domain. Alternatively, the domain name "street" can be user input into a larger language model to generate a domain embedding.

[0126] The second input image 1805b shows a woman and a man in a bakery. Using methods known in the art, the inventors have determined that the generated triplets are<woman, hold, bread> Again, this triplet accurately describes the scene, but lacks contextual information.

[0127] In contrast, in FIG. 18b, the methods disclosed herein can generate triplet 1820a.<woman, sells, bread> The predicate "sell" is more relevant to the scene and captures the context within the image. As shown, the domain 1815a "shopping" can be input into the model.

[0128] The inventors discovered that a scene graph generation method with more semantically meaningful predicates can be used, for example, for video surveillance purposes in different scenarios (e.g., airports, shopping malls, etc.). Instead of continuously monitoring video, a scene graph can automate anomaly detection through inference, saving time. Compared to other known methods, our model can provide more accurate interactions based on knowledge of the scene input. Therefore, anomaly detection in surveillance can be improved. For example, when an anomaly is detected, an alert such as a flashing light or sound can be triggered. The scene graph method can obtain consecutive or non-consecutive still images from the video and determine anomalies or changes in the relationship linking two items. If an anomaly or change is detected, the method can output an alert.

[0129] Furthermore, our scene graph generation method can be used for interactive image editing with more finely controlled object interactions. Common known methods often capture object interactions too descriptively or inaccurately, which leads to inconsistent / poorly controlled image editing. Our method can enable consistent and accurate image editing.

[0130] The scene graph generation method can be used with product recommendation systems to enable more controlled searches. For example, common methods often capture non-semantic information and give vague recommendations to users. However, the method disclosed herein allows for control over the exact information needed to make recommendations to users.

[0131] Figure 19 shows another example 1900 of generating a scene graph triplet using the methods disclosed herein. This example shows two different input images 1905, 1907 related to motorcycles. However, the first input image 1905 shows a man riding a motorcycle on the street, while the second input image 1907 shows a man buying a motorcycle in a showroom. The inventors have found that known methods generate the same predicate for both input images, and that the predicate is<man, on, bike> It was found that:

[0132] Our method uses the domain of the image to generate more meaningful and contextually relevant predicates. For example, for the first image, domain prediction predicts that the domain is "street," and therefore the triplet 1910 of street<man, riding on, bike> For the second image, the domain prediction predicts that the domain is "shopping", and therefore the triplet 1912 for shopping can be generated.<man, shopping, bike> Alternatively, the domain name can be input to a large language model to generate a domain embedding.

[0133] FIG. 20 shows a graphical user interface (GUI) 2000 with which a user can interact to generate a scene graph triplet. The input space of the GUI can display an "Upload Image" button 2005, an optional input "Provide Domain Name" button 2010, and a "Generate Triplet" button 2015. A user interacting with the GUI can click the Upload Image button to upload an image. Thus, the input buttons can have the following functions: Upload Button: A clickable button to upload an image. Provide Domain Name Button: A text input button, by default the input taken is None, otherwise the user can enter text into the box. Generate Triplet Button: A clickable button to generate the output.

[0134] Additionally, a domain name can optionally be entered into the GUI. For example, the domain name can be entered by user input from a keyboard or selected from a drop-down menu containing a list of domain names. If a domain name is not entered into the GUI, the model can generate a domain embedding using the domain prediction module 2. That is, the domain embedding can be generated from global information of the image. If a domain is provided to the GUI, the model can pass the text to the LLM and use the embedding generated from the LLM as the domain embedding. That is, if a domain is provided to the GUI, the model can use the domain prediction module 1 described above.

[0135] Once an image has been uploaded and a domain name optionally entered, the user can activate the "Generate Triplets" button to generate the structured text as triplets and view the output.

[0136] The figure shows an example of an output space that can be output by the GUI. The output can display an input image 2007, bounding boxes 2010a, 2010b, 2010c surrounding identified objects in the image, and structured text triplets 2015. The output space can visually link the objects in the bounding boxes with the corresponding portions of the triplets. In this example, the text triplets are output below the image and are underlined, with underlining (i.e., solid, dotted, dashed) linking the bounding boxes and the associated objects. In another example, the text can include arrows linking the text and objects and / or can be displayed in a graph format with objects as nodes and relationships linking the objects as vertices.

[0137] The GUI may include additional buttons (not shown) with which the user can interact, such as a save button that downloads the output image along with its bounding box and triplets, e.g., as a PNG image, and a share button that allows the user to share the output with other users.

[0138] 21 is a block diagram of an information processing appliance 2100 or computing device 2100, such as a data storage server, that may be used to embody the present invention, to perform some or all of the operations of the methods embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiments. The computing device 2100 may be used to implement any of the method steps described above, for example, any of steps S510-S550, and / or S1210-S1270, and / or S1205-S1275, and / or s1710-s1750.

[0139] Computing device 2100 includes a processor 2103 and memory 2104. Optionally, the computing device also includes a network interface 2107 for communicating with other such computing devices, such as other computing devices of embodiments of the present invention. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 2106, and a display unit, such as one or more monitors 2105. These elements may facilitate user interaction. The components may be connected to each other via a bus 2102.

[0140] Memory 2104 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause it to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing the methods disclosed herein, or any method disclosed herein, e.g., steps S510-S550. The term "computer-readable storage medium" may include any medium capable of storing, encoding, or retaining a set of instructions for execution by a machine and causing the machine to perform any one or more of the method steps of the present disclosure; therefore, the term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media. For example, but not limited to, the term "computer-readable storage medium" may include solid-state memory, optical media, and magnetic media. By way of example, and without limitation, such computer-readable media may include non-transitory computer-readable media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), and the like.

[0141] The processor 2103 is configured to control a computing device and perform processing operations, such as executing computer program code stored in the memory 2104, for example, to perform any of the method steps described herein. The memory 2104 stores data read and written by the processor 2103, and may store at least one feature extractor and / or a pre-trained image encoder and / or a domain-conditional decoder and / or a domain prediction module and / or an input image and / or at least one classifier and / or a triplet matching unit and / or other data and / or programs for performing any of the above method steps. These entities may be in the form of code blocks that are called as needed and executed within the processor.

[0142] As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. A processor may also include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. A processor may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. In one or more embodiments, a processor is configured to execute the operations and instructions for performing the operations discussed herein. The processor 2003 may be considered to include any of the units or modules described above. Any operation described as being performed by a unit may also be implemented as a method by a computer, such as the processor 2103.

[0143] To train the models described herein, we used one Nvidia A30 GPU with 24 GB of RAM. The processor used was an Intel® Xeon® Silver 4314 CPU with 256 GB of RAM. Inference was performed on the same system. We also successfully tested the models on a CPU-based system with an Intel® Core® i5 processor with at least 32 GB of RAM and a 16 GB GPU for accelerated training / testing. We found that the recommended minimum hardware requirements for the systems disclosed herein for training / testing may require an Intel i5 processor with at least 32 GB of RAM and a 16 GB GPU for accelerated training / testing.

[0144] The display unit 2105 may display data stored and / or generated by the computing device, such as a generated scene graph, an image generated using the scene graph, a generated scene graph triplet, and / or a GUI window (such as the GUI shown in FIG. 20), and / or interactive displays that allow a user to interact with the device 2100 (e.g., selection dialogs, input text boxes, and / or any other output described above, such as an option to enter images), and may display cursors, dialog boxes, and screens that allow a user to interact with programs and data stored on the computing device. The input mechanism 2106 may allow a user to input data and instructions into the computing device, such as allowing a user to enter any of the user inputs described above. For example, the display unit may display a GUI that includes a user panel or input space for user interaction, as described herein. A user may select an input image for the associated generated scene graph. A user may interact with the GUI and the display to generate and display a scene graph of an image.

[0145] The network interface (network I / F) 2107 can be connected to a network such as the Internet and can be connected to other such computing devices via the network. The network I / F 2107 can control input and output data to and from other devices via the network. Other peripheral devices such as a microphone, speaker, printer, power supply unit, fan, case, scanner, tracker ball, etc. may also be included in the computing device.

[0146] Methods embodying the present invention may be performed on a computing device / appliance 2100 such as that shown in FIG. 21. Such a computing device need not have all of the components shown in FIG. 21, but may consist of a subset of these components. For example, the device 2100 may include a processor 2103 and a memory 2104 coupled to the processor 2103. Alternatively, the device 2100 may include a processor 2103, a memory 2104 coupled to the processor 2103, and a display 2105. Methods embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may be a data storage device that stores at least a portion of the data.

[0147] The method embodying the present invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the data, for example, the image encoder and / or decoder and / or domain prediction may be stored in a server separate from the other units.

[0148] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can also be implemented as a computer program or computer program product, i.e., a computer program embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0149] A computer program may be in the form of a stand-alone program, a computer program portion, or multiple computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0150] The system described herein was developed in the Python programming language using the Pytorch and huggingface APIs. Additional Python libraries were also used, such as NumPy, cv2, and plotly for data processing and visualization. Of course, any other suitable programming language may be used.

[0151] The method steps of the present invention may be performed by one or more programmable processors executing computer programs to perform the functions of the present invention by operating on input data and generating output. The apparatus of the present invention may be implemented as programmed hardware or as dedicated logic circuitry including Field Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs), etc.

[0152] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memories that store instructions and data.

[0153] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments of the embodiments.

[0154] For the avoidance of doubt, the present invention relates to the following numbered items: (1) A computer-implemented method for generating structured text describing an image, comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; A method comprising: (2) The method according to item 1, further comprising the step of outputting the structured text. (3) The method according to any one of items 1 to 2, further comprising the step of predicting a prediction region of the image. (4) The method according to any one of items 1 to 3, wherein the step of extracting items from the image is performed using a feature extractor. (5) The method according to any one of items 1 to 4, wherein the extracted items are encoded using an image encoder. (6) The method according to any one of items 1 to 5, wherein the step of predicting the relationship between two items in the image based on the encoded extracted items and the domain embedding is performed by a decoder (image decoder). (7) A method according to any one of items 1 to 6, wherein the step of outputting the structured text includes outputting the structured text to a user display. (8) The method according to any one of items 1 to 7, wherein the output is from a graphical user interface. (9) The method according to any one of items 1 to 8, wherein the image is input by user input on a graphical user interface. (10) The method according to any one of items 1 to 9, wherein the user provides a predicted domain along with the image. (11) The method according to any one of items 1 to 9, wherein the method includes predicting the domain. (12) A method according to any of items 7 to 11, wherein the scene graph triplets on the user display are represented as an overlay on the image, with bounding boxes around items in the image indicating subjects and objects and links between the subjects and objects. (13) The method according to item 12, wherein the subject and / or the object and / or the link are labeled. (14) The relationship between two items in the image is: generating predicate embeddings from the extracted and encoded items; concatenating the domain embedding and the predicate embedding to generate an extended predicate embedding; predicting the relation from the extended predicate embedding; 14. The method according to any one of items 1 to 13, wherein the method is predicted by the above formula (1). (15) The method according to any one of items 5 to 14, wherein the image encoder and decoder form or form part of a transformer model. (16) The method of any of items 1 to 13, wherein a relationship between two items in the image is predicted by using a self-attention mechanism to condition a learnable query on the domain embedding, inputting the conditioned learnable query and the encoded extracted items into a cross-attention mechanism to generate an extended predicate embedding, and predicting the relationship from the extended predicate embedding. (17) The method according to any one of items 1 to 16, wherein the prediction domain of the image is generated from global information of the image. (18) The method according to any one of items 1 to 17, wherein the predicted domain is predicted using a trained neural network. (19) The method according to any one of items 17 to 18, wherein the global information includes a head token (sometimes called a class token) of the image generated by the image encoder. (20) The method according to item 19, wherein the head token is a randomly initialized token added to the input format of the image. (21) The method according to any one of items 19 to 20, wherein the head token accumulates information from other tokens of the image to form a representation of the global information of the image. (22) The method according to any one of items 18 to 21, wherein the trained neural network is a domain prediction unit including a multilayer perceptron (MLP) neural network. (23) The method according to item 22, wherein the MLP comprises three layers. (24) The method according to any one of items 22 to 23, wherein the domain prediction unit is trained using a linear layer, and weights of the domain prediction and linear layer are updated using a loss calculation between ground truth domain classes and domain classes generated from the domain embedding. (25) The method of item 24, wherein the loss calculation between the ground truth domain class and the domain class includes a one-hot coding format. (26) The method according to any one of items 22 to 25, wherein the domain prediction unit is trained using a pre-trained large-scale language model (LLM). (27) The method of item 26, wherein the domain prediction weights are updated by calculating a similarity metric between the domain embedding and a ground truth domain embedding generated by the LLM from a user-input domain name and minimizing a loss function determined from the similarity metric. (28) The method according to item 27, in which the minimization loss function is backpropagated through the domain prediction and the weights are updated. (29) The method according to any one of items 27 to 28, wherein the similarity metric is a similarity score. (30) The method according to any one of items 1 to 16, wherein the predicted domain of the image is input by a user as a domain name into a trained large-scale language model (LLM) neural network, and the LLM generates the domain embedding. (31) The method according to item 30, wherein the user inputs a prediction domain. (32) The method of item 30, wherein the user is presented with a list of predicted domains and prompted to select an appropriate domain from the list. (33) The method according to any one of items 1 to 32, wherein the predictive domain imports further information rather than directly from the image to capture the context of the image based on an overall impression of the image. (34) The method according to item 33, wherein the overall impression of the image includes the global information of the image. (35) The method according to item 34, wherein the domain prediction is trained on images grouped with similar overall impressions, and a domain embedding is generated from the collective training of each image. (36) The method comprises: detecting an interaction between two items in the image; inputting a further image, wherein two items in the further image are identical to two items in the image; comparing predicted relationships between the two images to detect interactions between two items in the images; 36. The method according to any one of items 1 to 35, further comprising: (37) The method according to item 36, further comprising the step of outputting a warning when the predicted relationship differs. (38) The method according to any one of items 36 to 37, further comprising the step of extracting consecutive or non-consecutive still images from the video and determining the change in the relationship linking two items in each still image. (39) An information processing device for training a neural network to generate structured text describing an image, comprising: a memory; and a processor coupled to the memory, the processor: Extracting items from the image; Encoding the extracted items; generating a domain embedding from the predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; An information processing device configured as follows. (40) A computer program that, when executed on a computer, causes the computer to perform a method for generating structured text describing an image, the method comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; a computer program comprising: (41) The computer program according to item 40, wherein the computer program is stored on a computer-readable medium. (42) The computer program according to item 41, wherein the computer-readable medium is non-transitory. (43) A digital electronic circuit, or computer hardware, firmware, software, or a combination thereof, configured to perform the method according to any one of items 1 to 42. (44) The method according to any of items 1 to 43, wherein the method is implemented as a computer program or computer program product, optionally the computer program being embodied on a non-transitory information carrier for execution by or for controlling the operation of one or more hardware modules. (45) Item 45. The computer-implemented method of item 44, wherein the non-transitory information carrier is a machine-readable storage device or a propagated signal. (46) 46. ​​The method according to any of items 1 to 45, wherein the method is implemented as a computer program, the computer program being in the form of a stand-alone program, a computer program portion, or a plurality of computer programs, optionally written in any form of programming language, including a compiled or interpreted language, and optionally deployed solely in any form including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment.

[0155] The present invention has been described in terms of specific embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results.

[0156] Those skilled in the art will understand that, unless mutually exclusive, a feature described in connection with any one of the above embodiments may also be applied mutatis mutandis to any other embodiment. Furthermore, unless mutually exclusive, any feature described herein may be applied to any embodiment and / or combined with any other feature described herein.

[0157] In addition to the above-described embodiments, the following supplementary notes are also disclosed. (Supplementary Note 1) A computer-implemented method for generating structured text describing an image, comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; A method comprising: (Supplementary Note 2) The step of predicting a relationship between two items in the image includes: generating predicate embeddings from the encoded extracted items; concatenating the domain embedding and the predicate embedding to generate an extended predicate embedding; predicting the relation from the extended predicate embedding; 2. The method of claim 1, comprising: (Supplementary Note 3) The step of predicting a relationship between two items in the image includes: using a self-attention mechanism for conditional learnable queries on the domain embedding; inputting the conditional learnable query and the encoded extracted items into a cross-attention mechanism to generate an extended predicate embedding; predicting the relation from the extended predicate embedding; 2. The method of claim 1, comprising: (Supplementary Note 4) The method of Supplementary Note 1, wherein the prediction domain of the image is generated from global information of the image. (Appendix 5) The method of Appendix 4, wherein the prediction domain is predicted using a trained neural network. (Supplementary Note 6) The method of Supplementary Note 4, wherein the global information includes a head token of the image generated by an image encoder. (Appendix 7) The method of appendix 5, wherein the trained neural network is a domain prediction unit including a multi-layer perceptron (MLP) neural network. (Appendix 8) The method according to Appendix 7, wherein the MLP includes three layers. (Supplementary Note 9) The method of Supplementary Note 7, wherein the domain prediction unit is trained using a linear layer, and weights of the domain prediction and linear layer are updated using a loss calculation between ground truth domain classes and domain classes generated from the domain embedding, and the loss calculation for the ground truth domain classes and the domain classes includes a one hot coding format. (Supplementary Note 10) The method of Supplementary Note 7, wherein the domain prediction unit is trained using a pre-trained large-scale language model (LLM), and the weights of the domain prediction are updated by calculating a similarity metric between the domain embedding and a ground truth domain embedding generated by the LLM from a user-input domain name, and minimizing a loss function determined from the similarity metric. (Supplementary Note 11) The method of Supplementary Note 1, wherein the predicted domain of the image is input by a user as a domain name into a trained large-scale language model (LLM) neural network, and the LLM generates the domain embedding. (Supplementary Note 12) The method of Supplementary Note 1, wherein the predictive domain imports additional information, rather than directly from the image, to capture the context of the image based on an overall impression of the image. (Appendix 13) The method of appendix 1, wherein the image is input by user input on a graphical user interface (GUI). (Supplementary Note 14) The method of Supplementary Note 1, wherein the method outputs the structured text as scene graph triplets on a user display. (Appendix 15) The method of Appendix 14, wherein the scene graph triplets on the user display are represented as an overlay on the image, with bounding boxes around items in the image indicating subjects and objects and links between the subjects and objects. (Appendix 16) The method of appendix 14, wherein the subject and / or object and / or link are labeled. (Supplementary Note 17) The method includes detecting an interaction between two items in the image; inputting a further image, wherein two items in the further image are identical to two items in the image; comparing predicted relationships between two items in the image and in the further image to detect an interaction between the two items in the image; 17. The method according to any one of claims 1 to 16, further comprising: (Supplementary Note 18) The method of Supplementary Note 17, further comprising the step of outputting a warning if the predicted relationship differs. (Supplementary Note 19) A computer program product that, when executed on a computer, causes the computer to perform a method for generating structured text describing an image, the method comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; a computer program comprising: (Supplementary Note 20) An information processing device for training a neural network to generate structured text describing an image, comprising: a memory; and a processor coupled to the memory, the processor: Extracting items from the image; encoding the extracted items; generating a domain embedding from the predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; An information processing device configured as follows. [Explanation of symbols]

[0158] 410 Feature Extractor 415 Image Encoder 420 Domain Prediction 422 Language Models 425 Domain-Conditional Decoder 430 Classifier 435 Triplet Matching

Claims

1. 1. A computer-implemented method for generating structured text describing an image, comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; A method comprising:

2. Predicting a relationship between two items in the image includes: generating predicate embeddings from the encoded extracted items; concatenating the domain embedding and the predicate embedding to generate an extended predicate embedding; predicting the relation from the extended predicate embedding; The method of claim 1 , comprising:

3. Predicting a relationship between two items in the image includes: using a self-attention mechanism for conditional learnable queries on the domain embedding; inputting the conditional learnable query and the encoded extracted items into a cross-attention mechanism to generate an extended predicate embedding; predicting the relation from the extended predicate embedding; The method of claim 1 , comprising:

4. The method of claim 1 , wherein the prediction domain of the image is generated from global information of the image.

5. The method of claim 4 , wherein the prediction domain is predicted using a trained neural network.

6. The method of claim 4 , wherein the global information includes a head token for the image generated by an image encoder.

7. The method of claim 5 , wherein the trained neural network is a domain prediction unit comprising a multi-layer perceptron (MLP) neural network.

8. The method of claim 7 , wherein the MLP comprises three layers.

9. 8. The method of claim 7, wherein the domain prediction unit is trained using a linear layer, and weights of the domain prediction and linear layer are updated using a loss calculation between ground truth domain classes and domain classes generated from the domain embedding, and the loss calculation for the ground truth domain classes and the domain classes comprises a one hot coding format.

10. 8. The method of claim 7, wherein the domain prediction unit is trained using a pre-trained large-scale language model (LLM), and the weights of the domain prediction are updated by calculating a similarity metric between the domain embedding and a ground truth domain embedding generated by the LLM from a user-input domain name, and minimizing a loss function determined from the similarity metric.

11. The method of claim 1 , wherein the predicted domain of the image is input by a user as a domain name into a trained large-scale language model (LLM) neural network, and the LLM generates the domain embedding.

12. The method of claim 1 , wherein the predictive domain imports additional information, rather than directly from the image, to capture the context of the image based on an overall impression of the image.

13. The method of claim 1 , wherein the image is input by user input on a graphical user interface (GUI).

14. The method of claim 1 , wherein the method outputs the structured text as scene graph triplets on a user display.

15. 15. The method of claim 14, wherein the scene graph triplets on the user display are represented as an overlay on the image, with bounding boxes around items in the image indicating subjects and objects and links between the subjects and objects.

16. The method of claim 14 , wherein the subject and / or object and / or link are labeled.

17. The method includes detecting an interaction between two items in the image; inputting a further image, wherein two items in the further image are identical to two items in the image; comparing predicted relationships between two items in the image and in the further image to detect an interaction between the two items in the image; The method of any one of claims 1 to 16, further comprising:

18. 18. The method of claim 17, further comprising outputting a warning if the predicted relationship differs.

19. 1. A computer program product which, when executed on a computer, causes the computer to perform a method for generating structured text describing an image, the method comprising: extracting items from the images; encoding the extracted items; generating a domain embedding from a predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; a computer program comprising:

20. 1. An information processing device for training a neural network to generate structured text describing an image, the device comprising: a memory; and a processor coupled to the memory, the processor comprising: Extracting items from the image; encoding the extracted items; generating a domain embedding from the predicted domain of the image; predicting a relationship between two items in the image by decoding the encoded extracted items and the domain embedding; classifying the two items and the predicted relationships to form the structured text as triplets; An information processing device configured as follows.

Images