Scene detection

Abstract

To provide a computer-implemented method, a program and an information processor for scene detection.SOLUTION: A method generates, on the basis of an input scene graph, a graph vector, encodes an input image to generate an image vector, and performs update processing to generate an updated graph vector and an updated object query vector, in which the update processing comprises: a step A of updating, on the basis of the graph vector, the object query vector; a step B of updating, on the basis of the image vector, the object query vector; a step C of updating, on the basis of the object query vector, the graph vector; and a step D of extracting an object region and a category from the updated object query vector and calculating, on the basis of the updated object query vector and the updated graph vector, a similarity between the input image and the input scene graph.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to image analysis, and in particular to image analysis using scene graphs, i.e. scene detection, and the present application discloses a computer-implemented method, a computer program product, and an information processing apparatus. [Background technology]

[0002] There is great demand for detecting specific objects, actions, and scenes within images and videos (e.g., for purposes such as image recognition and correlation, and image and / or video editing for social media posting, news reporting, or other real-world applications). The advent of artificial intelligence, machine learning, and neural networks (NNs) has improved techniques for image analysis and object detection and classification. However, a need exists for improvements as new applications emerge for such technologies. Summary of the Invention

[0003] In view of the above, a method for scene detection is desired.

[0004] According to an embodiment of the first aspect, the present application provides a method for generating a plurality of graph vectors based on an input scene graph, encoding the input image to generate a plurality of image vectors, and performing an update process (decoder process) to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process including (iteratively) updating at least one object query vector based on the plurality of graph vectors (such that at least one object query vector includes information indicative of at least one object in the scene graph), and updating at least one object query vector based on the plurality of image vectors (with information indicative of at least one (relevant) object detected in the input image). and (iteratively) updating a plurality of graph vectors based on the at least one object query vector (with information indicative of at least one (relevant) object detected in the input image); extracting from the at least one updated object query vector information indicative of a region of the at least one object (detected in the input image) and a category / class of the at least one object; and calculating a matching score indicative of a similarity between the (at least one object detected in the) input image and the input scene graph based on (similarity / correlation between) the at least one updated object query vector and the plurality of updated graph vectors.

[0005] Features of any aspect / embodiment may be applied to any other aspect / embodiment.

[0006] Reference will now be made, by way of example, to the accompanying drawings in which: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the comparison method. [Diagram 2] FIG. 1 is a diagram showing the comparison method. [Diagram 3] FIG. 1 is a diagram showing the comparison method. [Figure 4] FIG. 1 is a diagram illustrating a system. [Diagram 5] 1 is a diagram useful for understanding an embodiment. [Figure 6] This is a diagram useful for understanding the update process. [Figure 7] FIG. [Figure 8] FIG. 1 is a diagram illustrating a system. [Figure 9] 1A and 1B are diagrams showing results that are useful for understanding the embodiments. [Figure 10] FIG. [Figure 11] FIG. [Figure 12] FIG. 1 is a diagram illustrating a system. [Figure 13] FIG. [Figure 14] FIG. [Figure 15] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The following terms may be used herein (neither the list nor the definitions are exhaustive):

[0009] Neural Network: A machine learning network modeled on the human brain.

[0010] Image Encoder: A process that encodes an image's data in a different way than the input image, e.g., in the form of an image rather than pixels. May include compression.

[0011] Attention mechanism: a technique that aims to mimic cognitive attention. This effect emphasizes parts of the input data and degrades others. The goal is to make the network focus on important parts of the data, even if they are small parts of an image or a piece of text. Attention mechanisms can be thought of as extracting information from important parts of the data, for example for updating an object.

[0012] Graph Encoder: A process that encodes a graph as a set of vectors. For example, one can represent nodes as the concatenation of their embeddings and position embeddings, represent node relations as the shortest paths between them, and combine both with relation-augmented self-attention.

[0013] Scene graph: A graph that represents a particular combination of at least one object and at least one relationship and / or action (or "predicate"). For example, a scene graph may represent objects in an image and their relationships to each other. That is, a scene graph may correspond to a logical / spatial representation of a graphical scene.

[0014] To find a particular combination of objects and actions in an input image, for example, a neural network (NN) may be used to detect objects (e.g., dogs, people, cars, etc.) and actions (standing, sitting, walking, etc.) in the input image. Those detected candidates may then be filtered in post-processing to find a particular object (e.g., a running black dog). Furthermore, when the input is given in the form of text (text string), other NNs may be used to predict a matching score between the input text and the input image.

[0015] For example, Figure 1 is a schematic diagram of a comparison method according to the above method. The comparison method is for detecting a specific combination of objects and / or actions and / or relationships in an input image. In the comparative example, the specific combination to be detected is "a person near a dog."

[0016] As shown in FIG. 1, a first NN (NN A) is used to detect all objects in an input image. A scene graph is predicted based on the detected objects. Post-processing is required to detect a specific combination ("person near a dog"). Separately, a second NN (NN B) is used to calculate a matching score between the input image and a text string (e.g., "person near a dog" in this case) that indicates the specific combination to be detected in the input image.

[0017] Fig. 2 is a schematic diagram showing another comparison method. The comparison method shown in Fig. 2 may be used to detect an object in the comparison method shown in Fig. 1. The comparison method shown in Fig. 2 may be referred to as comparison method 1.

[0018] The comparative method 1 is sometimes called a Scene Graph Generation (SGG) method. The comparative method 1 includes receiving an image as input and predicting object regions, their categories, and the relationships between objects. The disadvantage of the comparative method 1 is that post-processing is required to find a specific combination of objects and actions / relationships, as was required in the method depicted in FIG. 1. Furthermore, if the comparative method 1 is used for object detection in the method depicted in FIG. 1, another method (and another NN) is required to calculate a matching score between a specific combination of objects and actions / relationships and the input image. Furthermore, it is difficult to predict abnormal objects (e.g., "pink elephants") using the comparative method 1, because such abnormal objects are out-of-distribution.

[0019] Fig. 3 is a schematic diagram showing another comparison method. The comparison method shown in Fig. 3 can be used to calculate a matching score between a text input and an input image, for example, in the comparison method shown in Fig. 1 (when the text input defines a specific combination with an object relationship / action). The comparison method shown in Fig. 3 can be called comparison method 2.

[0020] As shown in Figure 3, an image and a text string are received as input, and the comparison method involves predicting a matching score between the image and the text string. A drawback of comparison method 2 is that the region and category of the target object cannot be directly predicted (e.g., when implementing the method depicted in Figure 1, comparison method 1 is required as well as comparison method 2). Furthermore, the predicted score may be difficult to interpret in some cases.

[0021] 4 is a schematic diagram illustrating a system 20 for detecting a scene (i.e., a combination of at least one object and at least one action or relationship) in an input image. The system 20 receives as input an input image and a scene graph, and outputs information indicative of at least one object detected in the input image that corresponds to the scene graph, and a matching score between the input image and the scene graph.

[0022] The system 20 includes an image encoder 21, a graph encoder 23, a decoder 25, a region head 26, a category head 27, a node head 28, and a score head 29. The region head 26, the category head 27, the node head 28, and the score head 29 may be referred to as a region prediction unit, a category prediction unit, a node prediction unit, and a score prediction unit, respectively. Figure 4 shows some inputs and outputs. An input image (an example of which is shown in Figure 4) is input to the image encoder 21, which outputs an output I to the decoder 25. An object query (which may be referred to as an object query vector) and a matching query (which may be referred to as a matching query vector or an input matching query vector) are input to the decoder 25. A scene graph (an example of which is shown in Figure 4) is input to the graph encoder 23, which outputs an output G to the decoder 25.

[0023] The image encoder 21 generates image vectors and outputs them (as output I) to the decoder 25. The graph encoder 23 generates graph vectors and outputs them (as output G) to the decoder 25. The decoder 25 updates the object queries, matching queries, and graph vectors (encoded scene graphs) based on the image vectors (encoded images), the object queries, and the graph vectors. This updating may be called an updating process. The decoder thereby generates an updated object query (or object query vector) and an updated matching query (or matching query vector).

[0024] The "head" units extract specific information from the output of the decoder 25: the region head 26 extracts the region of at least one detected object from the updated object query; the category head 27 extracts the category of at least one detected object from the updated object query; and the score head 29 extracts the matching query from the updated matching query.

[0025] The scene graph input to the graph encoder 23 may contain "mask nodes" (or "masked nodes"). A mask node is a node in a scene graph that does not contain any information about the objects or relationships or actions it represents. In other words, a mask node is a node in a scene graph that does not contain a label. A mask node can be thought of as representing an object or an action or relationship to be predicted. In other words, a mask node can be said to be connected to at least one other node in the input scene graph and does not contain a label.

[0026] The decoder 25 outputs the updated graph vector. If the scene graph used contains mask nodes, the node head 28 extracts information from the updated graph vector indicating the labels of the mask nodes.

[0027] Node head 28 is not required, and other implementations of system 20, for example implementations in which the scene graph does not have mask nodes, may not have node head 28.

[0028] 4 includes some example data. Based on an example input image and an example scene graph (including a mask node), the region (in the form of coordinates) and category ("person") of at least one object in the input image is predicted, a label "close" is predicted for the mask node, and a matching score of "0.8" is predicted to indicate the similarity between the scene graph and the input image.

[0029] Figure 4 represents an embodiment of a graph encoder 23 (right hand side). The graph encoder in this particular embodiment works by estimating graph connectivity (i.e. which nodes each node is connected to) and by extracting and classifying tokens from node labels in the scene graph. This can be achieved using self-attention networks / mechanisms (e.g. as described in G. Mialon et al. GraphiT: Encoding Graph Structure in Transformers) or message passing mechanisms (e.g. graph neural networks).

[0030] The example input described with respect to Figure 4 includes multiple object queries. In other examples, only one object query may be included. Thus, the input may have at least one object query.

[0031] FIG. 5 is a schematic diagram illustrating an example image encoding process of the image encoder 21. The image encoder 21 in this example compresses an image (e.g., from size L×K to size M×N) and computes correlation and summation. The encoding process is to bring values ​​of similar regions closer together so that objects in the image can be distinguished. The image encoder 21 may use a transformer-based network with an attention mechanism. An example of the process of computing correlation and summation is shown as steps 1-3 in FIG. 5.

[0032] The provisional vectors are generated based on the input image such that the input image is represented as a number of provisional vectors. To generate the provisional vectors, a neural network, for example a convolutional neural network (e.g., ReNet), may be used. This results in a compressed image, for example, as a vector (M×N). The provisional vectors can be considered as representing regions of the input image. The provisional vectors are updated using an attention mechanism as described below.

[0033] Step 1 involves using an attention mechanism / network on the provisional vectors. The attention mechanism / network emphasizes parts of the data and deemphasizes other parts, with the goal of placing more emphasis on important parts of the data.

[0034] Step 2 comprises calculating the correlation between each interim vector and each other interim vector after step 1. An example grid / table / matrix of size M×N is depicted in FIG. 5. The matrix is ​​populated with values ​​indicative of the calculated correlation between one of the interim vectors and all other interim vectors (and itself). S matrices are generated and populated, where S=M×N, i.e., one matrix is ​​generated and populated for each interim vector. Many correlation algorithms / methods may be used, for example dot product or cosine similarity may be used to calculate the correlation score (which may be called attention score).

[0035] Step 3 comprises an addition step, where for each interim vector, the correlation values ​​are respectively multiplied by their corresponding interim vector and summed. For example, considering the first interim vector of the image of FIG. 5, the correlation values ​​between this first interim vector and the second interim vector are multiplied by the second interim vector and added to the sum, the correlation values ​​between the first interim vector and the third interim vector are multiplied by the third interim vector and added to the sum, and so on up to the Sth interim vector. The result of the summation is a first image vector (corresponding to the first interim vector), which is output by the image encoder 21 as part of the plurality of image vectors. The multiplication and summation process is repeated for the second interim vector, and the result of the summation is a second image vector (corresponding to the second interim vector), which is output by the image encoder 21 as part of the plurality of image vectors. The process is performed for each interim vector.

[0036] Steps 1-3 may be considered as a single step of updating the interim vector using an attention mechanism. For example, a key-query-value attention mechanism may be used, for example, to determine an attention score and perform a weighted sum to update the interim vector.

[0037] The encoding process described above with reference to FIG. 5 is just one example.

[0038] Although the image encoder 21 in this embodiment compresses the image, this is not required. Furthermore, there are other processes for generating image vectors based on an image, and the image encoder 21 is not limited to the embodiment of FIG.

[0039] 6 is a schematic diagram illustrating an example of an update process performed by the decoder 25, the update process comprising steps A to E. The update process is to update at least one object query (or object query vector), a graph vector, and a matching query to generate at least one updated object query, a plurality of updated graph vectors, and an updated matching query vector.

[0040] Step A comprises updating at least one object query based on the graph vector, which can be considered as updating at least one object query vector based on the multiple graph vectors such that the at least one object query vector includes information indicating at least one object in the scene graph. The updating of step A can be considered as adding and / or updating at least one object query vector based on the multiple graph vectors. Step A can be considered as updating value(s) of object query(s) from the graph to know which objects to detect.

[0041] Step B comprises updating at least one object query vector based on the multiple image vectors. This can be considered as updating at least one object query vector based on the multiple image vectors with information indicative of at least one (related) object detected in the input image (wherein "related" can be taken to mean that the information about the object is based on the scene graph / graph vector). Step B can be considered as adding and / or updating at least one value of at least one object query vector based on the multiple image vectors. For example, the value of the object query vector can be updated based on the multiple image vectors corresponding to a particular object. Step B can be considered as updating values ​​in the object query using images to obtain object information from the images.

[0042] Step C comprises updating the plurality of graph vectors based on the at least one object query vector. This can be considered as updating the plurality of graph vectors based on the at least one object query vector with information indicative of at least one (related) object detected in the input image (where "related" can be taken to mean that the object corresponds to an object in the scene graph, i.e., the object described by the graph vector). Step C can be considered as adding and / or updating at least one value of at least one of the plurality of graph vectors based on at least one of the at least one object query vector. For example, a value of a graph vector can be updated based on an object query vector that corresponds to a particular object (detected in the image).

[0043] As mentioned above, the mask node may not be included in the scene graph, so the associated processing is not required. If the mask node is included in the scene graph, step C can be considered to include adding and / or updating information in the graph vector indicating the mask node based on information of at least one object in the input image. Step C can be considered to include updating values ​​in the graph vector from object information (in the object query) to help determine what the mask node is (i.e., determine the label of the mask node). However, even if the mask node is not included in the scene graph, some mask node-related processing may be performed, which can improve the detection accuracy. This is because, for example, information from the scene graph, such as what kind of dog is near the man (black or white, standing or sitting, etc.), can be more complete.

[0044] Step D comprises updating a matching query vector based on the at least one object query vector and the plurality of graph vectors. This can be considered as updating the matching query vector with information indicative of at least one object detected in the input image from the object query vector and information indicative of at least one object in the scene graph from the graph vector. Step D can be considered as updating at least one value of the input matching query vector based on the at least one object query vector and the plurality of graph vectors. Step D can be considered as updating the matching query to know how well the graph and the detected object match, i.e., how well the graph vector and the object query vector match.

[0045] Any updated or added values ​​may depend on the correlation between the related vectors. For example, some correlations may be written as follows:

[0046] Updating the at least one object query vector based on the plurality of graph vectors (Step A) may include updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of graph vectors.

[0047] Updating at least one object query vector based on the multiple image vectors (step B) may comprise updating at least one object query vector based on a correlation between the at least one object query vector and the multiple image vectors. It will be appreciated that such correlation-based updating of the at least one object query based on the image vectors may result in the at least one object query being updated with information indicative of objects that are indicated by the at least one object query (before updating) and are present in the image (i.e., indicated by information in the image vector). It is further noted that the image vectors contain information about related and / or similar regions. That is, the image vector of a region contains information about other regions, e.g., their similarity and / or relationship with the region. Thus, if there is a correlation between an object query indicative of an object in a scene graph and an image vector indicative of a region in an image, this correlation may be based at least in part on a correlation between the scene graph object and other objects in the scene graph and between the image region / object and the relationship and / or similarity with other regions / objects in the image.

[0048] Updating the plurality of graph vectors based on the at least one object query vector (step C) may comprise updating at least one of the graph vectors based on a correlation between at least one of the at least one object query vector and the plurality of graph vectors. It will be appreciated that such correlation-based updates may result in information being added to the "related" graph vectors, i.e., information of objects / actions / relationships present in the input image.

[0049] Updating the input matching query vector based on the at least one object query vector (Step D) may include updating the input matching query vector based on a correlation between the at least one object query vector and the input matching query vector. Updating the input matching query vector based on the multiple graph query vectors (Step D) may include updating the input matching query vector based on a correlation between the multiple graph query vectors and the input matching query vector.

[0050] Updating the input matching query vector (Step D) may include updating the input matching query vector based on similarities / correlations between at least one object query vector and a plurality of graph vectors. For example, the updated matching query vector can be considered as a vector based on similarities / correlations between at least one object query vector and a graph vector.

[0051] The updating process may include using an attention-based network / mechanism or an attention scheme. That is, any of steps A-C may include using an attention-based network / mechanism or an attention network / mechanism to determine, for example, a correlation between two related vectors. The correlation may be referred to as an attention score. It will be appreciated that a high similarity between the two sets of information results in a high attention score (high correlation).

[0052] The updating process depicted in FIG. 6 comprises a step E, which comprises updating each of the multiple object query vectors, if there are multiple object query vectors, based on at least one other object query vector of the multiple object query vectors. Any of the above considerations for steps A-C may apply to this updating. This updating may be considered useful for two reasons (among others): (i) for example, if multiple object queries have information about the same object (i.e. "detecting" the same object), then this updating may help to avoid "double detections", since the information about the object may be merged into one of the object queries. (ii) Another reason this updating is useful is that the object queries may be updated with information about other objects detected by different object queries. This may be advantageous, for example, because the objective of the method disclosed herein is the detection of a scene in an image, and the scene may contain information indicating a relationship between two objects.

[0053] Step E further comprises updating each of the plurality of graph vectors based on at least one other of the plurality of graph vectors, which updating may be considered useful for reason (ii) above.

[0054] The update process may also include updating the graph vectors and / or the matching query vectors based on the direct image vectors. Any of the above considerations may apply to this update step.

[0055] In some embodiments, the update process includes repeatedly performing the update for a particular number of iterations / length of time. Steps A-E may be performed in any order. Steps may be performed in a different order in some iterations. The update process in some embodiments may not include step E and / or step D.

[0056] In an embodiment where step D is not included, the decoder 25 may calculate the correlation / similarity between the updated graph vector and the at least one updated object query vector to generate an updated matching query vector. Alternatively, the score head 29 may calculate the correlation / similarity between the updated graph vector and the at least one updated object query vector to generate an intermediate vector. In this case (when the score head 29 calculates the similarity), the matching query may not be input to the decoder 25.

[0057] To extract a matching score from a vector (i.e., an updated matching query vector or an intermediate vector), the score head 29 may reduce the vector to a one-dimensional vector and then use an activation function to extract a value. This value may be normalized to generate a matching score. A simple multi-layer perceptron (e.g., consisting of a first linear layer, an activation, and a second linear layer) may be used to extract the matching score.

[0058] As presented herein, a scene graph may be maintained in the form of linked data nodes.

[0059] 7 illustrates a method, which may be executed by the system 20, for detecting a scene (i.e., a combination of at least one object and at least one action or relationship) in an input image.

[0060] Step S11 involves loading the input image and the scene graph.

[0061] Step S12 comprises encoding the input image. This may comprise the encoding step described above with reference to figure 5. This may be performed by the image encoder 21.

[0062] Step S13 comprises encoding the scene graph (which may be called the input graph). This step may be performed by the graph encoder 23.

[0063] Step S14 includes generating an object query and a matching score query.

[0064] Step S15 comprises iteratively updating the query and graph vectors with each other and with the encoded image. Step S15 may comprise the encoding step described above with reference to FIG.

[0065] Step S16 comprises predicting the area and category of at least one object (corresponding to the input graph) in the input image, the mask nodes of the input graph (i.e. the labels of any mask nodes) and a matching score indicating the similarity between the input image and the input graph. The input scene graph may contain information indicating which nodes are "targets". This can be in the form of labels or information contained in the labels. Step S16 in this case comprises predicting preferentially the area / category of the object corresponding to the target in the input scene graph. Alternatively, the object corresponding to the target may be selected after the process of predicting the area and category of at least one object in the input image. In this way, the targets disclosed herein may be used in object detection tasks. That is, objects corresponding to (located in) a particular scene graph may be detected using the method disclosed herein.

[0066] In some embodiments, the method may not include (at least part of) step S14. The at least one object query and / or matching query may be provided directly rather than being generated.

[0067] It will be understood that the method of FIG. 7 corresponds to functionality of system 20 depicted in FIG. 4 and that features / aspects of one may apply to the other.

[0068] 8 illustrates a system 40 for detecting a scene (i.e., a combination of at least one object and at least one action or relationship) in an input image. The system 40 receives as input an input image and a scene graph, and outputs information indicative of at least one object detected in the input image that corresponds to the scene graph, and a matching score between the input image and the scene graph. The system 40 can be thought of as performing the method of FIG. 7.

[0069] The system 40 includes an image encoder 41, an object query generating unit 42, a graph encoder 43, a score query generating unit 44, a decoder 45, a region prediction unit 46, a category prediction unit 47, a node prediction unit 48, and a score prediction unit 49. FIG. 4 illustrates an input image and an input scene graph. The system 40 corresponds to the system 20, and corresponding elements are given corresponding reference numerals, and duplicated descriptions are omitted. The system 40 additionally includes an object query generating unit 42 and a score query generating unit 44 to generate at least one object query and a matching query (or at least one object query vector and a matching query vector). In other embodiments, the system 40 may not include the object query generating unit 42 and / or the score query generating unit 44, and at least one object query and / or a matching query may be directly provided. The object query vector and the matching query vector may be initialized by values ​​that are independent of the input image or scene graph, for example, random or predefined values, or "0". That is, the vectors may be considered as "blank".

[0070] FIG. 9 shows some input images, the input scene graph, and the result.

[0071] In part A, the scene graph has two nodes, labeled "person" and "dog", connected by a node labeled "near". The two people are detected in the input image (dashed box) as corresponding to objects in the scene graph, with a predicted matching score of 0.8. In this example, the node "person" is the target.

[0072] In part B, the scene graph has two nodes, labeled "person" and "dog", connected by a node labeled "near", to which is connected a further node labeled "walking". A person is detected in the input image as corresponding to an object in the scene graph, with a predicted matching score of 0.8. In this example, the node "person" is the target.

[0073] In part C, the scene graph has a node labeled "Person" connected to a node labeled "Clothing", which is connected to a node labeled with an image of a coat. A person is detected in the input image as corresponding to an object in the scene graph, with a predicted matching score of 0.75. In this example, the node "Person" is the target. Incidentally, note that in this example, the nodes are labeled with images. The labels of the nodes may include text and / or images and / or audio. The images or audio may be considered to be converted into vectors.

[0074] In part D, the scene graph has a node labeled "Dog", connected to a mask node. A dog is detected in the input image as corresponding to an object in the scene graph, and a label "standing" is predicted for the mask node based on the input image. A matching score of 0.9 is predicted. In this example, the node "Dog" is the target.

[0075] In part E, the scene graph has a mask node connected to a node labeled "statue" through a node labeled "near". A dog is detected in the input image as corresponding to an object in the scene graph, and the label "dog" is predicted for the mask node based on the input image. A matching score of 0.8 is predicted. In this example, the mask node is the target.

[0076] Below part E, a graph represents the probability of two detected objects, namely "dog" and "person". That is, both the dog and the person may be considered to be detected as objects in the input image (i.e., the information of these objects is added to the object query vector), and the probability may be considered to indicate the correlation between the object query vectors representing the dog and the person and the graph vector. In some implementations, an object threshold is used, whereby only objects with a probability or "correlation" above the threshold are extracted as "detected objects" from the object query vector. In some implementations, any object represented by the updated object query vector is extracted as a detected object. In some implementations, the number of detected objects depends on the number of object query vectors. For example, in part E, if there is only one object query vector, only "dog" is detected (e.g., because "dog" has a higher probability / correlation), even if "person" can be easily detected as an object corresponding to the scene graph.

[0077] In part F, a different input image is used (compared to A-E). The scene graph has a mask node connected to a node labeled "sitting". Multiple people and cats are detected in the input image as corresponding to objects in the scene graph, and the label "people / cats" (i.e., two labels "people" and "cats") are predicted for the mass nodes based on the input image. A matching score of 0.9 is predicted. Below part F, a graph represents the probability of the two detected objects "cat" and "people". In contrast to part E, in this case, the probabilities (i.e., correlation with the graph vectors) of the objects are similar (e.g., both can be considered to be above a threshold), so that the region prediction unit and the category prediction unit predict both objects as corresponding to objects in the scene graph. In the case of part F, it can be considered that there are multiple object query vectors, and therefore multiple objects may be detected in the image. In part F, the mask node can be considered to be used as a wildcard. In this example, the mask node is the target.

[0078] 10 illustrates an embodiment using the system 20 or 40 and the method disclosed herein to find at least one of a plurality of input images that best matches (is most similar to) a scene graph. Here, multiple input images are received instead of just one. A scene graph is also received as an input. The system 20 or 40 performs the above method / process for each input image (i.e., attempts to detect objects in the image that correspond to objects in the scene graph and predicts a matching score indicating the similarity between the input image and the scene graph). The input images may then be ranked based on their matching scores, and at least one image may be selected as the one that best matches the scene graph.

[0079] System 20 or 40 may be configured to perform the embodiment depicted in Fig. 10 (i.e., to rank input images and / or select at least one "best match"). The present disclosure provides methods and / or devices / systems for performing the embodiment of Fig. 10.

[0080] Figure 11 illustrates a method of training system 20 or 40 and / or the method illustrated in Figure 7. Step 21 includes loading training data having at least one input image and at least one scene graph, and information indicative of objects in the at least one input image that correspond to objects in the scene graph, and matching scores between the at least one input image and the at least one scene graph.

[0081] Steps S22 to S29 relate to one iteration of the training process, which consists in executing the method of Fig. 7 and then calculating an error based on the difference between the obtained result and the result indicated by the training data. One iteration is based on a combination of an input image and a scene graph.

[0082] Step S22 comprises masking at least one node of the relevant scene graph, i.e. removing the label of at least one node. This step is not mandatory: it may be performed only in some iterations (or at all).

[0083] Step S23 comprises encoding the input image to generate an image vector, step S24 comprises encoding the scene graph to generate a graph vector, and step S25 comprises generating an object query and a score / matching query, which may alternatively comprise being provided with a query.

[0084] Step S26 includes performing an update process to generate an updated vector, as described above. Step S27 includes predicting the region and category of at least one detected object, predicting the label of a mask node, and predicting a matching score, as described above.

[0085] Step S28 comprises comparing the results with those indicated by the training data (which act as ground truth data) and calculating the error between them. The error may include the difference between the predicted matching score and the matching score indicated by the training data. Any loss for classification problems may be used, such as any standard loss, e.g., cross-entropy loss.

[0086] Step S29 comprises determining whether the error has converged. Step S29 may comprise, for example, determining whether the error is below a threshold error for a certain number of iterations. If the error has converged (yes), the training process ends. If the error has not converged (no), the process proceeds to step S30. Step S30 comprises weight adjustment. Step S30 comprises adjusting at least one weight of at least one network used by the image encoder 21, the graph encoder 23, or the decoder 25. That is, the weight adjustment may be performed for at least one of the attention networks / mechanisms therein.

[0087] After step S30, the process returns to step S21 where another combination of training input images and training scene graphs is loaded.

[0088] The training process may have a predetermined number of iterations (e.g., rather than iterating until error convergence). The training process may not have step S21; instead, the images and scene graph for training may be loaded in an earlier step in the training process.

[0089] FIG. 12 is a schematic diagram of system 60. System 60 corresponds to systems 20 and 40, and corresponding reference numerals are used, and duplicated description is omitted. FIG. 12 includes information about the size and number of vectors at various stages of processing of system 60 in a specific example that is useful for understanding but does not limit the present invention. This information can be applied to systems 20 and 40 and the methods disclosed herein. Incidentally, the size of the compressed image depends on the convolutional neural network (CNN) with a compression ratio. In the example of FIG. 12, a CNN with a compression ratio of 4 is used.

[0090] As depicted, the region prediction may take the form of bounding box coordinates.

[0091] Fig. 13 is a schematic diagram illustrating an example of the use of system 20 or 40 or any method described herein. In Fig. 13, the disclosed method is used to find a person carrying a bag. For example, this may be useful in finding a person whose bag has been stolen, or to find a person carrying a particular bag based on a description of the perpetrator of a crime.

[0092] The scene graph is a node labeled "person" connected to a node labeled with an image of a bag via a node labeled "carrying". The input images in this case are stills from a video recording. The process described with reference to FIG. 10 is used, for example, to find at least one image that best matches the scene graph. The detected object (in this case the person carrying the bag) is output. A matching score may also be output. As shown in the lower part of FIG. 13, the system 20 or 40 may be used to track a scene, i.e. to track a person carrying a bag based on a series of recorded images / stills. In this case, the node labeled "person" is of interest. Such a use of the invention may be considered as object tracking.

[0093] FIG. 14 is a schematic diagram illustrating another example of the use of system 20 or 40 or any method described herein. In FIG. 14, the disclosed method is used to find an object placed on a cart. The scene graph is a mass node connected to a node labeled cart through a node labeled "top". The system detects two objects in the input image shown in the example, each placed on the cart. The system may also output a matching score.

[0094] The aspects disclosed herein can detect specific target objects, motions, and scenes using a single unified input without post-processing, i.e., a specific combination of at least one object and at least one action or region can be detected in the input image without the need for post-processing, and matching scores can also be generated without the need for a separate post-processing phase (or a separate NN as in the previous methods described above).

[0095] Aspects disclosed herein can be considered to relate to a method for detecting specific objects, motions, and scenes from input images using a scene graph as a single unified input query while eliminating post-processing to filter objects and improve detection accuracy by identifying objects and their attributes using the scene graph as priors.

[0096] The drawings are illustrated pictorially in the figures, it will be understood that photographs may be used.

[0097] 15 is a block diagram of an information processing device 900, such as a data storage server, or computing device 900, which may embody the present invention and may be used to perform some or all of the operations of the methods embodying the present invention and to carry out some or all of the tasks of the apparatus of the embodiments. The computing device 900 may be used to perform any of the method steps described above, for example any of steps S11-S16 and / or S21-S30.

[0098] The computing device 900 has a processor 993 and a memory 994. Optionally, the computing device also includes a network interface 997 for communication with other such computing devices, e.g., other computing devices of invention embodiments. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 996, and a display unit, such as one or more monitors 995. These elements may facilitate user interaction. The components may be connected to each other via a bus 992.

[0099] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data accessible to a computer (e.g., one or more processors) that cause the computer to perform one or more functions or operations. For example, computer-executable instructions may include instructions for performing any of the methods disclosed herein or any of the method steps disclosed herein, such as steps S11-S16 and / or S21-S30. Thus, the term "computer-readable storage medium" may also include any medium capable of storing, encoding or carrying a set of instructions for execution by a machine, causing the machine to perform any one or more of the method steps of the present disclosure. The term "computer-readable storage medium" may be interpreted accordingly to include, but is not limited to, solid-state memory, optical media, and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0100] The processor 993 is configured to control the computing device and perform processing operations, such as executing computer program code stored in the memory 994 to perform any of the method steps described herein. The memory 994 stores data read and written by the processor 993, and may store at least one input image and / or at least one vector and / or at least one weight of at least one network and / or at least one vector and / or at least one error and / or at least one prediction and / or other data. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, and the like. A processor may include a multiple instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or a processor implementing a combination of instruction sets. A processor may also include one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, and the like. In one or more embodiments, the processor is configured to execute instructions to perform the operations and operations discussed herein. The processor 993 can be considered to include any of the methods described above. Any operation described as being performed by a module can be performed by the computer, and for example, by the processor 993, as a method.

[0101] The display unit 995 may display representations of data stored by the computing device, such as representations of predictions (e.g., bounding boxes), and / or a "best match" image and / or a GUI window, and / or interactive representations that allow a user to interact with the apparatus 900, for example by drag-and-drop or selection interactions, and / or any other output as described above, and may also display cursors and dialog boxes and screens that allow interaction between a user and programs and data stored on the computing device. The input mechanism 996 may enable a user to input data and instructions into the computing device, such as allowing a user to enter user input as described above.

[0102] A network interface (network I / F) 997 may be connected to a network such as the Internet, and can be connected to other such computing devices via the network. The network I / F 997 may control data input from other devices or data output to other devices via the network. Other peripheral devices such as a microphone, a speaker, a printer, a power supply unit, a fan, a case, a scanner, a trackball, etc. may be included in the computing device.

[0103] A method embodying the present invention may be performed on a computing device / apparatus 900 such as the one depicted in FIG. 15. Such a computing device need not have all the components depicted in FIG. 15, but may consist of some of these components. For example, the apparatus 900 may have a processor 993 and a memory 994 connected to the processor 993. Alternatively, the apparatus 900 may have a processor 993, a memory 994 connected to the processor 993, and a display 995. A method embodying the present invention may be performed by a single computing device that communicates with one or more data storage servers over a network. The computing device may be a data storage itself that stores at least a portion of the data.

[0104] Methods embodying the invention may be performed by a plurality of computing devices cooperating with one another, one or more of which may be data storage servers that store at least a portion of the data.

[0105] The invention may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention may also be implemented as a computer program or computer program product, i.e. a computer program tangibly embodied in a non-transitory information carrier, for example a machine-readable storage medium, or in a propagated signal, for execution by or to control the operation of one or more hardware modules.

[0106] The computer program may take the form of a stand-alone program, a computer program portion, or more than one computer program, and may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed in one module, or in multiple modules that are at one site or distributed across multiple sites and interconnected by a communication network.

[0107] The method steps of the present invention may be performed by one or more programmable processors executing computer programs to perform the functions of the present invention by operating on input data and generating output. Apparatus of the present invention may be implemented as programmed hardware or as special purpose logic circuitry including, for example, a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (AISC).

[0108] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memory devices for storing instructions and data.

[0109] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments.

[0110] The present disclosure extends to the following appendices:

[0111] (Appendix 1) 1. A computer-implemented method comprising: generating a plurality of graph vectors based on an input / query scene graph; encoding an input image to generate a plurality of image vectors; performing an update process (decoder process) to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: (iteratively) updating the at least one object query vector based on the plurality of graph vectors (such that the at least one object query vector includes information indicative of at least one object in the scene graph); and (iteratively) updating the at least one object query vector based on the plurality of image vectors (with information indicative of at least one (relevant) object detected in the input image); (iteratively) updating the plurality of graph vectors based on the at least one object query vector (with information indicative of at least one (relevant) object detected in the input image); extracting information indicative of a region of at least one object (detected in the input image) and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the (at least one object detected therein) of the input image and the input scene graph based on the (similarity / correlation between) the at least one updated object query vector and the plurality of updated graph vectors; The method according to claim 1,

[0112] (Appendix 2) The update process comprises: iteratively updating the at least one object query vector based on the plurality of graph vectors; iteratively updating the at least one object query vector based on the plurality of image vectors; iteratively updating the plurality of graph vectors based on the at least one object query vector; having 2. The computer-implemented method of claim 1.

[0113] (Appendix 3) (iteratively) updating the at least one object query vector based on the plurality of graph vectors comprises updating the at least one object query vector based on the plurality of graph vectors such that the at least one object query vector includes information indicative of at least one object in the scene graph. 3. The computer-implemented method of claim 1 or 2.

[0114] (Appendix 4) (iteratively) updating the at least one object query vector based on the plurality of image vectors comprises updating the at least one object query vector based on the plurality of image vectors with information indicative of at least one (relevant) object detected in the input image. 4. The computer-implemented method of any one of claims 1 to 3.

[0115] (Appendix 5) (iteratively) updating the plurality of graph vectors based on the at least one object query vector comprises updating the plurality of graph vectors based on the at least one object query vector with information indicative of at least one (relevant) object detected in the input image. 5. A computer-implemented method according to any one of claims 1 to 4.

[0116] (Appendix 6) The updating process includes updating an input matching query vector to generate an updated matching query vector; updating the input matching query vector comprises (iteratively) updating the input matching query vector based on the at least one object query vector and the plurality of graph vectors to generate an updated matching query vector; Calculating the matching score includes calculating the matching score based on the updated matching query vector. 6. The computer-implemented method of any one of claims 1 to 5.

[0117] (Appendix 7) updating an input matching query vector based on the at least one updated object query vector and the plurality of updated graph vectors to generate an updated matching query vector; Calculating the matching score includes calculating the matching score based on the updated matching query vector. 7. A computer-implemented method according to any one of claims 1 to 6.

[0118] (Appendix 8) (iteratively) updating the at least one object query vector based on the plurality of graph vectors comprises adding and / or updating at least one value of the at least one object query vector based on the plurality of graph vectors. 8. A computer-implemented method according to any one of claims 1 to 7.

[0119] (Appendix 9) (iteratively) updating the at least one object query vector based on the plurality of graph vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of graph vectors. 9. A computer-implemented method according to any one of claims 1 to 8.

[0120] (Appendix 10) (iteratively) updating the at least one object query vector based on the plurality of image vectors comprises adding and / or updating at least one value of the at least one object query vector based on the plurality of image vectors. 10. The computer-implemented method of any one of claims 1 to 9.

[0121] (Appendix 11) and (iteratively) updating the at least one object query vector based on the plurality of image vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of image vectors. 11. The computer-implemented method of any one of claims 1 to 10.

[0122] (Appendix 12) (iteratively) updating the plurality of graph vectors based on the at least one object query vector comprises adding and / or updating at least one value of at least one of the plurality of graph vectors based on at least one of the at least one object query vector. 12. The computer-implemented method of any one of claims 1 to 11.

[0123] (Appendix 13) (iteratively) updating the plurality of graph vectors based on the at least one object query vector comprises updating at least one of the plurality of graph vectors based on a correlation between at least one of the at least one object query vector and the plurality of graph vectors. 13. A computer-implemented method according to any one of claims 1 to 12.

[0124] (Appendix 14) (iteratively) updating the at least one object query vector based on the plurality of graph vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of graph vectors; and / or (iteratively) updating the at least one object query vector based on the plurality of image vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of image vectors; and / or (iteratively) updating the plurality of graph vectors based on the at least one object query vector comprises updating at least one of the plurality of graph vectors based on a correlation between at least one of the at least one object query vector and the plurality of graph vectors. 2. The computer-implemented method of claim 1.

[0125] (Appendix 15) and iteratively updating the input matching query vector based on the at least one object query vector and the plurality of graph vectors comprises updating at least one value of the input matching query vector based on the at least one object query vector and the plurality of graph vectors. 15. The computer-implemented method of any one of claims 6 to 14.

[0126] (Appendix 16) Iteratively updating the input matching query vector based on the at least one object query vector comprises updating at least one value of the input matching query vector based on a correlation between the at least one object query vector and the input matching query vector; iteratively updating the input matching query vector based on the plurality of graph query vectors comprises updating the input matching query vector based on a correlation between the plurality of graph query vectors and the input matching query vector. 16. The computer-implemented method of any one of claims 6 to 15.

[0127] (Appendix 17) (iteratively) updating the input matching query vector comprises updating the input matching query vector based on similarities / correlations between the at least one object query vector and the plurality of graph vectors. 17. The computer-implemented method of any one of claims 6 to 16.

[0128] (Appendix 18) The updating process comprises using an attention-based network / mechanism / attention scheme. 18. A computer-implemented method according to any one of claims 1 to 17.

[0129] (Appendix 19) (iteratively) updating the at least one object query vector based on the plurality of graph vectors comprises using an attention-based network / attention scheme or an attention network / mechanism (to determine the correlation). 19. The computer-implemented method of any one of claims 1 to 18.

[0130] (Appendix 20) (iteratively) updating the at least one object query vector based on the plurality of image vectors comprises using an attention-based network / attention scheme or an attention network / mechanism (to determine the correlation). 20. The computer-implemented method of any one of claims 1 to 19.

[0131] (Appendix 21) (iteratively) updating the plurality of graph vectors based on the at least one object query vector comprises using an attention-based network / attention scheme or an attention network / mechanism (to determine the correlation). 21. The computer-implemented method of any one of claims 1 to 20.

[0132] (Appendix 22) and iteratively updating the input matching query vector based on the at least one object query vector and the plurality of graph vectors comprises using an attention-based network / attention scheme or an attention network / mechanism (to determine the correlation). 22. The computer-implemented method of any one of claims 1 to 21.

[0133] (Appendix 23) the input scene graph having mask nodes; 23. The computer-implemented method of any one of claims 1 to 22.

[0134] (Appendix 24) the mask node (which may be connected to at least one other node of the input scene graph) does not contain any information about the object or relationship or behavior it represents; 24. The computer-implemented method of claim 23.

[0135] (Appendix 25) The mask nodes do not contain labels. 25. The computer-implemented method of claim 23 or 24.

[0136] (Appendix 26) the mask nodes represent objects or relationships or actions to be predicted; 26. The computer-implemented method of any one of claims 23 to 25.

[0137] (Appendix 27) (iteratively) updating a plurality of graph vectors based on the at least one object query vector comprises adding and / or updating information in the graph vectors indicating mask nodes based on information of at least one object in the input image. 27. The computer-implemented method of any one of claims 23 to 26.

[0138] (Appendix 28) The computer-implemented method includes extracting information about the mask nodes from the plurality of updated graph vectors to predict a label (object or relation or action) of the mask nodes based on the input image. 28. The computer-implemented method of any one of claims 1 to 27.

[0139] (Appendix 29) the input scene graph comprises mask nodes whose labels are to be predicted; The computer-implemented method includes extracting information about the mask nodes from the plurality of updated graph vectors and / or the at least one updated object query vector to predict a label (object or relationship or action) of the mask nodes based on the input image. 23. The computer-implemented method of any one of claims 1 to 22.

[0140] (Appendix 30) generating the plurality of graph vectors comprises encoding (or using an encoder) the input scene graph; 30. The computer-implemented method of any one of claims 1 to 29.

[0141] (Appendix 31) the input scene graph comprises at least one object and (each node representing) at least one of a relationship (related to the at least one object) or an action (related to the at least one object); 31. The computer-implemented method of any one of claims 1 to 30.

[0142] (Appendix 32) the input scene graph having first and second objects and a relationship between the first and second objects; 32. The computer-implemented method of any one of claims 1 to 31.

[0143] (Appendix 33) generating the plurality of graph vectors comprises encoding information of the at least one object and the at least one relationship or action into the plurality of graph vectors. 33. The computer-implemented method of any one of claims 1 to 32.

[0144] (Appendix 34) generating the plurality of graph vectors comprises encoding the input scene graph using a self-attention network / mechanism or a message passing mechanism. 34. The computer-implemented method of any one of claims 1 to 33.

[0145] (Appendix 35) Encoding the image comprises using a (transformer-based) self-attention network / mechanism. 35. The computer-implemented method of any one of claims 1 to 34.

[0146] (Appendix 36) the object query vector before the update process does not contain information about any object to be detected; 36. The computer-implemented method of any one of claims 1 to 35.

[0147] (Appendix 37) the object query vector before the update process is initialized with values ​​that are unrelated to the input scene graph or the input image; 37. The computer-implemented method of any one of claims 1 to 36.

[0148] (Appendix 38) the input matching query vector before the update process is initialized with a value that is unrelated to the input scene graph or the input image or the object query vector; 38. The computer-implemented method of any one of claims 1 to 37.

[0149] (Appendix 39) the updating process comprises (iteratively) updating the plurality of graph vectors based on the plurality of image vectors; 39. The computer-implemented method of any one of claims 1 to 38.

[0150] (Appendix 40) the updating process comprises (iteratively) updating the input matching query vector based on the plurality of image vectors; 40. The computer-implemented method of any one of claims 1 to 39.

[0151] (Appendix 41) the at least one object query vector comprises a plurality of object query vectors; the updating process comprises (iteratively) updating each of the plurality of object query vectors based on at least one other object query vector of the plurality of object query vectors; 41. The computer-implemented method of any one of claims 1 to 40.

[0152] (Appendix 42) the updating process comprises updating each of the plurality of graph vectors based on at least one other graph vector of the plurality of graph vectors. 42. The computer-implemented method of any one of claims 1 to 41.

[0153] (Appendix 43) the at least one object query vector comprises a plurality of object query vectors, and the updating process comprises (iteratively) updating each of the plurality of object query vectors based on at least one other object query vector of the plurality of object query vectors; and / or the updating process comprises (iteratively) updating each of the plurality of graph vectors based on at least one other graph vector of the plurality of graph vectors. 43. The computer-implemented method of any one of claims 1 to 42.

[0154] (Appendix 44) The updating process comprises repeating an updating step a predefined number of times or for a predefined length of time. 44. The computer-implemented method of any one of claims 1 to 43.

[0155] (Appendix 45) the updating process comprising repeating the updating steps in a random order in each iteration; 45. The computer-implemented method of any one of claims 1 to 44.

[0156] (Appendix 46) the input scene graph having a first node representing a first object, a second node representing a relationship, and a mask node; the second node is connected between the first node and the mask node; 46. ​​The computer-implemented method of any one of claims 1 to 45.

[0157] (Appendix 47) The input scene graph includes information indicating a target node; The computer-implemented method includes selecting at least one detected object as corresponding to the node of interest. 47. The computer-implemented method of any one of claims 1 to 46.

[0158] (Appendix 48) performing the computer-implemented method of any one of claims 1 to 47 a plurality of times, each time using a different input image, based on an input scene graph; selecting at least one object detected in a plurality of said input images as a node of interest; A computer-implemented method.

[0159] (Appendix 49) performing the computer-implemented method of any one of claims 1 to 47 multiple times, each time with a different input image, based on an input scene graph to find at least one of the input images that is most similar to the input scene graph. A computer-implemented method.

[0160] (Appendix 50) The computer-implemented method includes ranking the input images based on their matching scores. 50. The computer-implemented method of claim 48 or 49.

[0161] (Appendix 51) The computer-implemented method includes selecting at least one of the input images having a highest matching score. 51. The computer-implemented method of any one of claims 48 to 50.

[0162] (Appendix 52) the input images are a sequence of images from a video; 52. The computer-implemented method of any one of claims 48 to 51.

[0163] (Appendix 53) the input images and the input scene graph are training images and training scene graphs, respectively, and are associated with at least one of training regions (as ground truth data), training categories, training mask node labels, and training matching scores; The computer-implemented method further comprises comparing at least one of the regions, the categories, the labels of the mask nodes, and the matching scores to at least one of the training regions, training categories, training mask node labels, and training matching scores, respectively, and updating at least one network weight based on the comparison. 53. The computer-implemented method of any one of claims 1 to 52.

[0164] (Appendix 54) comparing at least one of the regions, the categories, the labels of the mask nodes, and the matching scores with at least one of the training regions, training categories, training mask node labels, and training matching scores, respectively, comprises calculating an error therebetween; updating the at least one network weight comprises updating the at least one network weight based on the error. 54. The computer-implemented method of claim 53.

[0165] (Appendix 55) updating the at least one network weight comprises updating at least one network weight of the at least one attention-based network / mechanism or self-attention network / mechanism. 55. The computer-implemented method of claim 53 or 54.

[0166] (Appendix 56) updating the at least one network weight comprises updating weights of each attention-based network / mechanism and self-attention network / mechanism. 56. The computer-implemented method of any one of claims 53 to 55.

[0167] (Appendix 57) performing the training process multiple times using multiple sets of training scenes and training input images; The training process comprises or is the computer-implemented method of any one of claims 53 to 56. A computer-implemented method.

[0168] (Appendix 58) The computer-implemented method comprises running / iterating / repeating the training process until an error converges or falls below an error threshold (iterations). 58. The computer-implemented method of claim 57.

[0169] (Appendix 59) The computer-implemented method comprises running / iterating / repeating the training process until an error falls below an error threshold (a predetermined number of consecutive iterations). 59. The computer-implemented method of claim 57 or 58.

[0170] (Appendix 60) The computer implemented method comprises running / iterating / repeating the training process a defined number of times. 60. The computer-implemented method of any one of claims 57 to 59.

[0171] (Appendix 61) The computer-implemented method comprises subsequently performing the computer-implemented method of any one of claims 48 to 52. 61. The computer-implemented method of any one of claims 57 to 60.

[0172] (Appendix 62) When executed on a computer, the computer generating a plurality of graph vectors based on an input / query scene graph; encoding an input image to generate a plurality of image vectors; performing an update process (decoder process) to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: (iteratively) updating the at least one object query vector based on the plurality of graph vectors (such that the at least one object query vector includes information indicative of at least one object in the scene graph); and (iteratively) updating the at least one object query vector based on the plurality of image vectors (with information indicative of at least one (relevant) object detected in the input image); (iteratively) updating the plurality of graph vectors based on the at least one object query vector (with information indicative of at least one (relevant) object detected in the input image); extracting information indicative of a region of at least one object (detected in the input image) and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the (at least one object detected therein) of the input image and the input scene graph based on the (similarity / correlation between) the at least one updated object query vector and the plurality of updated graph vectors; A computer program product for carrying out the method comprising:

[0173] (Appendix 63) A memory and a processor connected to the memory, the processor comprising: generating a plurality of graph vectors based on an input / query scene graph; encoding an input image to generate a plurality of image vectors; performing an update process (decoder process) to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: (iteratively) updating the at least one object query vector based on the plurality of graph vectors (such that the at least one object query vector includes information indicative of at least one object in the scene graph); and (iteratively) updating the at least one object query vector based on the plurality of image vectors (with information indicative of at least one (relevant) object detected in the input image); (iteratively) updating the plurality of graph vectors based on the at least one object query vector (with information indicative of at least one (relevant) object detected in the input image); extracting information indicative of a region of at least one object (detected in the input image) and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the (at least one object detected therein) of the input image and the input scene graph based on the (similarity / correlation between) the at least one updated object query vector and the plurality of updated graph vectors; An information processing device configured to execute the above. [Explanation of symbols]

[0174] 20,40,60 system 21,41 Image Encoder 23,43 Graph Encoder 25,45 Decoder 26 Area Head 27 Category Head 28 Node Head 29 Score Head 42 Object Query Generation Unit 44 Score Query Generation Unit 46 Area Prediction Unit 47 Category Prediction Units 48 Node Prediction Unit 49 Score Prediction Unit 900 Information processing equipment (computing devices) 992 Bus 993 Processor 994 Memory 995 Display Unit 996 Input Mechanism 997 Network Interface

Claims

1. 1. A computer-implemented method comprising: generating a plurality of graph vectors based on an input scene graph; encoding an input image to generate a plurality of image vectors; performing an update process to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: updating the at least one object query vector based on the plurality of graph vectors; updating the at least one object query vector based on the plurality of image vectors; updating the plurality of graph vectors based on the at least one object query vector. extracting information indicative of a region of at least one object and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the input image and the input scene graph based on the at least one updated object query vector and the plurality of updated graph vectors; The method according to claim 1,

2. The update process comprises: iteratively updating the at least one object query vector based on the plurality of graph vectors; and / or iteratively updating the at least one object query vector based on the plurality of image vectors; and / or iteratively updating the plurality of graph vectors based on the at least one object query vector. having 2. The computer-implemented method of claim 1.

3. The updating process includes updating an input matching query vector to generate an updated matching query vector; updating the input matching query vector comprises updating the input matching query vector based on the at least one object query vector and the plurality of graph vectors to generate an updated matching query vector; Calculating the matching score includes calculating the matching score based on the updated matching query vector.

3. A computer-implemented method according to claim 1 or 2.

4. Updating the at least one object query vector based on the plurality of graph vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of graph vectors; and / or and / or updating the at least one object query vector based on the plurality of image vectors comprises updating the at least one object query vector based on a correlation between the at least one object query vector and the plurality of image vectors. updating the plurality of graph vectors based on the at least one object query vector comprises updating at least one of the plurality of graph vectors based on a correlation between at least one of the at least one object query vector and the plurality of graph vectors.

3. A computer-implemented method according to claim 1 or 2.

5. updating the input matching query vector comprises updating the input matching query vector based on a similarity between the at least one object query vector and the plurality of graph vectors.

4. The computer-implemented method of claim 3.

6. the input scene graph includes mask nodes whose labels are to be predicted; The computer-implemented method includes extracting information about the mask nodes from the plurality of updated graph vectors and / or the at least one updated object query vector to predict the labels of the mask nodes based on the input image.

3. A computer-implemented method according to claim 1 or 2.

7. performing the computer-implemented method of claim 1 or 2 a number of times, each time with a different input image, based on an input scene graph to find at least one of the input images that is most similar to the input scene graph. A computer-implemented method.

8. the input images and the input scene graph are training images and training scene graphs, respectively, associated with at least one of training regions, training categories, training mask node labels, and training matching scores; the input scene graph includes mask nodes whose labels are to be predicted; The computer-implemented method further comprises comparing at least one of the regions, the categories, the labels of the mask nodes, and the matching scores to at least one of the training regions, training categories, training mask node labels, and training matching scores, respectively, and updating at least one network weight based on the comparison.

3. A computer-implemented method according to claim 1 or 2.

9. When executed on a computer, the computer generating a plurality of graph vectors based on an input scene graph; encoding an input image to generate a plurality of image vectors; performing an update process to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: updating the at least one object query vector based on the plurality of graph vectors; updating the at least one object query vector based on the plurality of image vectors; updating the plurality of graph vectors based on the at least one object query vector. extracting information indicative of a region of at least one object and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the input image and the input scene graph based on the at least one updated object query vector and the plurality of updated graph vectors; A computer program product for carrying out the method comprising:

10. A memory and a processor connected to the memory, the processor comprising: generating a plurality of graph vectors based on an input scene graph; encoding an input image to generate a plurality of image vectors; performing an update process to update the plurality of graph vectors and at least one object query vector to generate a plurality of updated graph vectors and at least one updated object query vector, the update process comprising: updating the at least one object query vector based on the plurality of graph vectors; updating the at least one object query vector based on the plurality of image vectors; updating the plurality of graph vectors based on the at least one object query vector. extracting information indicative of a region of at least one object and a category of the at least one object from the at least one updated object query vector; calculating a matching score indicative of a similarity between the input image and the input scene graph based on the at least one updated object query vector and the plurality of updated graph vectors; An information processing device configured to execute the above.

Images