Scene graph modification based on natural language commands

Abstract

Embodiments of the present disclosure relate to natural language command based scene graph modification. Systems and methods for natural language processing are described. Embodiments are configured to receive a structured representation of a search query, where the structured representation includes a plurality of nodes and at least one edge connecting two of the nodes; receive a modification expression for the search query, where the modification expression includes a natural language expression; generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features; and perform a search based on the modified structured representation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to natural language processing, and more particularly to natural language processing using conditional graph modification techniques. BACKGROUND

[0002] Natural language processing (NLP) refers to the use of computers to interpret natural language. In some cases, NLP tasks involve assigning annotation data, such as grammatical information, to words or phrases within a natural language expression. Different kinds of machine learning algorithms have been applied to NLP tasks. These algorithms can take as input a set of features generated from natural language data. Some algorithms, such as decision trees, use hard if-then rules. Other systems use neural networks or statistical models that make soft, probabilistic decisions based on attaching real-valued weights to input features. These models can express relative probabilities of multiple answers.

[0003] Structured representations, such as parse trees and dependency graphs, play an important role in many NLP tasks. Accordingly, many NLP devices include a parser that maps text to a parse tree or dependency graph. However, conventional NLP devices are unable to modify an existing structured representation based on a natural language modification command. Accordingly, there is a need in the art for systems and methods for altering structured representations. SUMMARY

[0004] The present disclosure provides systems and methods for generating a structured representation for a modified natural language expression. In some embodiments, a structured representation representing an original expression is encoded using a first neural network, and a modified expression representing a change to the original expression is encoded using a second neural network. The encoded structured representation and the encoded modified expression are combined in an embedding space, and the combination result is used to generate an output structured representation representing the modified expression.

[0005] A method, apparatus, and non-transitory computer-readable medium for natural language processing are described. Embodiments of the method, apparatus, and non-transitory computer-readable medium are configured to receive a structured representation of a search query, where the structured representation includes a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, where the modification expression includes a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

[0006] An apparatus and method for natural language processing are described. Embodiments of the apparatus and method include a graph encoder configured to generate structured representation features based on a structured representation, a text encoder configured to generate natural language expression features based on a modification expression, a feature fusion network configured to produce combined features based on the structured representation features and the natural language expression features, a node decoder configured to generate a set of nodes based on the combined features, and an edge decoder configured to generate a set of edges based on the combined features and the set of nodes.

[0007] A method, apparatus, and non-transitory computer-readable medium for training a neural network for natural language processing are described. Embodiments of the method, apparatus, and non-transitory computer-readable medium are configured to identify training data comprising a plurality of annotated training examples, where each of the annotated training examples comprises a source structured representation, a target structured representation, and at least one modification expression, generate, using a graph encoder, structured representation features for the source structured representation, generate, using a text encoder, natural language expression features for the at least one modification expression, combine, using a feature fusion network, the structured representation features and the natural language expression features to produce combined features, generate, using a node decoder, a set of nodes based on the combined features, generate, using an edge decoder, a set of edges based on the combined features and the set of nodes, compare the set of nodes and the set of edges to the target structured representation, and update the neural network based on the comparison. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 An example of a process for providing search results is shown in accordance with various aspects of the present disclosure.

[0009] Figure 2 An example of a system for natural language processing is shown in accordance with various aspects of the present disclosure.

[0010] Figure 3 An example of a neural network for natural language processing is shown in accordance with various aspects of the present disclosure.

[0011] Figure 4 An example of cross-attention fusion is shown in accordance with various aspects of the present disclosure.

[0012] Figure 5 An example of an adjacency matrix style edge decoder is shown in accordance with various aspects of the present disclosure.

[0013] Figure 6 An example of a flat edge-level decoder is shown in accordance with various aspects of the present disclosure.

[0014] Figure 7An example of a scenario diagram is shown in accordance with various aspects of the present disclosure.

[0015] Figure 8 An example of an insertion process is shown in accordance with various aspects of the present disclosure.

[0016] Figure 9 An example of a deletion process is shown in accordance with various aspects of the present disclosure.

[0017] Figure 10 An example of an object replacement process is shown in accordance with various aspects of the present disclosure.

[0018] Figure 11 An example of an attribute replacement process is shown in accordance with various aspects of the present disclosure.

[0019] Figure 12 An example of a process for natural language processing is shown in accordance with various aspects of the present disclosure.

[0020] Figure 13 An example of a process for combining graphical features and textual features is shown in accordance with various aspects of the present disclosure.

[0021] Figure 14 An example of a process for generating a modified structured representation based on combined features is shown in accordance with various aspects of the present disclosure.

[0022] Figure 15 An example of a process for training a neural network for natural language processing is shown in accordance with various aspects of the present disclosure. DETAILED DESCRIPTION

[0023] The present disclosure relates to systems and methods for performing natural language processing (NLP). Some embodiments of the present disclosure relate to conditional graph modification, in which a source graph representing a natural language expression is modified based on a modification command. In some examples, a neural network is trained to perform the modification.

[0024] Structured representations, such as graphs and parse trees, play an important role in many NLP tasks. Accordingly, some NLP devices include a parser that maps text to a graph or a parse tree. However, conventional NLP devices are unable to apply manipulations to the structured representations. In some cases, during an interaction between a human and a computer, text is parsed to produce a structured representation. However, a typical human and computer interaction session or conversation is not single-turn (e.g., image search). Accordingly, parsing a single sentence can not be sufficient.

[0025] If a conventional NLP device is used during a conversation with a human user, it will not be able to update a previously generated graph based on subsequent rounds of the conversation. For example, a user performing a search can start with a phrase that describes the main object or topic they are looking for in the search. Depending on the results, the user can modify their query to add more constraints or give additional information. In this case, a static representation is not suitable to track the user’s updated intent. Thus, conventional NLP devices are not able to engage in seamless multi-round interactions where the user changes the intent of a previous formulation.

[0026] Accordingly, the present disclosure provides methods and systems that perform graph modification. Unlike conventional NLP devices, the NLP devices of the present disclosure include an unconventional element of a neural network model that is trained to update an existing scene graph given a new user command. Thus, the improved NLP devices according to the present disclosure are able to update an existing graph based on subsequent formulations by a user. Accordingly, the improved NLP devices can engage in seamless multi-round interactions with a human user where the user changes the intent of a previous formulation.

[0027] In some embodiments, the structured representation representing the original formulation is encoded using a first neural network, and the modified formulation representing a change to the original formulation is encoded using a second neural network. The encoded structured representation and the encoded modified formulation are combined in an embedding space, and the combination result is used to generate an output structured representation representing the modified formulation. In some examples, the model is based on a graph-based sparse transformer and cross-attention information fusion. According to embodiments, a large graph modification dataset is created and used for training of the model.

[0028] Figure 1 An example of a process for providing search results according to various aspects of the present disclosure is shown. Figure 1 The process of FIG. 1 is an example of how a system for modifying a structured representation can be applied to a system to search images based on a natural language search query. However, this is not a limiting example, and the systems and methods of the present disclosure can also be applied to other applications.

[0029] In some examples, Figure 1 The operations of FIG. 1 are performed by a system including a processor that executes a set of codes to control the functional elements of a device. Additionally or alternatively, the

[0030] In operation 100, a user provides a search query. The user interacts with a computer (or any electronic device capable of human-device interaction) in a human and computer interaction session or dialogue. The dialogue can be more than a single round of exchange. In some cases, the user starts with a phrase that describes the main object or subject she is looking for (e.g., the search phrase or query is “young boy wearing a black shirt”). Depending on the generated results, the user can modify her search query to add more constraints or provide additional information to optimize the search. In some cases, the operations of this step involve referring to Figure 2 the user described, or can be performed by the same.

[0031] In operation 105, the server generates a structured representation (e.g., a graph) that represents the search query. A structured representation can refer to a formalized representation of a linguistic expression. The structured representation (like the natural language expression) can also represent an image. Representing a natural language expression as a structured representation can improve image search results because the structured representation clearly identifies the objects and relationships to be included in the search. In some cases, the operations of this step involve referring to Figure 2 the server described, or can be performed by the same.

[0032] In some cases, the server can use a neural network model to encode the structured representation (i.e., with a graph encoder) and the modified query (i.e., with a text encoder) into an embedding space. The outputs of the encoders can be referred to as structured representation features and natural language expression features, respectively. In some cases, the structured representation features and the natural language expression features are represented as vectors in a common embedding space.

[0033] In some cases, the structured representation can include two or more nodes representing objects or concepts and edges representing relationships between the nodes. In one example, “black” and “shirt” are nodes connected by an “attribute” relationship. In another example, “man” and “hat” are nodes and “wearing” describes the relationship.

[0034] According to embodiments of the present disclosure, a graph can be generated by a graph generator. In the case of a graph representing an image, the graph is referred to as a scene graph (i.e., a semantic representation of an image). The graph includes nodes and edges, where each of the edges connects two respective nodes. The nodes represent objects, while the edges represent attributes or relationships between the respective nodes. For example, a user’s initial search query can be “young boy wearing a black shirt.” A parser is used to parse the search query into a list of objects, e.g., “boy” and “shirt.” In some examples, the objects and their associated attributes and relationships form a set of triples, e.g., (boy, wearing, shirt), (boy, attribute, young), and (shirt, attribute, black).

[0035] In operation 110, the database provides search results based on the graph (e.g., one or more images corresponding to the search query). In some examples, the database provides the search results to the user to enable the user to find content to make a decision. In some cases, the user can modify the search based on the search results (i.e., take the search results as-is, or modify the search query to reach some other results). In some cases, the operations of this step involve referencing Figure 2 the described database, or can be performed by the same.

[0036] In operation 115, the user provides a modification to the search query. In some examples, depending on the generated results from the database, the user can choose to modify her search query to add more constraints or provide additional information to optimize the search. In the above example, the user no longer requires a black shirt. Instead, the user provides the modification query "change to pink shirt."

[0037] As in this example, the modification query need not include a complete expression nor reference every node or edge in the original search query (or original graph). Thus, in some cases, the system determines which nodes and edges are being changed and why they are being changed. In this case, the search query is modified to "young boy wearing a pink shirt," where the attribute associated with the shirt (describing the color) is changed. In various cases, the server adds, changes, or removes nodes or edges of the graph based on the modification query. In some cases, the operations of this step involve referencing Figure 2 the described user, or can be performed by the same.

[0038] In operation 120, the server modifies the graph based on the modification. The objects, attributes, and relationships between the two objects are modified, so an updated structure representation (i.e., scene graph) is generated based on the modification. For example, when the user's modification query is "change to pink shirt," the shirt is no longer black, but pink. The updated objects and their updated associated attributes and relationships form a set of triples, such as (boy, wears, shirt), (boy, attribute, young), and (shirt, attribute, pink).

[0039] In another example, the user no longer requires a shirt regardless of color and modifies the search query to "young boy." The nodes representing the shirt and black are removed, respectively. The server modifies the graph by removing the nodes and edges associated with these nodes. In some cases, the operations of this step involve referencing Figure 2 the described server, or can be performed by the same.

[0040] In operation 125, the database provides updated search results based on the modified graph. For example, the database provides the updated search results to the user, such that the user makes a decision based on the search results (i.e., adopts the updated search results, or continues to modify the updated search query). In some cases, the operations of this step involve referring to Figure 2 the described database, or can be performed by it. In some cases, multiple iterations between the user, the server, and the database are performed until the user is satisfied with the search results.

[0041] Figure 2 An example of a system for natural language processing according to various aspects of the disclosure is shown. The example shown includes a user 200, a user device 205, a server 210, a cloud 245, and a database 250. In one embodiment, the server 210 includes a processor unit 215, a memory unit 220, an input component 225, a neural network 230, a graph generator 235, and a search component 240. However, in some embodiments, the components and functions of the server are located on the user device 205.

[0042] The user 200 can communicate with the server 210 via the user device 205 and the cloud 245. For example, the user 200 provides a search query, and the server 210 generates a graph representing the search query. The search results based on the graph are stored in the database 250. The user 200 provides a modification to the search query, and the server modifies the graph based on the modification. The updated search results are added to the database 250.

[0043] The user device 205 can be a personal computer, a laptop computer, a mainframe computer, a palm computer, a personal assistant, a mobile device, or any other suitable processing device.

[0044] The cloud 245 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, the cloud 245 provides resources without being actively managed by a user. The term “cloud” is sometimes used to describe a data center available to many users over the Internet. Some large cloud networks have functionality distributed from a central server over multiple locations. If a server has a direct or close connection to a user, it is specifically referred to as an edge server. In some cases, the cloud 245 is limited to a single organization. In other examples, the cloud 245 is available to many organizations. In one example, the cloud 245 includes a multi-tiered communication network that includes multiple edge routers and core routers. In another example, the cloud 245 is based on a local switch set in a single physical location.

[0045] The server 210 provides one or more functionalities to users linked through one or more of various networks. In some cases, the server 210 includes a single microprocessor board that includes a microprocessor responsible for controlling various aspects of the server. In some cases, the server 210 uses microprocessors and protocols to exchange data with other devices / users on one or more networks via the Hypertext Transfer Protocol (HTTP) and the Simple Mail Transfer Protocol (SMTP), although other protocols such as the File Transfer Protocol (FTP) and the Simple Network Management Protocol (SNMP). In some cases, the server 210 is configured to send and receive Hypertext Markup Language (HTML) formatted files (e.g., for displaying web pages). In various embodiments, the server 210 includes a general purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing apparatus.

[0046] The processor unit 215 is intelligent hardware device, e.g., a general purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof. In some cases, the processor unit 215 is configured to operate a memory array using a memory controller. In other cases, a memory controller is integrated into the processor unit 215.

[0047] In some cases, the processor unit 215 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor unit 215 includes a special-purpose component for modem processing, baseband processing, digital signal processing, or transmission processing.

[0048] Examples of the memory unit 220 include random access memory (RAM), read only memory (ROM), or hard disks. Examples of the memory unit include solid state memory and hard disk drives. In some examples, the memory unit 220 is used to store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform various functions described herein. In some cases, the memory unit 220 contains a basic input / output system (BIOS), or the like, which controls basic hardware or software operation such as the interaction with peripheral components or devices. In some cases, a memory controller operates the memory unit. For example, the memory controller can include a row decoder, a column decoder, or both. In some cases, the memory units within the memory unit 220 store information in the form of logical states.

[0049] In some embodiments, the server 210 includes an artificial neural network (ANN). An ANN is a hardware or software component that includes many connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another (just like a physical synapse in the brain). When a node receives a signal, it processes the signal and then transmits the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is computed by a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how the signal is processed and transmitted.

[0050] During a training process, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss function that in some way corresponds to the difference between the current results and target results). The weights of the edges increase or decrease the strength of the signals transmitted between nodes. In some cases, nodes have a threshold below which signals are not transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, signals traverse certain layers multiple times.

[0051] In some embodiments, the server 210 includes a recurrent neural network (RNN). An RNN is a class of ANN in which the connections between nodes form a directed graph along a sequence (i.e., in time). This enables RNNs to model temporal dynamics in sequences, such as predicting what element should come next in a sequence. Thus, RNNs are suitable for tasks that involve ordered sequences, such as text recognition (words are ordered in sentences). The term RNN can include finite impulse recurrent networks (characterized by nodes forming a directed acyclic graph) and infinite impulse recurrent networks (characterized by nodes forming a directed cyclic graph).

[0052] According to some embodiments, the input component 225 receives a structured representation of a search query, where the structured representation includes a set of nodes and at least one edge connecting two of the nodes. The input component 225 receives the search query as a natural language expression. A graph generator (described below) is configured to generate the structured representation based on the natural language expression.

[0053] The input component 225 receives a modified expression for the search query, where the modified expression includes the natural language expression. For example, the modified expression is "show me men wearing jackets." The modified expression creates or modifies a relationship between two objects (i.e., men, jackets).

[0054] According to one embodiment of the disclosure, the input component 225 receives a semantic representation of an image (i.e., a scene graph). In some cases, a parser is used to parse a sentence (i.e., a search query) into a list of objects. For example, one search query is "young boy wearing a black shirt." The list of objects includes "boy" and "shirt." These objects and their associated attributes and relationships form a set of triples, such as (boy, wearing, shirt), (boy, attribute, young), and (shirt, attribute, black). In some embodiments, data quality can be improved and analyzed with a human annotator.

[0055] According to some embodiments, the neural network 230 generates a modified structured representation based on the structured representation and the modified expression. In some cases, the neural network 230 combines structured representation features and natural language expression features. The neural network 230 is described with reference to Figure 3 Examples of corresponding elements are described, or include aspects thereof.

[0056] In some embodiments, the neural network 230 includes an encoder-decoder architecture that uses graph-conditioned transformers and cross-attention networks. In some cases, the neural network 230 learns how to update an existing scene graph given a new user's command (i.e., a natural language command). In one example, the neural network 230 is based on graph-based sparse transformers and cross-attention information fusion.

[0057] According to some embodiments, the graph generator 235 generates a structured representation. In some examples, the graph generator 235 is configured to generate a structured representation based on a natural language expression input. The structured representation includes a set of nodes and at least one edge connecting two of the nodes. The natural language expression is created by a search query typed by a user. For example, the graph generator 235 can receive a reference expression as input and generate a parse tree that is mapped to a hierarchical subject, predicate, object (<S, P, O>) graph structure.

[0058] In some embodiments, the graph generator 235 generates a structured representation using a parsing process that incorporates a "label attention model," where the attention heads of the graph generator network correspond to labels for different syntactic categories to be assigned. In one embodiment, there is one attention head per label. Thus, for example, a label attention head that determines whether a word corresponds to a "noun phrase" category focuses on a different part of the input expression than a label attention head that determines whether a word corresponds to a "verb phrase" category.

[0059] In some embodiments, the graph generator 235 employs object ontologies to understand image objects (e.g., a "hiker" is a kind of "person") and to detect and classify object properties (e.g., "tan" is a kind of color, or "wood" is a kind of material). To improve the accuracy of the mapping between linguistic objects and visual labels, some embodiments employ a base ontology to ensure semantic consistency between the mapping between linguistic objects and visual labels.

[0060] According to some embodiments, the search component 240 performs a search based on the modified structured representation. In some examples, the search component 240 retrieves a set of images corresponding to the modified structured representation based on the search. The search component 240 can be configured to perform a search based on the modified structured representation including the set of nodes and the set of edges.

[0061] Figure 3 An example of a neural network 310 for natural language processing is shown in accordance with various aspects of the disclosure. The example shown includes a source graph 300, a modified query 305, a neural network 310, a list of nodes 340, and a target graph 345.

[0062] The source graph 300 is a reference to Figure 4 The example of corresponding elements described, or include various aspects thereof. In some cases, the source graph 300 is a scene graph (i.e., a semantic representation of an image). The source graph 300 is a semantic form that represents a desired image as a graph of objects with relationships and attributes. In some cases, a parser is used to parse a sentence into a list of objects, and the objects and their associated attributes and relationships form a set of triples. A graph generator is configured to generate a structured representation, where the structured representation includes a set of nodes and at least one edge connecting two of the nodes. For example, the source graph 300 includes four objects (i.e., an image, a copy space, a tea cup, and a table). The objects and their associated attributes and relationships form a set of triples, which in this case are (image, has, copy space), (image, of, tea cup), and (tea cup, property, table).

[0063] According to embodiments of the disclosure, given a scene graph The neural network 310 constructs triples (x, y, z), where x is the source graph, y indicates the modified query 305, and z represents the target graph.

[0064] The modified query 305 is a reference to Figure 4Examples of corresponding elements are described, or include aspects thereof. The modification query 305 represents an expression of a search query entered by a user, where the modification expression includes a natural language expression. The modification query 305 creates or modifies a relationship between two objects. For example, the modification query 305 is "remove the table." In another example, the modification query 305 is "show me a man wearing a jacket." The modification expression creates or modifies a relationship between two objects (i.e., man, jacket).

[0065] In one embodiment of the disclosure, the neural network 310 includes a graph encoder 315, a text encoder 320, a feature fusion network 325, an edge decoder 330, and a node decoder 335. The neural network 310 is described with reference to Figure 2 Examples of corresponding elements are described, or include aspects thereof. The encoder-decoder architecture is used for conditional graph modification. According to embodiments of the disclosure, the graph encoder 315 is built on top of the self-attention architecture of a machine translation model. Additionally, one embodiment provides a graph-conditioned sparse transformer, where relevant information is directly embedded into the self-attention grid.

[0066] According to embodiments of the disclosure, the encoder-decoder architecture includes a graph encoder 315 (i.e., a graph-conditioned, sparse-connected transformer), early and late feature fusion models for combining information from the input text (e.g., the modification query 305) and the source graph 300.

[0067] According to embodiments of the disclosure, the task takes as input a source graph and a modification query, and the neural network 310 includes two encoders to model the source graph and the text information (i.e., the natural language expression) separately. The source graph and the text modification query are encoded into a joint representation, and then the neural network 310 generates the target graph during a two-stage operation. The target nodes are generated via a node-level RNN. Then, the neural network 310 produces the target edges on the nodes with another RNN.

[0068] For the decoders (i.e., the edge decoder 330 and the node decoder 335), the graph modification task is treated as a sequence generation task. Further, to encourage information sharing between the input graph and the modification query, techniques such as late feature fusion through gating and early feature fusion through cross-attention are used.

[0069] According to some embodiments, the graph encoder 315 can be configured to generate structured representation features based on the structured representation. In some examples, the graph encoder 315 includes a sparse connected transformer network. The graph encoder 315 generates structured representation features for the source structured representation. In some cases, a graph convolutional network (GCN) and a graph transformer are used as the graph encoder 315 for graph-to-sequence transduction in semantics-based text generation. The graph encoder 315 is an example of the corresponding element described with reference to Figure 4 described, or comprising various aspects thereof.

[0070] The transformer architecture depends on a fully connected self-attention grid to obtain contextualized representations from a sequence of elements. According to embodiments of the present disclosure, a graph-conditioned, sparse connected transformer is used to encode information from the source graph 300.

[0071] According to some embodiments, the text encoder 320 generates natural language expression features based on the modification expression. In some examples, the text encoder 320 includes a transformer network. For example, the modification expression is “remove the table.” The text encoder 320 generates natural language expression features for the at least one modification expression. The text encoder 320 is an example of the corresponding element described with reference to Figure 4 described, or comprising various aspects thereof.

[0072] According to embodiments of the present disclosure, a standard transformer encoder is used to encode the modification query 305. In another embodiment, the parameters of the graph encoder 315 and the text encoder 320 are shared with each other to facilitate semantic alignment.

[0073] According to some embodiments, the feature fusion network 325 combines the structured representation features and the natural language expression features to produce a set of combined features. In some examples, the feature fusion network 325 includes a stage gating mechanism. In some other examples, the feature fusion network 325 includes a cross-attention network.

[0074] According to some embodiments, the feature fusion network 325 combines the structured representation features and the natural language expression features to produce combined features. The feature fusion network 325 is an example of the corresponding element described with reference to Figure 4 described, or comprising various aspects thereof.

[0075] According to some embodiments of the present disclosure, there are two sources of information, each of which requires a separate encoder. One way is to concatenate the two sources of information to generate a concatenated representation.

[0076] To enhance the model's ability to combine encoder information for better use, a parameterization method with gating mechanisms is applied in this paper. Through gating, the neural network 310 can filter useful information from the graph based on a modified query 305, and vice versa. To achieve deeper interaction between the graph encoder and the text encoder, one embodiment of this disclosure fuses features at an early stage before contextualized node and token representations are learned. This is achieved via early fusion techniques such as cross-attention.

[0077] According to some embodiments, the edge decoder 330 generates an edge set based on a combination of features and a set of nodes, wherein the modified structured representation includes both a set of nodes and a set of edges. In some examples, the edge decoder 330 includes an adjacency matrix decoder. In some other examples, the edge decoder 330 includes a flat edge-level decoder. The edge decoder 330 is a reference... Figure 5 and 6 Examples of the corresponding elements described, or including aspects thereof.

[0078] According to some embodiments, node decoder 335 generates a set of nodes based on combined features. In some examples, node decoder 335 includes an RNN. Gated recurrent unit (GRU) units are used in the RNN node decoder 335. In one example, node decoder 335 includes a basic autoregressive model.

[0079] According to an embodiment, node list 340 is generated from node decoder 335. Node list 340 includes three nodes: image, teacup, and copy space.

[0080] Target Figure 345 is a reference Figure 8 to Figure 11 Examples of the corresponding elements described, or including aspects thereof. Target graph 345 is a scene graph (i.e., a semantic representation of the image). Target graph 345 is a semantic form that represents the desired image as an object graph with relations and attributes. In some cases, a parser is used to parse a sentence into a list of objects, and the objects and their associated attributes and relations form a set of triples. The graph generator is configured to generate a structured representation, where the structured representation includes a set of nodes and at least one edge connecting two nodes in the set. For example, target graph 345 includes three objects (i.e., image, teacup, and copy space). The objects and their associated attributes and relations form a set of triples, in this case (image, have, copy space) and (image, of, teacup). Due to the modification of query 305 (i.e., “remove table”), the previous triple (teacup, attribute, table) is removed.

[0081] Figure 4An example of cross-attention fusion is shown in accordance with various aspects of the present disclosure. The example shown includes a source graph 400, a modified query 405, a graph encoder 410, a text encoder 415, and a feature fusion network 420. The feature fusion network 420 further includes graph attention 425, cross-attention 430, and text attention 435, and joint embedding 440.

[0082] The source graph 400 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 3 The source graph 400 is an example of, or includes various aspects of, the corresponding element described with reference to

[0083] The modified query 405 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 3 The modified query 405 is an example of, or includes various aspects of, the corresponding element described with reference to

[0084] The graph encoder 410 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 3 The graph encoder 410 is an example of, or includes various aspects of, the corresponding element described with reference to

[0085] According to an embodiment of the present disclosure, a graph-conditioned, sparsely connected transformer is used to encode information from a graph. The transformer architecture is dependent on a fully connected self-attention grid to obtain contextualized representations from a sequence of elements. The architecture of the graph encoder 410 with sparsely connected transformer is illustrated herein.

[0086] Graphs are structured data, comprising components, nodes, and edges, as compared to natural language text. To effectively encode graphs, one embodiment of the present disclosure encodes not only information from these constituent components, but also information from their interactions, i.e., node-edge associations and connectivity. Thus, the graph encoder 410 merges information from all edges into the nodes from which these edges originate. The edge-aware node embedding x i This can be obtained from a list of source graph nodes and edges, as follows:

[0087]

[0088] where and T ε are embedding tables for node and edge labels, respectively, and is the set of nodes connected (inbound and outbound) to the ith node in the graph.

[0089] After having edge-aware node embeddings, the graph encoder 410 uses a sparsely connected transformer to learn contextualized embeddings of the whole graph. The sparsely connected transformer can not incorporate positional encodings into the graph input, as nodes are not in a predetermined sequence. Given the edge information from x ε , the connectivity information is enforced by making a node visible to its first-order neighbors only. The transformer's attention grid is denoted as A. If x i，j ∈x ε or is zero, the following equation is defined: A[x i , x j ] = f(x i , x j ), where f denotes a normalized inner product function.

[0090] According to embodiments of the present disclosure, the sparsely connected transformer uses edge labels in the input embeddings and sparse layers in self-attention to provide graph node representations conditioned on graph structure. The node representations in the output of the sparsely connected transformer are denoted by .

[0091] The text encoder 415 is an example of, or includes aspects of, the corresponding element described with reference to Figure 3 . According to embodiments of the present disclosure, a standard transformer encoder is used to encode the modified query y = (y1,.., y |y| ) into To facilitate semantic alignment, the parameters of the graph encoder 410 and the text encoder 415 are shared with each other.

[0092] The feature fusion network 420 is an example of, or includes aspects of, the corresponding element described with reference to Figure 3 . According to some embodiments, there are two sources of information, each of which requires a separate encoder. One way is to concatenate the two sources of information. The combination is denoted as follows:

[0093]

[0094] According to embodiments of the present disclosure, the decoder component is responsible for communicating information between the two encoders through its connection with the two encoders.

[0095] Some embodiments employ a technique called late fusion via gating. To improve the model's ability to combine encoder information for better usage, a parameterized approach with a gating mechanism is applied herein. Through the gating mechanism, the neural network can filter useful information from the graph based on the modified query 405, and vice versa.

[0096] According to embodiments of the disclosure, a special token [CLS] is added to the graph in front of the query sentence. The representation of this special token in the encoder captures the overall understanding, which is represented by and m y , respectively. The neural network uses these overall meaning vectors to filter useful information from the representations of the graph nodes and the modified query tokens as follows:

[0097]

[0098]

[0099]

[0100]

[0101] where MLP is a multi-layer perceptron, denotes element-wise multiplication, and σ is an element-wise sigmoid function used to construct the gating and functions. The updated nodes and tokens are used in the joint encoder representation of Equation 2.

[0102] This gating mechanism is referred to as late fusion because it can not allow information from the graph and the text to interact in their respective low-level encoders. For example, the fusion happens after the contextualized information has been learned.

[0103] Some embodiments also employ a technique referred to as early fusion via cross-attention. To allow deeper interaction between the graph encoder 410 and the text encoder 415, one embodiment fuses features at an early stage before the representations of the contextualized nodes and tokens are learned. This is achieved via the early fusion technique of cross-attention.

[0104] According to embodiments, the parameters of the graph and query encoders are shared with each other to enable encoding of both sources in the same semantic space. For example, the neural network uses the same transformer encoder for both sources. In cross-attention, the feature fusion network 420 concatenates x (from Equation 1) and y before the transformer encoder. Thus, the input to the encoder is [x, y]. In the transformer, the representation of each query token is updated by self-attending to the representations of all query tokens and graph nodes in the previous layer. However, according to the connections of the sparse connected transformer and all query tokens, the representation of each graph node is updated by self-attending to only its graph neighbors. The final representation m is taken from the output of the transformer.

[0105] Figure 5 An example of an adjacency matrix style edge decoder 500 is shown in accordance with various aspects of the present disclosure. The illustrated example includes edge decoder 500 and graph 510. The source graph and text query (i.e., modified query) are encoded into a joint representation, and then the target graph is generated in two stages. The target nodes are generated via a node-level RNN. Then, the model produces target edges on the nodes with another RNN.

[0106] In some cases, a GRU unit is used for the RNN decoder. The node decoder includes a base autoregressive model:

[0107]

[0108]

[0109]

[0110] where z ＜t represents the node generated before time step t, is a Luong style attention, and m is a memory vector that fuses information from the encoder.

[0111] Edge decoder 500 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 3 and Figure 6 In one embodiment, edge decoder 500 includes decoder element 505. Decoder element 505 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 6 In one embodiment, graph 510 includes nodes 515 and edges 520.

[0112] The edge decoder includes an adjacency matrix style edge decoder 500. The rows or columns of the adjacency matrix are indexed by the nodes in the order generated by the node-level decoder. The adjacency matrix style edge decoder 500 includes an autoregressive decoder for each row that emits the label of each edge from an edge vocabulary to other nodes, including a special token [NULL] that shows that an edge does not exist. The lower triangular part of the matrix is important for the adjacency matrix style edge decoder 500 because the node decoder generates nodes in a topologically ordered way. The dashed upper triangular part of the adjacency matrix is dedicated for parallel computation, and they will be discarded.

[0113] An attention decoder including a GRU unit is used to generate edges. Its operation is similar to the node-level decoder shown in Equations 8 and 9. However, for more accurate type of edge generation, the hidden states of the source and target nodes (from the node decoder) are combined as input when updating the hidden state of the edge decoder:

[0114]

[0115] where is the hidden state of the edge encoder for row i and column j, and z i，j-1 is the label of the edge generated previously from node i to j-1.

[0116] Using an adjacency matrix style edge encoder 500 can have several consequences when generating edges. First, the dummy edges in the adjacency matrix cause computational waste. Second, the edges generated by previous rows are not conditioned on when the edges in the next row are generated. However, it is still beneficial to use information about the out edges of previous nodes to improve the accuracy of the generation of the out edges of the next node.

[0117] Figure 6 An example of a flat edge-level encoder is shown in accordance with various aspects of the present disclosure. The edge encoder 600 is an example of, or includes various aspects of, the corresponding elements described with reference to Figure 3 and Figure 5 In one embodiment, the edge encoder 600 includes an encoder element 605. The encoder element 605 is an example of, or includes various aspects of, the corresponding element described with reference to Figure 5

[0118] According to one embodiment of the present disclosure, the flat edge-level encoder 600 is used to flatten the lower triangle of the Figure 5 adjacency matrix shown. The dummy edges are removed, and the rows of the lower triangle matrix are concatenated to form a sequence of node pairs for which edges are generated between the respective nodes. This edge generation method uses information about all previously generated edges when a new edge is generated.

[0119] Figure 7 An example of a scenario graph is shown in accordance with various aspects of the present disclosure. The example shown includes a first node 700, a second node 705, a third node 710, a fourth node 715, a first edge 720, a second edge 725, and a third edge 730.

[0120] ​A scene graph can be referred to as a semantic representation of an image. A scene graph is a semantic form that represents a desired image as a graph of objects with relationships and attributes. Semantic representations are adapted for use in retrieval systems. In some cases, a parser is used to parse a sentence into a list of objects, such as “boy” and “shirt.” These objects and their associated attributes and relationships form a set of triples, such as (boy, wearing, shirt), (boy, attribute, young), and (shirt, attribute, black). For example, a graph generator can receive a reference expression as input and generate a parse tree that is mapped to a hierarchical subject, predicate, object (<S, P, O>) graph structure. Based on the initial scene graph and a new query from a user, a neural network generates a modified scene graph based on the existing scene graph and the modified query.

[0121] In some embodiments, the graph generator generates a structured representation using a parsing process that includes a “label attention model,” where the attention heads of the graph generator network correspond to labels for different syntactic categories to be assigned. In one embodiment, there is one attention head per label. Thus, for example, a label attention head that determines whether a word corresponds to a “noun phrase” category focuses on a different part of the input expression than a label attention head that determines whether a word corresponds to a “verb phrase” category.

[0122] In some embodiments, the graph generator employs an object ontology to understand image objects (e.g., a “hiker” is a kind of “person”) and to detect and classify object attributes (e.g., “tan” is a kind of color, or “wood” is a kind of material). To improve the accuracy of the mapping between linguistic objects and visual labels, some embodiments employ a base ontology to ensure that the semantic mapping between linguistic objects and visual labels is consistent.

[0123] Figure 8 An example of an insertion process is shown in accordance with various aspects of the disclosure. The example shown includes a source graph 800, a modification query 805, and a target graph 810. The source graph 800 is an example of the corresponding element described with reference to Figure 3 and Figure 4 described, or includes various aspects thereof. The modification query 805 is an example of the corresponding element described with reference to Figure 3 and Figure 4 described, or includes various aspects thereof. The target graph 810 is an example of the corresponding element described with reference to Figure 3 described, or includes various aspects thereof.

[0124] One embodiment of the present disclosure includes steps to create data for a multi-operation graph modification (MGM) task. It is assumed that MGM depends on at least one operation on a source graph. The steps include performing operations on modified graphs using four actions (terminate, insert, delete, substitute) paired with a heuristic algorithm. The model samples an action and that action is performed on the last modified graph until terminate is sampled or available nodes are exhausted. In some cases, large graphs can support more modifications while small graphs have less freedom. Additionally, in one embodiment, modified nodes are not changed more than once. Thus, as the edit sequence gets longer, the probability of terminate can increase while the probability of other actions can decrease. The heuristic algorithm is referred to as Algorithm 1 below, which provides a dataset with different edit sequence lengths.

[0125]

[0126]

[0127] According to embodiments, a scene graph includes a semantic representation of an image. In some cases, a parser is used to parse a sentence (i.e., a search query) into a list of objects (e.g., "boy," "shirt"). These objects and their associated attributes and relationships form a set of triples. According to embodiments of the present disclosure, given a scene graph A triple (x, y, z) is constructed, where x is a source graph, y is a modification query, and z represents a target graph. One embodiment of the present disclosure uniformly selects and applies an action a from a set of possible graph modification operations A = {insert, delete, substitute}. Based on a user's search entry or modification query, an operation (e.g., delete, insert, and substitute) is applied to the source graph.

[0128] According to embodiments, an action includes an insert command. Insert can be seen as an inverse of a delete operation. One embodiment produces a source graph x via a "delete" operation on where the target graph z is set to Similar to the delete operator, an insert query (i.e., an example of a modification query) y is generated by workers from a crowd-sourcing application (e.g., TurkPrime) or templates. In most cases, a crowd-sourcing application is used to obtain training data to train a neural network to modify an existing scene graph based on a modification query. For example, a source graph 800 includes a node (i.e., man). A modification query 805 includes a natural language expression "show me a man wearing a jacket." A target graph 810 has two nodes connected by an edge. The two nodes include a man and a jacket corresponding to the insert command from the modification query 805.

[0129] Figure 9An example of a deletion process is shown in accordance with various aspects of the present disclosure. The example shown includes a source graph 900, a modification query 905, and a target graph 910.

[0130] Source graph 900 is an example of corresponding elements described with reference to Figure 3 and Figure 4 or includes various aspects thereof. Modification query 905 is an example of corresponding elements described with reference to Figure 3 and Figure 4 or includes various aspects thereof. Target graph 910 is an example of corresponding elements described with reference to Figure 3 or includes various aspects thereof.

[0131] According to embodiments of the present disclosure, the action includes a deletion operation. A node is randomly selected from (representing source graph x), and the system removes the node and its associated edges. The remaining nodes and edges are represented as target graph z. Modification query y is either from a randomly selected deletion template or generated by a worker from a crowd-sourcing application. For example, source graph 900 has three nodes (i.e., objects) and two edges (i.e., properties). The nodes are “young,” “man,” and “hat.” The edges include “wearing.” The objects and their associated properties and relationships form a set of triples (man, property, young) and (man, wearing, hat). Source graph 900 is a scene graph, which is a semantic representation of the search query “young man wearing a hat.” Modification query 905 includes the natural language expression “take away his hat.” Target graph 910 has two nodes and one edge. These objects and their associated properties and relationships form a triple (man, property, young). In some cases, the template is based on a dataset (e.g., Edit Me dataset) and a crowd-sourcing application (e.g., Turkish Robots). The crowd-sourcing application is used to obtain training data to train a neural network to modify an existing scene graph (e.g., source graph 900) based on modification query 905.

[0132] Figure 10 An example of an object replacement process is shown in accordance with various aspects of the present disclosure. According to embodiments of the present disclosure, the action includes a replacement operation (i.e., object replacement, property replacement). The example shown includes a source graph 1000, a modification query 1005, and a target graph 1010. Source graph 1000 is an example of corresponding elements described with reference to Figure 3 and Figure 4 or includes various aspects thereof. Modification query 1005 is an example of corresponding elements described with reference to Figure 3 and Figure 4 or includes various aspects thereof. Target graph 1010 is an example of corresponding elements described with reference to Figure 3 or includes various aspects thereof.

[0133] According to embodiments, randomly selected nodes from the source graph 1000 are replaced by semantically similar nodes to obtain the target graph. To locate the new nodes, the AllenNLP toolkit can be used to obtain a list of candidate words based on their semantic similarity scores with the old nodes. For example, the source graph 1000 has two nodes (i.e., objects) and one edge (i.e., attribute). The nodes are “small” and “clock.” These objects and their associated attributes and relationships form a triple (small, attribute, hour hand). The source graph 1000 is a scene graph, which is a semantic representation of the search query (i.e., hour hand). The modified query 1005 includes the natural language expression “show more hour hands.” The target graph 1010 has two nodes and one edge. These objects and their associated attributes and relationships form a triple (small, attribute, hour hand). The node representing “clock” (i.e., singular) is replaced by a new node representing “clocks” (i.e., plural).

[0134] Figure 11 An example of an attribute replacement process is shown in accordance with various aspects of the present disclosure. According to embodiments of the present disclosure, the actions include replacement operations (i.e., object replacement, attribute replacement). The example shown includes a source graph 1100, a modified query 1105, and a target graph 1110. The source graph 1100 is an example of the corresponding element described with reference to Figure 3 and Figure 4 or includes various aspects thereof. The modified query 1105 is an example of the corresponding element described with reference to Figure 3 and Figure 4 or includes various aspects thereof. The target graph 1110 is an example of the corresponding element described with reference to Figure 3 or includes various aspects thereof.

[0135] According to embodiments, randomly selected nodes from the source graph 1100 are replaced by semantically similar nodes to obtain the target graph. To locate the new nodes, the AllenNLP toolkit can be used to obtain a list of candidate words based on their semantic similarity scores with the old nodes. For example, the source graph 1100 has two nodes (i.e., objects) and one edge (i.e., attribute). The nodes are “white” and “surfboard.” These objects and their associated attributes and relationships form a triple (surfboard, attribute, white). The source graph 1100 is a scene graph, which is a semantic representation of the search query (white surfboard). The modified query 1105 includes the natural language expression “I want a pink surfboard; it cannot be white.” The target graph 1110 has two nodes and one edge. These objects and their associated attributes and relationships form a triple (pink, attribute, surfboard). The node representing “white” is replaced by a new node representing “pink.”

[0136] Figure 12Examples of processes for natural language processing in accordance with various aspects of the present disclosure are shown. In some examples, these operations are performed by a system including a processor executing a set of codes to control the functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed in accordance with methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are comprised of various sub-steps, or performed in conjunction with other operations.

[0137] In operation 1200, the system receives a structured representation of a search query, where the structured representation includes a set of nodes and at least one edge connecting two nodes in the nodes. The set of nodes represents objects, and the edge represents an associated attribute and relationship. In one example, the search query is "young boy wearing a black shirt," which can be represented by a structured representation. The structured representation includes four nodes and three edges. The nodes represent four objects (i.e., boy, shirt, young, black). The edges include attributes such as "wearing" (i.e., conveying a relationship between two objects). These objects and their associated attributes and relationships form a set of triples, such as (boy, wearing, shirt), (boy, attribute, young), and (shirt, attribute, black). In some cases, the operations of this step involve referring to Figure 2 the input components described, or can be performed by them.

[0138] In operation 1205, the system receives a modification expression for the search query, where the modification expression includes a natural language expression. According to an embodiment, the modification expression is used to modify the source graph according to an operation or action. One embodiment uniformly selects and applies an action a from the set of all possible graph modification operations A = {INSERT, DELETE, SUBSTITUTE}. The operation (e.g., delete, insert, and substitute) is applied to the source graph. In some cases, the operations of this step involve referring to Figure 2 the input components described, or can be performed by them.

[0139] According to an embodiment, the modification query is generated from a randomly selected deletion template or by a worker from a crowd-sourcing application. For example, the source graph is a scene graph, which is a semantic representation of the search query "young man wearing a hat." The modification query includes the natural language expression "take away his hat."

[0140] In operation 1210, the system generates a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features. In some cases, the operations of this step involve referring to Figure 2 and 3 the neural network described, or can be performed by them.

[0141] According to embodiments, the neural network includes an encoder-decoder architecture with a graph encoder, a text encoder, a feature fusion network, an edge decoder, and a node decoder. The graph encoder includes a sparse-connected transformer conditioned on the graph. The graph encoder is used to generate structured representation features based on the structured representation. The text encoder is used to generate natural language expression features based on the modified expression. In some embodiments, an early feature fusion model and a late feature fusion model are used to combine information from the input text (i.e., the modification query) and the source graph. One embodiment formulates this task as a conditional generation task. Given the source graph and the modification query, the neural network can generate the target graph by maximizing the conditional probability.

[0142] In operation 1215, the system performs a search based on the modified structured representation. This search can be performed by an image search engine from an image database. The user utilizes the image search engine to trace the origin of the target image, or to find additional images that are similar or related to the target image. The user is requested to enter a search query in a search box and / or use filters for a more in-depth search. In some cases, the operations of this step involve referring to Figure 2 the search component described, or can be performed by it.

[0143] Figure 13 An example of a process for combining graph features and text features according to various aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor executing a set of codes to control functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to methods and processes described according to various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps, or are performed in conjunction with other operations.

[0144] In operation 1300, the system generates structured representation features based on the structured representation using a graph encoder. In some cases, the operations of this step involve referring to Figure 3 and Figure 4 the graph encoder described, or can be performed by it.

[0145] According to embodiments of the present disclosure, the encoder-decoder framework takes as input the source graph and the modification query, so the network framework includes a graph encoder and a text encoder to model graph and text information, respectively. The neural network includes five components, a graph encoder, a text encoder, a feature fusion network, an edge decoder, and a node decoder.

[0146] A graph-conditioned, sparsely connected transformer is used to encode information from the source graph. The source graph and the modification query are encoded into a joint representation. Compared to natural language text, a graph is structured data, including components, nodes, and edges. To effectively encode a graph, one embodiment of the present disclosure encodes not only information from these constituent components, but also information from their interactions, i.e., node-edge associations and connectivity. Thus, the graph encoder merges information from all edges into the nodes from which these edges originate. The edge-aware node embeddings are obtained from a list of source graph nodes and edges, as illustrated in Equation 1, where embedding tables for node and edge labels are generated separately. A set of nodes connected (inbound and outbound) to the ith node in the graph is created. After obtaining the edge-aware node embeddings, the graph encoder uses a sparsely connected transformer to learn contextualized embeddings of the entire graph. The sparsely connected transformer can not merge positional encodings into the graph input, as nodes are not in a predetermined sequence. Given the edge information from x ε Connectivity information is enforced by making a node visible to its first-order neighbors only. The attention grid of the transformer is computed.

[0147] According to embodiments of the present disclosure, the sparsely connected transformer provides graph node representations conditioned on graph structure using edge labels in input embeddings and sparse layers in self-attention. The node representations in the output of the sparsely connected transformer are represented by .

[0148] In operation 1305, the system generates natural language expression features based on the modification expression using a text encoder. In some cases, the operations of this step involve or can be performed by the text encoder described with reference to Figure 3 and Figure 4 .

[0149] According to embodiments of the present disclosure, a standard transformer encoder is used to encode the modification query. To facilitate semantic alignment, the parameters of the graph encoder and the text encoder are shared with each other.

[0150] In operation 1310, the system combines the structured representation features and the natural language expression features using a feature fusion network to produce a set of combined features. According to some embodiments, there are two sources of information, and each of the two sources requires an encoder. One way is to concatenate the two sources of information. The combined representation is illustrated in Equation 2 above. In some cases, the operations of this step involve or can be performed by the feature fusion network described with reference to Figure 3 and Figure 4 .

[0151] To improve the model’s ability to combine encoder information for better usage, some embodiments of the present disclosure introduce a parameterized approach with a gating mechanism. With the gating mechanism, the model can filter useful information from the graph based on the modification query and vice versa. According to embodiments of the present disclosure, a special token [CLS] is added to the graph in front of the query sentence. The representation of this token in the encoder captures the overall understanding. The model uses these overall meaning vectors to filter useful information from the graph node and modification query token representations. An element-wise sigmoid function is used to construct the gate. The updated nodes and tokens are used in the joint encoder representation. This gating mechanism is referred to as late fusion because it can not allow information from the graph and text to interact in their respective low-level encoders. For example, the fusion happens after the contextualized information has been learned.

[0152] According to embodiments, the feature fusion network uses early fusion via cross- attention. To allow deeper interaction between the graph encoder and the text encoder, one embodiment fuses features at an early stage before the contextualized node and token representations are learned. This is achieved via a cross-attention, early fusion technique.

[0153] According to embodiments, the parameters of the graph and query encoders are shared to enable encoding of both sources in the same semantic space. For example, the model uses the same transformer encoder for both sources. In the cross-attention, the model concatenates x (from Equation 1) and y before the transformer encoder. Thus, the input to the encoder is [x, y]. In the transformer, the representation of each query token is updated by self-attending to the representations of all query tokens and graph nodes in the previous layer. However, according to the connections of the sparse connected transformer and all query tokens, the representation of each graph node is updated by self-attending to only its graph neighbors. The final representation m is taken from the output of the transformer.

[0154] Figure 14 An example of a process for generating a modified structured representation based on combined features is shown in accordance with various aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control the functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed in accordance with the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps, or performed in conjunction with other operations.

[0155] In operation 1400, the system produces a set of combined features. According to some embodiments, a feature fusion network is used to combine structured representation features (from the graph encoder) and natural language expression features (from the text encoder) to produce the set of combined features. In some cases, the operations of this step involve reference toFigure 3 and Figure 4 The described feature fusion network, or can be executed by it.

[0156] In operation 1405, the system generates a set of nodes based on the combined features using a node decoder. In some cases, the operations of this step involve referring to the described node decoder, or can be executed by it. Figure 3 The described node decoder, or can be executed by it.

[0157] According to an embodiment, the node decoder comprises an RNN. In some cases, a GRU unit is used for the RNN decoder. The node decoder comprises a base autoregressive model.

[0158] In operation 1410, the system generates a set of edges based on the combined features and the set of nodes using an edge decoder, wherein the modified structured representation comprises the set of nodes and the set of edges. In some cases, the operations of this step involve referring to the described edge decoder, or can be executed by it. Figure 3 Figure 5 and Figure 6 The described edge decoder, or can be executed by it.

[0159] According to an embodiment, the edge decoder applies an adjacency pattern generation method. The rows or columns of the adjacency matrix are indexed by the order in which the nodes are generated by the node decoder. For each row, the autoregressive decoder emits a label for each edge from an edge vocabulary to the other nodes, including a special token [NULL] that shows that the edge does not exist. The lower triangular part of the matrix is important because the node decoder is assumed to generate nodes in a topologically ordered way. The dashed upper triangular part of the adjacency matrix is dedicated to parallel computation, and they will be discarded.

[0160] According to an embodiment, an attention decoder using a GRU unit is used to generate edges. The attention (edge) decoder operates in a similar way as the node decoder. To generate typed edges more accurately, the hidden states of the source and target nodes (from the node decoder) are merged as input when updating the hidden state of the edge decoder.

[0161] There are several consequences of using this edge generation method. First, the virtual edges in the adjacency matrix cause computational waste. Second, the edges generated by a previous row are not conditioned on when the edges in the next row are generated. But it can be beneficial to use information about the outgoing edges of the previous nodes to improve the generation accuracy of the outgoing edges of the next node. Therefore, one embodiment of the disclosure flattens the lower triangle of the adjacency matrix. The virtual edges are removed, and the rows of the lower triangular matrix are concatenated to form a sequence of node pairs when the edges are generated between the corresponding nodes. This flat edge-level decoder uses information about all the previously generated edges when a new edge is generated.

[0162] ​In operation 1415, the system generates a modified structured representation based on the structured representation and the modified expression. In some cases, the operations of this step involve referring to Figure 2 and Figure 3 the neural network described or that can be executed by it.

[0163] Figure 15 An example of a process for training a neural network for natural language processing is shown in accordance with various aspects of the present disclosure. In some examples, these operations are performed by a system including a processor executing a set of codes to control the functional elements of a device. Additionally or alternatively, certain processes are performed using special purpose hardware. Generally, these operations are performed in accordance with the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps, or are performed in conjunction with other operations.

[0164] One embodiment formulates this task as a conditional generation task. Formally, given a source graph and a modification query y, the target graph can be produced by maximizing the conditional probability Since graphs consist of lists of typed nodes and edges, the conditional probability is also decomposed as:

[0165]

[0166] where and z ε represent the nodes and edges of graph respectively.

[0167] Given a training dataset of input-output pairs represented by the model is trained by maximizing the conditional log-likelihood l CLL = l Node + l Edge where,

[0168]

[0169]

[0170] During learning and decoding, nodes are arranged according to a topological order that exists for all directed graphs in the user-generated dataset and the synthetic dataset.

[0171] According to embodiments of the present disclosure, the graph encoder includes three stacked sparse transformers, each layer having four heads. The embedding size is 256, and the dimension of the inner layer of the feedforward network is 512. Both the node-level and edge-level decoders are one layer of GRU-RNN with a hidden size of 256, and the embedding size is also 256. 30 epochs and 300 epochs are trained for synthetic data and user-generated data, respectively, with a batch size of 256. The model is evaluated on the dev set at each epoch, and the checkpoint with the best graph accuracy is selected for inference. For example, the experiment is run on a single Nvidia Tesla V100. A gated recurrent unit (GRU) is a gating mechanism in recurrent neural networks. GRU is similar to long short-term memory (LSTM) with a forget gate, but has fewer parameters than LSTM because it lacks an output gate.

[0172] Graph processing unit (GPU) time (ms / step) is measured under different settings of the training phase. The cross-attention model of the present disclosure is more efficient in GPU computation time than other models. The number of parameters under different settings is also measured and recorded. The number of parameters for each model is recorded.

[0173] The performance of the model validation / dev set and the best baseline are recorded. Based on the results, there is no significant difference between the performance trends of the dev set and the test set.

[0174] In operation 1500, the system identifies training data including a set of annotated training examples, where each of the annotated training examples includes a source structured representation, a target structured representation, and at least one modification expression.

[0175] In operation 1505, the system generates structured representation features for the source structured representation using a graph encoder. According to embodiments of the present disclosure, graph-conditioned, sparsely connected transformers are used to encode information from graphs. The graph encoder merges information from all edges to the nodes from which these edges originate. After having edge-aware node embeddings, the graph encoder uses sparsely connected transformers to learn contextualized embeddings of entire graphs. In some cases, the operations of this step involve referring to, or can be performed by, the graph encoder described in Figure 3 and Figure 4 .

[0176] In operation 1510, the system generates natural language expression features for the at least one modification expression using a text encoder. According to embodiments of the present disclosure, the text encoder includes standard transformer encoders that are used to encode the modification expressions. In some cases, the operations of this step involve referring to, or can be performed by, the text encoder described in Figure 3 and Figure 4 .

[0177] In operation 1515, the system uses a feature fusion network to combine the structured representation features and the natural language expression features to produce combined features. According to some embodiments, the two information sources (i.e., the graph and the textual query) are combined by concatenation. Early and late feature fusion models are used to combine information from the graph and the input text. In some cases, the operations of this step involve referring to Figure 3 and Figure 4 the feature fusion networks described, or can be performed by them.

[0178] To improve the model’s ability to combine the encoders’ information for better usage, a parameterized approach with a gating mechanism is used herein. With the gating mechanism, the model can filter useful information from the graph based on the modified query, and vice versa. The gating mechanism is referred to as late fusion because it can not allow the information from the graph and the text to interact in their respective low-level encoders. For example, the fusion happens after the contextualized information has been learned.

[0179] According to embodiments, to allow deeper interaction between the graph and the text encoders, cross-attention (an early fusion technique) is used to fuse the features at an early stage before the contextualized node and token representations are learned.

[0180] In operation 1502, the system generates a set of nodes based on the combined features using a node decoder. According to embodiments of the present disclosure, the node decoder comprises an RNN. Additionally, a GRU unit is used for the RNN decoder. The node decoder comprises a base autoregressive model. In some cases, the operations of this step involve referring to Figure 3 the node decoder described, or can be performed by it.

[0181] In operation 1525, the system generates a set of edges based on the combined features and the set of nodes using an edge decoder. According to embodiments of the present disclosure, the edge decoder comprises an adjacency matrix decoder. Alternatively, the edge decoder comprises a flat edge-level decoder. In some cases, the operations of this step involve referring to Figure 3 , Figure 5 and Figure 6 the edge decoder described, or can be performed by it.

[0182] In operation 1530, the system compares the set of nodes and the set of edges to a target structured representation. According to embodiments of the present disclosure, training data is input into a neural network. The neural network generates a predicted structured representation (i.e., the set of nodes and the set of edges) using an encoder-decoder architecture. The target structured representation is a ground truth value for comparison. For each training iteration, a loss function is used to compute or estimate the loss (or error) between the predicted structured representation and the target structured representation.

[0183] In operation 1535, the system updates the neural network based on the comparison. After each training iteration, the parameters of the neural network are updated (e.g., using gradient descent). As the neural network model is trained, the parameters (weights and biases) are adjusted in a manner that minimizes the loss function.

[0184] Evaluation

[0185] While there are multiple scene graph annotation datasets for images, alignment between graphs and text is not available. Further, image live scene graphs (e.g., visual genome dataset) also contain many non-significant objects and relationships, while search queries focus more on the main objects and their connections. According to embodiments of the present disclosure, benchmark datasets are created due to the lack of large-scale and high-quality datasets. Three datasets are created to evaluate the model through a data creation process. The data creation process starts from caption datasets, such as MSCOCO and Google Conceptual Captions (GCC). To construct the scene graphs, the process uses a scene graph parser to parse the MSCOCO description data and a random subset of GCC captions. The parser is built on top of a dependency syntax parser. In some cases, the process includes generating modified scene graphs and modified queries based on these scene graphs, and a method of leveraging human annotation to improve and analyze data quality. The first two datasets include MSCOCO and GCC, while the third dataset is collected using a crowd-sourcing application (e.g., Turkian Robo).

[0186] The first two datasets add annotations over the captions of MSCOCO and GCC. In some examples, the above parser is used to create 200k scene graphs from MSCOCO and 420k scene graphs from GCC data. Comparing the two datasets, the graphs from MSCOCO are relatively simple, while the GCC graphs are more complex. According to search logs, image search queries are short, so MSCOCO graphs represent a closer match to actual search queries, while GCC graphs pose a greater challenge to the model. Based on the experiments conducted, the model improves the performance of synthetic data and user-generated data by up to 8.5% in F1 score. In statistical analysis of binary classification, the F1 score (also F score or F-measure) is a measure of the accuracy of a test. It is calculated through the precision and recall of the test, where precision is the number of correctly identified positive results divided by the number of all positive results (including those not correctly identified), and recall is the number of correctly identified positive results divided by the number of all samples that should be identified as positive.

[0187] In addition to using templates, one embodiment of the present disclosure uses a crowd-sourcing application to obtain more diverse and natural modified queries (e.g., Turkish robots). The crowd-sourcing process shows workers examples including the source graph, the target graph, and three acceptable modified queries. Then, the workers are asked to fill in their own descriptions for the unannotated instances. The template-based version of the dataset is referred to as “synthetic,” while the user-generated content is referred to as “user-generated.”

[0188] Through preliminary experiments, there were some challenges during the data collection process. First, understanding the graphs depends on knowledge related to NLP, so not all Turkish robot workers can provide good modified queries. Second, some graphs can have disconnected components in the data due to deletion and parser errors. Third, there are many graphs that are too complex to represent search queries, as most search queries are relatively short (i.e., limited to one or two objects). To mitigate the impact of the challenges, one embodiment of the present disclosure manually filtered the data by removing graphs with disconnected components, low-quality instances, or overly long descriptions (i.e., more than 5 nodes). As an example, the final dataset contains 32,000 instances.

[0189] To test the quality of the crowd-sourced dataset, a user study with 15 test persons was performed. The 15 test persons were not aware of the nature of the work and the way the dataset was collected. The test persons were provided with a random set of instances, each instance being a triple of source graph, modified query, and target graph. The test persons provided a score indicating the quality of each instance based on criteria, including how well the modified query reflects the target graph and how natural the query and graph are.

[0190] The score distribution from 200 randomly selected instances was recorded. Most of the 3 or 4 (i.e., quality score) was due to the modified query or graph being unnatural. The test persons tended to give a 1 (i.e., quality score) for instances with semantic errors. That is, the modified query did not match the change. Overall, the test persons rated the data with an average score of 3.76.

[0191] Based on the search logs, more than 95% of the queries were limited to one or two nodes, so it is unlikely that more than one editing operation would be applied. Therefore, the instances in the modified MSCOCO and GCC were constructed with one editing operation. However, in some cases, there can be long search descriptions, which lead to the possibility of longer editing operation sequences. According to an embodiment of the present disclosure, a multi-operation version of the dataset (i.e., MGM task from the GCC data) was created.

[0192] The graph size distribution of source and target graphs is almost the same in the set. As the length of the text description increases, the source graph becomes more complex accordingly. The percentage of query length from the search log is analyzed. In most cases, the length of the search query can be less than five tokens. Therefore, in practice, it is unlikely to encounter large graphs (i.e., more than three nodes) and long modification queries. The distribution of the number of nodes and edges on synthetic data and user-generated data is recorded.

[0193] Obtaining annotations from users is expensive, especially for complex tasks such as the graph modification task of the present disclosure. Therefore, one embodiment explores the possibility of augmenting user-generated data with synthetic data to train the model. However, data augmentation with synthetic data can lead to different distributions. This is illustrated when testing the model trained with synthetic data on user-generated data. For example, graph generation accuracy drops to around 20%, and adding more synthetic data can not help. To effectively mix the data distribution, one embodiment of the present disclosure up-samples the user-generated data and mixes it with synthetic data at a 1:1 ratio in each mini-batch.

[0194] Up-sampled data augmentation is compared with transfer learning (another method of learning from synthetic data and user-generated data). According to the embodiment, the model is pre-trained with synthetic data, and then the model is fine-tuned on user-generated data.

[0195] Graph accuracy (%) results under different data settings are recorded. 30000 means adding 30000 synthetic instances. It shows that, compared with transfer learning, using up-sampled data augmentation is an effective method of utilizing both data sources. In addition, as the size of synthetic data increases, the described scheme also improves performance to a certain point of stability. For example, after injecting 9000 instances (data ratio 3:1), the performance reaches stability. Both up-sampling and pre-training produce better models compared to using only synthetic data or user-generated data. The graph accuracy of the model trained exclusively on user-generated data is 60.90% (best result).

[0196] In some cases, templates are used for synthetic data. For example, a simplified template is used for synthetic data, where each operation has ten templates.

[0197] Five baselines are considered for comparison. In the“copy-source” baseline, the system copies the source graph to the target graph. It is based on the observation that users specifically modify a small portion of the source graph. In the“Text2Text” baseline, the baseline model flattens the graph and reconstructs natural sentences in a similar way to the modification query. In the“modified-graph RNN” baseline, the baseline model flattens the graph using a node ordering based on a breadth-first search (BFS) technique (i.e., the topological braid is jumped out by the order of the nodes that appear in the original query), and uses an RNN as an encoder and a decoder. In the other two baselines,“graph transformer” and“deep convolutional graph network” (DCGCN), the baseline model encodes the source graph using a graph transformer and a deep convolutional graph network.

[0198] The models from the present disclosure are configured in various ways, and the results of different configurations of the models are recorded. According to an embodiment of the present disclosure, the fully connected transformer uses dense connections for the graph encoder. In contrast, the sparse transformer uses the connectivity structure of the source graph in self-attention. The information from the graph and query encoders can be combined through concatenation, through gated late fusion, or through cross-attention early fusion. The adjacency matrix style of edge decoding can be replaced by flat edge generation.

[0199] One embodiment of the present disclosure uses two automatic metrics for evaluation. First, the precision / recall / F1 score of the generated nodes and edges are computed. Second, the evaluation metric uses strict match accuracy, which is defined as the generated graph that is identical to the target graph for a correct prediction.

[0200] One embodiment of the present disclosure divides the synthetic MSCOCO data into 196K / 2K / 2K for training / development / testing, and divides the GCC data into 400K / 7K / 7K for training / development / testing. In some cases, the crowd-sourced user-generated data is randomly divided into 30K / 1K / 1K for training / development / testing.

[0201] Results of the models and baselines on synthetic and user-generated datasets are reported. Based on the experimental results, various configurations of the model outperform the baselines significantly. For example, DCGCN and graph transformer are strong baselines, providing SOTA performance in tasks such as AMR-to-text generation and syntax-based neural machine translation. The large number of edge types in the tasks weakens their capabilities. Moreover, the assumption that flat edge generation is preferred over adjacency matrix style edge generation is validated in this paper. Furthermore, the bidirectional communication between the graph and query encoders through the gating mechanism consistently outperforms simple concatenation in edge-level and node-level generation. Finally, cross-attention (early fusion technique) results in a large increase in all metrics. In some cases, it is much more difficult to generate graphs for crowd-sourced data than synthetic data due to the diversity of semantics and expressions introduced by the annotators. Therefore, all models suffer a performance drop.

[0202] The performance trends of different configurations of the model are almost identical on user-generated and synthetic data. As the graphs become more complex, the model performs worse in reasoning the relationships between nodes for GCC data, which results in a drop in edge F1 score and graph accuracy.

[0203] To evaluate multiple operational scenarios, two datasets were created with an average number of operations of 1.44 and 2.01, respectively. For each dataset, the baselines and methods were trained on the full training set. The test set was grouped into four bins according to the number of operations. The performance of all models decreased as the number of operations increased. The performance of the model provided by embodiments of the present disclosure was significantly better than the baselines.

[0204] According to embodiments of the present disclosure, the best performance comes from the model that includes cross-attention, flat edge decoder, and sparse transformer. In most cases, this configuration outperforms the baselines. Cross-attention is used for relationship modification and handling longer descriptions because early feature fusion reduces noise in graph generation and better preserves fine-grained details than the baseline models.

[0205] Embodiments of the present disclosure relate to conditional graph modification, in which a model is used to modify a source graph based on a modification command. One configuration of the model based on sparse transformer conditioned on the graph and cross-attention information fusion outperforms the baselines adapted from machine translation and graph generation.

[0206] Therefore, the present disclosure includes at least the following embodiments.

[0207] A method for natural language processing is described. Embodiments of the method are configured to receive a structured representation of a search query, where the structured representation includes a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, where the modification expression includes a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

[0208] An apparatus for natural language processing is described. The apparatus includes a processor, memory in electronic communication with the processor, and instructions stored in the memory. The instructions are operable to cause the processor to receive a structured representation of a search query, where the structured representation includes a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, where the modification expression includes a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

[0209] A non-transitory computer readable medium storing code for natural language processing is described. In some examples, the code includes instructions executable by a processor to receive a structured representation of a search query, where the structured representation includes a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, where the modification expression includes a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

[0210] Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include receiving the search query as an initial natural language expression. Some examples further include generating the structured representation using a graph generator. Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include generating structured representation features based on the structured representation using a graph encoder.

[0211] Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include generating natural language expression features based on the modification expression using a text encoder. Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include combining the structured representation features and the natural language expression features using a feature fusion network to produce a set of combined features.

[0212] Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include generating a set of nodes based on the combined features using a node decoder. Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include generating a set of edges based on the combined features and the set of nodes using an edge decoder, wherein the modified structured representation includes the set of nodes and the set of edges.

[0213] Some examples of the method, apparatus, and non-transitory computer-readable medium described above further include retrieving a plurality of images corresponding to the modified structured representation based on the search.

[0214] An apparatus for natural language processing is described. Embodiments of the apparatus include a graph encoder configured to generate structured representation features based on a structured representation, a text encoder configured to generate natural language expression features based on a modified expression, a feature fusion network configured to produce combined features based on the structured representation features and the natural language expression features, a node decoder configured to generate a set of nodes based on the combined features, and an edge decoder configured to generate a set of edges based on the combined features and the set of nodes.

[0215] A method of providing an apparatus for natural language processing is described. The method includes a graph encoder configured to generate structured representation features based on a structured representation, a text encoder configured to generate natural language expression features based on a modified expression, a feature fusion network configured to produce combined features based on the structured representation features and the natural language expression features, a node decoder configured to generate a set of nodes based on the combined features, and an edge decoder configured to generate a set of edges based on the combined features and the set of nodes.

[0216] Some examples of the apparatus and method described above further include a graph generator configured to generate the structured representation based on the natural language expression. Some examples of the apparatus and method described above further include a search component configured to perform a search based on the modified structured representation including the set of nodes and the set of edges.

[0217] In some examples, the graph encoder includes a sparse-connected transformer network. The text encoder includes a transformer network. The feature fusion network includes a stage gating mechanism. The feature fusion network includes a cross-attention network. The node decoder includes a recurrent neural network (RNN). In some examples, the edge decoder includes an adjacency matrix decoder. The edge decoder includes a flat edge-level decoder.

[0218] A method for natural language processing is described. Embodiments of the method are configured to: identify training data comprising a plurality of annotated training examples, where each of the annotated training examples comprises a source structured representation, a target structured representation, and at least one modification expression; generate, using a graph encoder, structured representation features for the source structured representation; generate, using a text encoder, natural language expression features for the at least one modification expression; combine, using a feature fusion network, the structured representation features and the natural language expression features to produce combined features; generate, using a node decoder, a set of nodes based on the combined features; generate, using an edge decoder, a set of edges based on the combined features and the set of nodes; compare the set of nodes and the set of edges to the target structured representation; and update a neural network based on the comparison.

[0219] An apparatus for natural language processing is described. The apparatus comprises a processor, memory in electronic communication with the processor, and instructions stored in the memory. The instructions are operable to cause the processor to: identify training data comprising a plurality of annotated training examples, where each of the annotated training examples comprises a source structured representation, a target structured representation, and at least one modification expression; generate, using a graph encoder, structured representation features for the source structured representation; generate, using a text encoder, natural language expression features for the at least one modification expression; combine, using a feature fusion network, the structured representation features and the natural language expression features to produce combined features; generate, using a node decoder, a set of nodes based on the combined features; generate, using an edge decoder, a set of edges based on the combined features and the set of nodes; compare the set of nodes and the set of edges to the target structured representation; and update a neural network based on the comparison.

[0220] A non-transitory computer-readable medium storing code for natural language processing is described. In some examples, the code includes instructions executable by a processor to: identify training data comprising a plurality of annotated training examples, where each of the annotated training examples comprises a source structured representation, a target structured representation, and at least one modification expression; generate, using a graph encoder, structured representation features for the source structured representation; generate, using a text encoder, natural language expression features for the at least one modification expression; combine, using a feature fusion network, the structured representation features and the natural language expression features to produce combined features; generate, using a node decoder, a set of nodes based on the combined features; generate, using an edge decoder, a set of edges based on the combined features and the set of nodes; compare the set of nodes and the set of edges to the target structured representation; and update a neural network based on the comparison.

[0221] In some examples, the neural network is updated using an end-to-end training technique, in which parameters of the graph encoder, the text encoder, the feature fusion network, the node decoder, and the edge decoder are updated during each training iteration.

[0222] The description and drawings described herein represent examples and do not represent all implementations within the scope of the claims. For example, the operations and steps can be rearranged, combined, or otherwise modified. Also, structure and devices can be represented in a form that is simplified for clarity and ease of description, and the illustrated examples can not necessarily discuss all of the components or options that are available to implement the described concepts. Similar components or features can have the same reference label, and similar or equivalent features can have the same reference label.

[0223] Some modifications of the disclosure can be apparent to those of ordinary skill in the art, and the principles defined herein can be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not to be limited to the examples and designs described herein but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

[0224] The described methods can be implemented or performed by a device including a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor can also be implemented as a combination of a

[0225] A computer readable medium includes a non-transitory computer readable medium and a communication medium and includes any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium can be any available medium that can be accessed by a computer. For example, non-transitory computer readable media can include random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0226] Also, connection components can be appropriately referred to as computer-readable media. For example, if the code or data is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies are included in the definition of medium.

[0227] In this disclosure and the following claims, the word "or" means an inclusive "or" such that, for example, a phrase A or B means A or B or AB or AT or B. Also, the phrase "based on" is not used to express conditional closeness. For example, a step described as "based on condition A" can be based on condition A and condition B. In other words, the phrase "based on" should be interpreted as "based, at least in part, on." Also, the word "a" or "an" means "at least one."

Claims

1. A method for natural language processing, comprising: Receive a query that includes an initial natural language expression, the initial natural language expression comprising multiple elements; Generate a structured representation of the query, wherein the structured representation includes multiple nodes corresponding to the multiple elements and at least one edge connecting two of the multiple nodes; A graph encoder is used to generate multiple node embeddings corresponding to the plurality of nodes based on the at least one edge; Receive a modification command for the query, wherein the modification command includes a natural language expression describing a change to an element among the plurality of elements in the query, and wherein the modification command specifies a sign change to an operator of the structured representation of the query; A text encoder is used to generate multiple text embeddings representing the modification command; Using a feature fusion network, a combined embedding representing the query and the modification command is generated based on the multiple node embeddings and the multiple text embeddings; as well as A modified structured representation is generated by decoding the combined embedding, wherein the modified structured representation includes updated nodes and updated edges representing the query with the changes indicated by the modification command.

2. The method according to claim 1, wherein: The multiple node embeddings and the multiple text embeddings are located in a common embedding space.

3. The method according to claim 1, wherein: The structured representation includes a scene graph, and the modified structured representation includes a modified scene graph.

4. The method according to claim 1, further comprising: The search is performed based on the modified structured representation.

5. The method of claim 1, wherein generating the modified structured representation further comprises: The transformer network is applied to multiple combined embeddings, including the multiple node embeddings and the multiple text embeddings.

6. The method of claim 5, wherein generating the modified structured representation further comprises: The multiple node embeddings are modified based on the modification command to obtain updated multiple node embeddings; as well as The multiple text embeddings are modified based on the modification command to obtain updated multiple text embeddings, wherein the multiple combined embeddings include the updated multiple node embeddings and the updated multiple text embeddings.

7. The method according to claim 1, further comprising: The search is performed based on the modified structured representation. as well as Based on the search, multiple images corresponding to the modified structured representation are retrieved.

8. An apparatus for natural language processing, comprising: A graph generator is configured to generate a structured representation of a query, the query including an initial natural language expression comprising multiple elements, wherein the structured representation includes multiple nodes corresponding to the multiple elements and at least one edge connecting two of the multiple nodes; A graph encoder is configured to generate multiple node embeddings corresponding to the plurality of nodes based on the at least one edge; A text encoder is configured to generate multiple text embeddings representing modification commands for the query, wherein the modification commands include natural language expressions describing changes to elements among the multiple elements in the query, and wherein the modification commands specify sign changes to operators of the structured representation of the query. The feature fusion network is configured to generate a combined embedding representing the query and the modification command based on the plurality of node embeddings and the plurality of text embeddings, and to generate a modified structured representation by decoding the combined embeddings; A node decoder is configured to generate updated nodes based on multiple combined embeddings that combine the multiple node embeddings and the multiple text embeddings; as well as An edge decoder is configured to generate updated edges based on the multiple combined embeddings and the updated multiple nodes.

9. The apparatus according to claim 8, further comprising: The search component is configured to perform a search based on the modified structured representation, which includes the updated plurality of nodes and the updated plurality of edges.

10. The apparatus according to claim 8, wherein: The graphic encoder includes a sparsely connected transformer network.

11. The apparatus according to claim 8, wherein: The text encoder includes a transformer network.

12. The apparatus according to claim 8, wherein: The feature fusion network includes a stage gating mechanism.

13. The apparatus according to claim 8, wherein: The feature fusion network includes a cross-attention network.

14. The apparatus according to claim 8, wherein: The node decoder includes a recurrent neural network (RNN).

15. The apparatus according to claim 8, wherein: The edge decoder includes an adjacency matrix decoder.

16. The apparatus according to claim 8, wherein: The edge decoder includes a flat edge-level decoder.

17. A method for training a neural network, the method comprising: The identifier includes training data comprising multiple annotated training examples, each of which includes a source structured representation of a query, a target structured representation, and at least one modification command for the query. The query includes an initial natural language expression comprising multiple elements. The source structured representation includes multiple nodes corresponding to the multiple elements and at least one edge connecting two of the multiple nodes. The at least one modification command includes a natural language expression describing a change to an element among the multiple elements in the query. The at least one modification command specifies a sign change for an operator in the source structured representation of the query. A graph encoder is used to generate multiple node embeddings corresponding to the plurality of nodes based on at least one edge of the source structured representation; A text encoder is used to generate multiple text embeddings representing the at least one modification command; Using a feature fusion network, a combined embedding representing the query and the at least one modification command is generated based on the plurality of node embeddings and the plurality of text embeddings; A modified structured representation is generated by decoding the combined embedding, wherein the modified structured representation includes an updated plurality of nodes and an updated plurality of edges representing the query having the changes indicated by the at least one modification command; The updated nodes and edges are compared with the target structured representation; as well as The neural network is updated based on the comparison.

18. The method of claim 17, wherein: The neural network is updated using an end-to-end training technique, wherein the parameters of the graph encoder, the text encoder, the feature fusion network, the node decoder, and the edge decoder are updated during each training iteration.

Images