Generate 3D digital content from natural language requests

Abstract

Embodiments of the present disclosure relate to generating three-dimensional digital content from natural language requests. Methods, systems, and non-transitory computer-readable storage media for generating a three-dimensional scene based on a natural language phrase are disclosed. For example, the disclosed system may analyze a natural language phrase to determine dependencies involving entities and commands in the natural language phrase. The disclosed system may then use the dependencies to generate an entity-command representation of the natural language phrase. Additionally, the disclosed system may generate a semantic scene graph for the natural language phrase from the entity-command representation to indicate contextual relationships of the entities and commands. Furthermore, the disclosed system generates the requested three-dimensional scene by using at least one of a plurality of available three-dimensional scenes, the plurality of available three-dimensional scenes being identified using a semantic scene graph of the natural language phrase.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to generating three-dimensional digital content from natural language requests. Background Art

[0002] Recent advances in computational design, virtual reality / augmented reality (VR / AR), and robotics have placed increasing demands on 3D digital content. For example, many computer-aided design or VR / AR systems utilize 3D models. High-quality 3D models can significantly improve the aesthetic design, realism, and immersion in a 3D environment.

[0003] Conventional systems typically utilize software applications that allow content creators to use a variety of tools to create three-dimensional digital content. Conventional software applications provide a high degree of customization and precision, allowing content creators to generate anything from basic three-dimensional shapes to highly detailed, complex three-dimensional scenes with many three-dimensional objects. Although conventional systems provide a lot of control to content creators, such applications have a large number of tools to perform a large number of operations. Therefore, using conventional systems to create three-dimensional scenes usually requires a lot of training to learn to use content creation tools, which may provide a high barrier to entry for new users. Accordingly, creating three-dimensional digital content is limited by the expertise and capabilities of the content creator.

[0004] Furthermore, even when the user is skilled and knowledgeable, conventional systems typically require navigation between various user interfaces and / or various control menus in order to generate a three-dimensional scene. Therefore, even for skilled and knowledgeable users, creating a 3D scene using conventional systems is both time-consuming and inefficient.

[0005] These and other shortcomings exist with respect to conventional systems for creating three-dimensional digital content. Summary of the invention

[0006] One or more embodiments utilize systems, methods, and non-transient computer-readable storage media to provide benefits and / or solve one or more of the aforementioned or other problems in the art, which systems, methods, and non-transient computer-readable storage media intelligently generate three-dimensional digital content based on natural language requests. More particularly, the disclosed system includes a framework for generating a language-based representation of an existing 3D scene, which encodes geometric information and semantic scene information about the 3D scene. When a natural language command to generate or modify a 3D scene is received, the disclosed system also generates a representation of a natural language phrase, which encodes geometric information and semantic information and a relationship with the command. Then, the disclosed system maps the natural language representation to one or more language-based representations of a 3D scene or subscene. Then, the disclosed system uses the identified 3D scene or subscene to generate or modify the 3D scene based on the natural language command.

[0007] For example, in one or more embodiments, the disclosed system analyzes a natural language phrase requesting the generation of a three-dimensional scene to determine dependencies involving one or more entities and / or commands in the natural language phrase. Specifically, the disclosed system utilizes dependencies involving (multiple) entities and (multiple) commands to generate an entity-command representation of the natural language phrase, which is annotated with attributes and relationships of (multiple) entities and (multiple) commands. In addition, the disclosed system uses at least one three-dimensional scene from a database of previously generated three-dimensional scenes to generate a three-dimensional scene based on the entity-command representation. Specifically, the disclosed system can select a three-dimensional scene from a database by associating the entity-command representation with a semantic scene graph of the three-dimensional scene. Therefore, the disclosed system can generate a three-dimensional scene from a natural language request efficiently, flexibly and accurately by determining a representation of the request that allows comparison with a representation of an existing three-dimensional scene.

[0008] Additional features and advantages of one or more embodiments of the present disclosure will be set forth in the description which follows and, in part, will be apparent from the description, or may be learned by practicing such example embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Various embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] Figure 1 illustrates an example environment in which a three-dimensional 3D modeling system may operate according to one or more implementations;

[0011] Figure 2A schematic diagram illustrating a process of generating a three-dimensional scene from a natural language phrase according to one or more implementations;

[0012] FIG. 3A to FIG. 3C A schematic diagram illustrating a process of parsing a natural language phrase according to one or more implementations is illustrated;

[0013] FIG. 4A to FIG. 4C A schematic diagram illustrating different entity-command representations for generating a three-dimensional scene according to one or more implementations is illustrated;

[0014] Figure 5 A schematic diagram illustrating selecting a previously generated three-dimensional scene according to one or more implementations;

[0015] FIG. 6A to FIG. 6C An embodiment of generating a three-dimensional scene from a series of natural language phrases according to one or more implementations is illustrated;

[0016] Figure 7 The diagram illustrates a method according to one or more implementations. Figure 1 A schematic diagram of a three-dimensional 3D modeling system is shown;

[0017] Figure 8 A flow chart illustrating a series of actions for synthesizing a three-dimensional scene using natural language according to one or more implementations; and

[0018] Fig. 9 A block diagram of an exemplary computing device is illustrated in accordance with one or more embodiments. DETAILED DESCRIPTION

[0019] One or more embodiments of the present disclosure include a natural language-based three-dimensional modeling system (also referred to as a "natural language-based 3D system" or simply a "3D modeling system") that generates a three-dimensional scene based on a natural language request. For example, the 3D modeling system uses natural language processing to analyze a natural language phrase and determine dependencies involving components of the natural language phrase. In particular, the 3D modeling system can determine the relationship between various nouns and / or verbs in the phrase, and generate an entity-command representation of the phrase based on the determined relationship. In addition, the 3D modeling system also generates semantic scene graphs for existing 3D scenes and sub-scenes. These semantic scene graphs encode geometric information and semantic scene information about the 3D scene. Then, the 3D modeling system maps the entity-command representation of the phrase to the semantic scene graph of the existing three-dimensional scene. The 3D modeling system uses the identified three-dimensional scene to generate a three-dimensional scene to satisfy the request in the natural language phrase. By generating a semantic scene graph based on the entity-command representation of the natural language request, the 3D modeling system can accelerate and simplify the process of generating a three-dimensional scene by quickly finding an existing three-dimensional scene corresponding to the content of the request.

[0020] As mentioned, the 3D modeling system can use natural language processing to analyze a natural language phrase that includes a request to generate a three-dimensional scene. In one or more embodiments, the 3D modeling system uses natural language processing to transform the natural language phrase into a representation that the 3D modeling system can use to compare with similar representations of existing three-dimensional scenes. Specifically, the 3D modeling system labels one or more entities and one or more commands in the natural language phrase, and then determines dependencies involving the entities and commands. For example, the 3D modeling system can determine dependencies by parsing the natural language phrase and creating a dependency tree that assigns parent tokens and annotation tags to each token in the phrase.

[0021] In one or more embodiments, the 3D modeling system generates an entity-command representation of a natural language phrase using the determined dependencies. Specifically, the 3D modeling system converts a dependency representation (e.g., a dependency tree) of phrase tokens into an entity-command representation that provides a detailed graphical representation of the components of the natural language phrase and their relationships. For illustration, the entity-command representation may include a list of entities annotated with corresponding attributes and relationships and a list of command verbs that operate on the entities.

[0022] In one or more additional embodiments, the 3D modeling system determines canonical entity-command representations for multiple natural language phrases that include the same concept request. For example, the 3D modeling system may determine that there are multiple different ways to express a request to build the same three-dimensional scene. Because the phrases include different parsed structures, the 3D modeling system creates a different entity-command representation for each form. The 3D modeling system may then select the entity-command representation of the descriptive form as the canonical entity-command representation, so that future requests that include the same concept request utilize the canonical entity-command representation.

[0023] After generating the entity-command representation of the natural language phrase, the 3D modeling system generates a semantic scene graph of the natural language phrase. Specifically, the 3D modeling system converts the entity-command representation into a semantic scene graph for generating a three-dimensional scene. For example, the 3D modeling system determines object categories, entity counts, qualifiers, and relationships from the entity-command representation. For illustration, the 3D modeling system includes object nodes, relationship nodes, and edge nodes based on the determined information to represent the relative positioning and relationships of objects within the requested three-dimensional scene.

[0024] Using the 3D scene graph constructed for the natural language phrase, the 3D modeling system then generates the three-dimensional scene. In one or more embodiments, the 3D modeling system generates the three-dimensional scene using at least one three-dimensional scene from the database of available three-dimensional scenes from the database. For example, the 3D modeling system may compare the semantic scene graph of the natural language phrase with the semantic scene graph of the available three-dimensional scenes to select at least one scene in the scene that best matches the semantic scene graph of the natural language phrase.

[0025] As mentioned, 3D modeling systems provide a number of advantages over conventional systems. For example, 3D modeling systems increase the flexibility of the three-dimensional scene generation process. In particular, 3D modeling systems increase flexibility by allowing users to use natural language phrases to generate three-dimensional scenes. By using natural language requests, 3D modeling systems improve the usability of three-dimensional modeling programs for inexperienced users by improving the input methods available to users when generating three-dimensional scenes. In contrast, conventional systems typically require users to have a deep understanding of modeling tools and how to use them.

[0026] In addition, the 3D modeling system improves the speed and efficiency of the three-dimensional scene generation process. Specifically, by constructing object representations (e.g., semantic scene graphs) of natural language phrases for generating three-dimensional scenes, the 3D modeling system reduces the time required to generate the three-dimensional scene. For example, in contrast to conventional systems that require users to use multiple three-dimensional modeling tools to generate each object within a scene, the 3D modeling system described herein can interpret natural language requests for three-dimensional scenes and then quickly generate three-dimensional scenes without using any additional tools. Therefore, the 3D modeling system improves conventional user interfaces and modeling systems by using natural language-based generation of 3D scenes to improve the efficiency of computing devices.

[0027] In addition to the foregoing, the 3D modeling system also improves consistency and reduces complexity when determining requests and generating three-dimensional scenes based on natural language phrases. In particular, the 3D modeling system determines canonical entity-command representations that can be used to represent various different forms of requests for concepts (i.e., scene editing / construction concepts). For example, the 3D modeling system can determine that an entity-command representation of a particular form (e.g., a description form) is a canonical entity-command representation that represents different forms of requests for generating / editing three-dimensional scenes.

[0028] Additionally, the 3D modeling system improves the accuracy of the three-dimensional scene generation process. In particular, by creating a semantic scene graph of natural language phrases based on entity command representations of the phrases, the 3D modeling system generates a representation of the phrases that the system can use to identify similar existing three-dimensional scenes. The semantic scene graph provides the 3D modeling system with a representation that the 3D modeling system can easily compare to available three-dimensional scenes in a database of three-dimensional scenes, so that the 3D modeling system can accurately determine the relationship between objects within the three-dimensional scene based on the request.

[0029] As explained by the foregoing discussion, the present disclosure utilizes various terms to describe the features and advantages of the 3D modeling system. Additional details are now provided regarding the meaning of the terms. For example, as used herein, the term "natural language phrase" refers to text or speech that includes ordinary language employed by a user. Specifically, a natural language phrase may include text or speech that does not have a special syntax or formal structure that is specifically configured to interact with a computing device. For example, a natural language phrase may include a conversational request to generate or modify a scene. For illustration, a user may use natural language when speaking a request or otherwise entering a request into a computing device.

[0030] As used herein, the term "three-dimensional scene" refers to a digital representation of one or more objects in a three-dimensional environment. For example, a three-dimensional scene may include any number of digital objects located at one or more positions within the three-dimensional environment according to a set of coordinate axes (e.g., x-axis, y-axis, z-axis). For illustration, displaying objects of a three-dimensional scene on a display device of a computing device involves reconstructing the objects on the display device using the mathematical positioning of the objects on the coordinate axes. Reconstructing the three-dimensional scene takes into account the size and shape of each object based on the coordinates used by the computing device to reconstruct the objects for multiple vertices. Mathematical positioning also allows the computing device to present the relative positioning of objects relative to each other.

[0031] As used herein, the terms "entity" and "command" refer to the components of speech or text of a phrase analyzed by a natural language processor. Specifically, the 3D modeling system uses a natural language processor to identify tokens in a sentence by identifying a string with an assigned / identified meaning. Then, the 3D modeling system marks the token as an entity or command. As used herein, the term "entity" refers to an identified noun in a sentence that satisfies a set of conditions, including: determining that a noun does not have a compound dependency relationship with another noun, that a noun is not an abstract concept, and that a noun does not represent a spatial region. In particular, an entity may include a basic noun associated with one or more attributes, one or more counts, one or more relationships, or one or more qualifiers based on the structure of the sentence (e.g., annotated with one or more attributes, counts, relationships, or qualifiers). As used herein, the term "command" refers to an identified command verb that operates on one or more entities. Specifically, a command may include a basic verb associated with one or more attributes or targets. Examples of entities and commands are described in more detail below. In one or more embodiments, the 3D modeling system marks all verbs in the basic form as commands.

[0032] As used herein, the term "entity-command representation" refers to a logical representation of entities and commands and their relationships in a natural language phrase. Specifically, the entity-command representation includes a list of entities annotated with attributes and relationships and a list of command verbs that operate on the entities. As described in more detail below, the entity-command representation can also be illustrated as a block diagram indicating attributes and relationships (e.g., dependencies) between different components of a natural language phrase.

[0033] As used herein, the term "semantic scene graph" refers to a graphical representation of a three-dimensional scene that includes geometric information for objects in the scene and semantic information for the scene. In particular, in addition to object relationships (e.g., pairs and groups) indicating the relative positioning of object instances, the semantic scene graph may also include object instances and object-level attributes. Additionally, the semantic scene graph includes two node types (object nodes and relationship nodes) with edges connecting object nodes and relationship nodes. Therefore, the semantic scene graph can represent object relationships and compositions in both natural language phrases and three-dimensional scene data.

[0034] Additional details about the 3D modeling system will now be provided with respect to an illustrative diagram depicting an exemplary implementation. For purposes of illustration, Figure 1An embodiment of an environment 100 is included in which a natural language 3D modeling system 102 can operate. In particular, the environment 100 includes a client device 103 associated with a user, (multiple) server devices 104, and a content database 106, which communicate via a network 108. In addition, as shown, the client device 103 includes a client application 110. In addition, the server device (s) 104 includes a content creation system 112, which includes the 3D modeling system 102.

[0035] The content database 106 is a database containing multiple existing / available three-dimensional scenes or sub-scenes. Specifically, the content database 106 includes three-dimensional scenes with different objects and / or object arrangements. As described below, the content database 106 provides various three-dimensional scenes, and the 3D modeling system 102 can use these three-dimensional scenes to generate three-dimensional scenes from natural language phrases. In one or more embodiments, the content database 106 includes content from multiple content creators. Additionally, the content database 106 can be associated with a content creation system 112 (e.g., including content from users of the content creation system 112) and / or a third-party system (e.g., including content from users of the third-party system).

[0036] In one or more embodiments, the 3D modeling system 102 also accesses the content database 106 to obtain semantic scene graph information for multiple available three-dimensional scenes. In particular, the 3D modeling system 102 can obtain semantic scene graphs for multiple available three-dimensional scenes in the content database 106. The 3D modeling system 102 can obtain the semantic scene graph by analyzing the three-dimensional scene and then generating the semantic scene graph. Alternatively, the 3D modeling system 102 obtains a pre-generated semantic scene graph from the content database 106 or a third-party system.

[0037] As mentioned, the server device(s) 104 may host or implement a content creation system 112. The content creation system 112 may manage content creation for multiple users. The content creation system 112 may include or be associated with multiple content creation applications that allow multiple users to create and otherwise interact with various types of digital content. The content creation system may also provide content storage and sharing capabilities (e.g., via the content database 106) so that content creators can view the content of other content creators.

[0038] In addition, in one or more embodiments, the 3D modeling system 102 communicates with the client device 103 via the network 108 to receive one or more requests to generate a three-dimensional scene. For example, the client device 103 may include a client application 110 that enables communication with the 3D modeling system 102 based on natural language. For example, the client application 110 may include a web browser or a local computing device. In an example implementation, a user of the client device 103 may use the client application 110 to input a natural language phrase including a request to generate a three-dimensional scene. For illustration, the client application 110 may allow the user to input a natural language phrase using keyboard input or voice input. For voice input, the client application 110 or the 3D modeling system 102 may convert the voice input into text input (e.g., using speech to text analysis).

[0039] The 3D modeling system 102 may receive a natural language phrase from the client device 103, and then analyze the natural language phrase to determine a request. Specifically, the 3D modeling system 102 parses the natural language phrase to identify components of the natural language phrase and corresponding dependencies. The 3D modeling system 102 uses the components and dependencies to generate an entity-command representation of the natural language phrase. Based on the entity-command representation, the 3D modeling system 102 generates a semantic scene graph for the natural language phrase to indicate objects and their relative positions within a three-dimensional scene.

[0040] The 3D modeling system 102 generates a three-dimensional scene based on a semantic scene graph of a natural language phrase. In one or more embodiments, the 3D modeling system 102 compares the semantic scene graph of the natural language phrase with the semantic scene graphs of a plurality of available three-dimensional scenes in the content database 106 to select one or more three-dimensional scenes. The 3D modeling system 102 can then use the selected scenes to generate a three-dimensional scene that satisfies the natural language request received from the client device 103. The 3D modeling system 102 can also allow the user to modify the generated three-dimensional scene using one or more additional natural language phrases. After generating the three-dimensional scene, the 3D modeling system 102 can store the three-dimensional scene in the content database 106 for use in generating future three-dimensional scenes.

[0041] Although Figure 1The illustrated environment is depicted as having various components, but the environment 100 may have any number of additional or alternative components (e.g., any number of server devices, client devices, content databases, or other components in communication with the 3D modeling system 102). For example, the 3D modeling system 102 may allow any number of users associated with any number of client devices to generate a three-dimensional scene. Furthermore, the 3D modeling system 102 may communicate with any number of content databases to access existing three-dimensional scenes for generating the three-dimensional scene based on a natural language request. Additionally, one or more components or entities in the environment 100 may implement the operations of the 3D modeling system 102 described herein. For example, alternatively, the 3D modeling system 102 may be implemented in whole (or in part) on the client device 103 or on a separate client device.

[0042] As mentioned above, the 3D modeling system 102 may parse natural language phrases to generate semantic scene graphs for the phrases for use in generating a three-dimensional scene. Figure 2 A schematic diagram of a process for generating a three-dimensional scene from a natural language phrase is illustrated. In particular, the process includes a series of actions 200 in which the 3D modeling system 102 processes the natural language phrase to convert the natural language phrase into a form that the 3D modeling system 102 can use to compare with an already existing three-dimensional scene.

[0043] In one or more embodiments, the series of actions 200 includes a first action 202 of identifying a natural language phrase. Specifically, the 3D modeling system 102 identifies the natural language phrase based on a user input of a user. The user input may include text input (e.g., via a keyboard or touch screen) or voice input (e.g., via a microphone). The 3D modeling system 102 may also determine that the user input includes multiple natural language phrases, and then process each phrase separately. Alternatively, the 3D modeling system 102 may determine that multiple natural language phrases are combined to form a single request, and process the multiple natural language phrases together.

[0044] After the natural language phrase is identified, the series of acts 200 includes an act 204 of parsing the phrase to create a dependency tree. In one or more embodiments, the 3D modeling system uses natural language processing to analyze the natural language phrase. For example, the 3D modeling system uses natural language processing to parse the natural language phrase to determine different phonetic components of the phrase and relationships and dependencies between components within the phrase. One way a natural language processor can determine these dependencies is by creating a dependency tree.

[0045] A dependency tree (or dependency-based parse tree) is a tree comprising multiple nodes with dependency grammar. For example, in one or more embodiments, the nodes in the dependency tree are all terminals, so that there is no distinction between terminal categories (formal grammatical elements) and non-terminal categories (syntactic variables), as in the case of a constituency-based parse tree. In addition, in one or more embodiments, the dependency tree has fewer nodes than a constituency-based parse tree because the dependency tree lacks phrase categories (e.g., sentences, noun phrases, and verb phrases). In contrast, a dependency tree includes a structure that indicates the dependency relationships of components of a natural language phrase by setting a verb as the structural center of the dependency tree (e.g., the first parent node) and then building a tree based on noun dependencies (whether direct or indirect) on the verb node.

[0046] In one or more embodiments, the 3D modeling system converts a natural language phrase into a dependency tree by first identifying the tokens of the natural language phrase. As previously mentioned, a token comprises a string of characters (e.g., a word) with an identified meaning. The 3D modeling system then assigns an annotation tag to each token in the natural language phrase. This includes: identifying nouns, verbs, adjectives, etc. in the natural language phrase, and then determining the dependencies of each component in the phrase with respect to other components in the phrase. Figure 3A Examples of assigning parent tokens and annotation tags to each token in a phrase are described in more detail. According to at least some embodiments, a 3D modeling system uses an established framework for determining lexical dependencies (eg, a general dependency framework).

[0047] The series of actions 200 includes an action 206 of generating an entity-command representation of a natural language phrase. In one or more embodiments, the 3D modeling system converts a low-level dependency representation of a natural language phrase in a dependency tree into an entity-command representation by labeling different components from the dependency tree. Specifically, generating the entity-command representation involves an action 208a of labeling entities and an action 208b of labeling commands. According to one or more embodiments, the 3D modeling system 102 labels entities and commands in a natural language phrase by analyzing the natural language phrase to identify a specific type of speech component, and then provides a corresponding label to the identified type of speech component. For example, as for Figure 3A and Figure 3B Described in more detail, the 3D modeling system marks all nouns in a sentence as entities unless one of the following conditions is met: 1) the noun has a compound dependency with another noun (e.g., "computer" in "computer desk"); 2) the noun is an abstract concept (e.g., "addition", "attraction"); or 3) the noun represents a special area (e.g., "right", "side"). In addition, the 3D modeling system marks all verbs in the base form as commands.

[0048] In addition, when labeling entities, the 3D modeling system 102 dereferences pronouns. In particular, the 3D modeling system 102 replaces pronouns in the natural language phrase with corresponding nouns in the phrase to identify an association relationship involving the base noun, rather than creating a new entity in the entity-command representation for the pronoun itself. For example, the 3D modeling system can use coreference information from a natural language processing framework (e.g., the Stanford CoreNLP framework). If there is ambiguity for a non-pronoun scenario that can reference two or more different objects in a natural language phrase (or in more than one natural language phrase), the 3D modeling system 102 does not use such coreference information. In this example, the 3D modeling system 102 resolves the ambiguity when aligning the entity with an object in the requested three-dimensional scene.

[0049] Generating entity-command representations may involve an act 210 of performing pattern matching to assign additional attributes to tagged entities and commands. In one or more embodiments, the 3D modeling system 102 uses pattern matching on a dependency tree to assign attributes that are not tagged as entities or commands. Specifically, the 3D modeling system 102 may analyze natural language phrases to identify nouns and verbs that describe or modify a base noun or base verb. To illustrate, the 3D modeling system 102 may identify spatial nouns, count adjectives, group nouns, and adjective verbs, and then augment the base noun / verb with annotations indicating the association. Figure 3B and the accompanying description indicate the results of pattern matching to assign non-entity / command attributes to a simple natural language phrase, thereby producing an entity-command representation of the natural language phrase.

[0050] Optionally, in one or more embodiments, the series of actions 200 includes an action 212 of creating a canonical entity-command representation. As briefly mentioned previously, a request to generate a specific three-dimensional scene can often take different forms in a natural language phrase. In particular, the flexibility of language often allows users to express requests in a variety of different ways using different words, combinations of words, and the order of words to illustrate essentially the same thing. Since there can be different ways to form the same request, the 3D modeling system 102 can generate a plurality of different entity-command representations corresponding to the different ways.

[0051] To increase consistency and reduce complexity when determining requests and generating three-dimensional scenes based on natural language phrases, the 3D modeling system 102 can determine a single entity-command representation that can be used to represent all different forms of requests for a concept (i.e., scene editing / construction concepts). FIG. 4A to FIG. 4CDescribed in more detail, the 3D modeling system 102 may determine that an entity-command representation of a particular form (e.g., a description form) is a canonical entity-command representation representing a request of a different form for generating / editing a three-dimensional scene. Accordingly, the 3D modeling system 102 may attempt to transform the entity-command representation of a natural language phrase into a canonical entity-command representation via a set of pattern matching rules.

[0052] In one or more embodiments, if a command cannot be applied as a graphical transformation in the entity-command representation, the 3D modeling system keeps the command unchanged and uses a dedicated function to execute the command with respect to the scene. For example, for commands such as "delete" or "rotate", the 3D modeling system may use a dedicated function to execute the "delete" or "rotate" function on the corresponding object within the three-dimensional scene (e.g., after constructing the three-dimensional scene without the executed command). Alternatively, the 3D modeling system may notify the user that the command was not understood and that the user is required to enter a new natural language phrase.

[0053] After determining the entity-command representation for the natural language phrase, the series of actions 200 includes an action 214 of converting the entity-command representation into a semantic scene graph. In particular, the 3D modeling system 102 converts the entity-command representation into a form that the 3D modeling system 102 can use to generate a three-dimensional scene. As previously mentioned, the semantic scene graph serves as a bridge between user language commands and scene modeling operations, which directly modify the three-dimensional scene. Accordingly, converting the entity-command representation into a semantic scene graph allows the 3D modeling system to easily construct a three-dimensional scene based on the identified positional relationships of objects in the semantic scene graph.

[0054] In one or more embodiments, converting the entity-command representation includes determining a layout of objects within the three-dimensional scene. Specifically, the 3D modeling system 102 uses the entity-command representation to classify the base nouns according to object categories, determine entity counts, entity qualifiers, and relationships. Figure 3C As described in more detail, the 3D modeling system 102 then constructs a semantic scene graph to include object nodes, relationship nodes, and edges indicating pairwise and grouped locations of objects within the three-dimensional scene. The semantic scene graph of the natural language phrase allows the 3D modeling system 102 to determine a spatial representation of the requested three-dimensional scene of the object for generating the natural language phrase.

[0055] Finally, the series of actions 200 includes an action 216 of generating a three-dimensional scene based on the semantic scene graph. As briefly described previously, the 3D modeling system 102 can access a database including a plurality of available three-dimensional scenes (e.g., previously generated three-dimensional scenes) that include various objects and object layouts. The 3D modeling system 102 compares the semantic scene graph of the natural language phrase with the semantic scene graph of the available three-dimensional scenes, and then selects one or more three-dimensional scenes based on the degree of match of the semantic scene graphs. The 3D modeling system then uses the selected (multiple) three-dimensional scenes (or portions of the three-dimensional scenes) to generate a three-dimensional scene based on the user request.

[0056] For illustration, the 3D modeling system 102 obtains a semantic scene graph for an available three-dimensional scene. Then, the 3D modeling system 102 compares the semantic scene graph of the natural language phrase with the obtained semantic scene graph to determine the similarity of the structure of the graph. For example, if two semantic scene graphs match completely, the semantic scene graph has the same node and edge label structure. Similarly, a local match indicates that some nodes (or edges) in the semantic scene graph are the same, but other nodes (or edges) are different. Then, the 3D modeling system 102 can rank the semantic scene graph based on the similarity of the node structure to determine one or more available three-dimensional scenes that best match the requested three-dimensional scene.

[0057] To generate the requested three-dimensional scene, the 3D modeling system 102 may use one or more available three-dimensional scenes that best match the requested three-dimensional scene. For example, the 3D modeling system may extract one or more objects from the available three-dimensional scene and then place the one or more objects into the three-dimensional environment at the user's client device. For illustration, the 3D modeling system 102 may place the object(s) at coordinates that reflect the relative positions of the objects on a set of coordinate axes. For example, as described with respect to FIG. 6A to FIG. 6C Described in more detail, if the user is modifying an existing scene, the 3D modeling system 102 can insert the object into the existing scene. Alternatively, the 3D modeling system 102 can enable the user client device to load a file containing an existing three-dimensional scene directly into a new project workspace.

[0058] As about Figure 2 And in the corresponding FIG. 3A to FIG. 6C As described in , the 3D modeling system 102 can perform operations for processing natural language phrases to create entity-command representations of natural language phrases (and corresponding semantic scene graphs). These operations allow the 3D modeling system to receive natural language requests to generate new three-dimensional scenes and / or to modify existing three-dimensional scenes using existing scenes. Figure 2 And the following about FIG. 3A to FIG. 6C The illustrated and described actions and operations provide a corresponding structure for example steps for using natural language processing to generate entity-command representations involving dependencies of one or more entities and one or more commands in a natural language phrase.

[0059] Figure 3A The dependency tree 300 generated by the 3D modeling system 102 for a natural language phrase is shown. Specifically, Figure 3A A natural language phrase that reads "Put some books on the desk" is illustrated. A dependency tree illustrates the dependencies (including dependency / relationship types) involving different phrase components in a natural language phrase. The dependency tree provides a consistent vocabulary structure that a 3D modeling system can use to identify different words in a phrase.

[0060] In one or more embodiments, the 3D modeling system 102 identifies a plurality of tokens in a natural language phrase corresponding to a plurality of character strings having the identified meanings. As constructed, the natural language phrase includes a request for the 3D modeling system to place some books on a desk within the three-dimensional modeling environment. The 3D modeling system 102 first identifies each character string having the identified meaning, such that in this case, the 3D modeling system 102 identifies each character string in the natural language phrase as a token.

[0061] After identifying the tokens in the natural language phrase, the 3D modeling system 102 annotates each token with a label indicating the specific component of speech to which the token belongs. For example, the 3D modeling system 102 can identify and label nouns (including whether the noun is plural), verbs, adjectives, prepositions, and determiners in the natural language phrase. Figure 3A , the 3D modeling system 102 determines that "put" is a verb, "some" is an adjective, "books" is a plural noun, "on" is a preposition, "the" is a determiner, and "desk" is a singular noun. Accordingly, the 3D modeling system 102 has identified each word in the phrase and tagged it with its corresponding token type / category.

[0062] In addition, when generating the dependency tree, the 3D modeling system 102 also determines dependencies of each token in the context of the natural language phrase. In particular, each dependency indication involves a relationship of the token to one or more other tokens in the natural language phrase. For illustration, Figure 3AIt indicates that the 3D modeling system identifies the verb token "put" as acting on the direct object "book" as a noun modifier to place the direct object on the preposed noun "table" according to the identified preposition "on". In addition, the 3D modeling system 102 determines that "some" is an adjective modifying "book" to indicate that some number of books are placed on the table. The 3D modeling system 102 also determines that "the" is a qualifier of "table". By identifying and labeling the components and their relationships, the 3D modeling system generates a tree with the verb "put" as the structural center of the tree (e.g., the root node).

[0063] Therefore, the 3D modeling system 102 utilizes a natural language processor to determine labels for any nouns, verbs, and attributes in the natural language phrase. For more complex sentences, the 3D modeling system 102 can generate a larger dependency tree with more branches based on the corresponding relationships and the number of nouns, verbs, modifiers, and prepositions. As an example, the phrase "In the center of the table is a vase" includes multiple nouns that the 3D modeling system 102 identifies and labels, some of the multiple nouns are base nouns, and some of the multiple nouns are compound nouns, etc. Although the structure of the phrase is more complex than Figure 3A The embodiment of is more complex, but the 3D modeling system is able to create an accurate dependency tree for a phrase with "is" as the center of the structure.

[0064] Once the 3D modeling system 102 has created a dependency tree for a natural language phrase, the 3D modeling system 102 may then generate an entity-command representation for the natural language phrase. Figure 3B Picture shows Figure 3A An entity-command representation 302 of a natural language phrase is shown. In particular, the 3D modeling system 102 generates the entity-command representation 302 based on the components and dependencies identified in the dependency tree 300 for the natural language phrase.

[0065] In one or more embodiments, the 3D modeling system 102 generates entity-command representations by defining a set of entities and a set of commands for natural language phrases based on a dependency tree. In particular, the entity includes a category of a basic noun and is associated with any attributes of the basic noun, a count of the noun, a relationship that links the noun to another entity within the sentence, and any qualifiers corresponding to the noun. The entity category includes basic nouns (e.g., "table", "plate", "arrangement") used to describe objects in a three-dimensional scene. As mentioned, the basic noun includes those nouns that do not have a compound dependency relationship with another noun, are not abstract concepts, and do not represent spatial regions. The attributes of the basic noun include one or more modifiers (e.g., "modern", "blue (very, dark)") to modify the basic noun. The count of the noun includes an integer or qualitative descriptor representing the number of entities in the group (e.g., "2", "three", "many", "some"). The relationship that links the noun to another entity includes a group of (string, entity) pairs describing the connection with another specific entity in the sentence (e.g., "on...: desk", "to the left of...: keyboard"). Qualifiers corresponding to nouns include words, phrases, or affixes that express reference to the noun in context (eg, "a," "the," "another," "each").

[0066] Additionally, a command includes a base verb and is associated with any attributes for the verb and any targets of the verb. For illustration, a base verb includes a verb that is used to describe a command (e.g., "move," "rearrange"). The attributes of a verb include a set of one or more modifier words that help modify the verb / command (e.g., "closer," "dim (greater)"). The target of a verb is a list of (string, entity) pairs that represent different types of connections with a specific entity (e.g., "direct object: laptop," "on: desk").

[0067] As in Figure 3B As shown in FIG. 1 , the 3D modeling system 102 places the identified entities and commands in a logical configuration. Figure 3A The dependency tree 300 shown is converted into an entity-command representation 302, which is a logical configuration that relates entities and commands based on the determined dependencies from the dependency tree 300. For example, to convert the phrase "Put some books on the desk" into an entity-command representation, the 3D modeling system 102 identifies two separate entities - a first entity 304a ("books") and a second entity 304b ("desk").

[0068] In one or more embodiments, the 3D modeling system 102 uses pattern matching to determine other properties of the natural language phrase and assigns them to the corresponding entity or command in the entity-command representation 302. For example, if present, the 3D modeling system 102 determines that amod(noun: A, adjective: B) assigns token B as an attribute or count of the entity planted at A. When performing pattern matching, the 3D modeling system 102 enhances the standard speech part used by the natural language processor to create a dependency tree 300 with four classes used for scene understanding. The first class includes spatial nouns, which are spatial regions relative to an entity (e.g., "right", "center", "side"). The second class includes counting adjectives, which are adjectives that represent object counts or general qualifiers (e.g., "all", "many"). The third class includes group nouns that embody special meanings on sets of object classes (e.g., "stack", "arrange"). The fourth class includes adjectives, which the 3D modeling system 102 can model as attribute modifications to direct objects (e.g., "clean", "brighten").

[0069] To illustrate, the 3D modeling system 102 identifies associated components of the phrase that correspond to each of the entities. Specifically, when constructing the entity-command representation 302, the 3D modeling system 102 determines that "some" is a count adjective that modifies the first entity 304a, and creates a block 306 corresponding to "some" in connection with the first entity 304a. Additionally, the 3D modeling system 102 determines that "the" is a qualifier corresponding to the second entity 304b, and creates a block 308 corresponding to "the" in connection with the second entity 304b.

[0070] In addition to associating the appropriate blocks with the entities, the 3D modeling system 102 also associates the entities with the command(s) within the entity-command representation 302. In particular, the 3D modeling system 102 first determines that "put" is a command 310 within the phrase. Then, the 3D modeling system 102 determines that the first entity 304a is the target (i.e., the direct object) of the command 310, and creates a block 312 to indicate a target relationship between the first entity 304a and the command 310. The 3D modeling system 102 also determines that the second entity 304b has a relationship with the command 310 and the first entity 304a via a preposition (i.e., "on"), and creates a block 314 associating the second entity 304b with the command 310. Accordingly, the 3D modeling system 102 associates the first entity 304a and the second entity 304b via the determined relationship and the command 310.

[0071] Although Figure 3B The diagram shows Figure 3A302 for the phrase shown, but the 3D modeling system 102 can generate entity-command representations for more complex phrases involving any number of nouns and commands. For example, a natural language phrase can include multiple requests. The 3D modeling system 102 can parse the natural language phrase to determine each separate request and then generate a separate entity-command representation for each request. To illustrate, for a natural language phrase that states "move the chairs around the dining table farther away and move some books on the desk to the table", the 3D modeling system 102 can determine that the natural language phrase includes a first request to move the chairs around the table farther away and a second request to move some books from the desk to the table.

[0072] Additionally, as previously mentioned, the 3D modeling system 102 can use the coreference information to dereference the pronouns. This means that the 3D modeling system 102 does not create a new entity for the pronoun, but rather determines the corresponding noun, and then applies any attributes, relationships, etc. to the corresponding noun instead of the new entity. For example, in the phrase "add a dining table and put the plate on it", the 3D modeling system determines that "it" corresponds to the dining table.

[0073] In one or more embodiments, the 3D modeling system 102 determines that there is an ambiguity in a non-pronoun scene (e.g., two nouns with the same name or meaning), however, the 3D modeling system 102 can resolve the ambiguity when aligning entities with objects in the 3D three-dimensional scene. For example, in the phrase mentioned above requesting to move a book from a desk to a table, "table" may refer to the table in the first request to move a chair farther away, or "table" may refer to a different table in the same scene. Accordingly, the 3D modeling system 102 can create a new entity for "table" (e.g., in a separate entity-command representation). The 3D modeling system 102 will then attempt to resolve the ambiguity when constructing or modifying the requested three-dimensional scene by determining whether the scene includes one table or more than one table.

[0074] As in Figure 3C , after generating the entity-command representation 302, the 3D modeling system 102 generates a semantic scene graph 316 corresponding to the entity-command representation 302. In one or more embodiments, the 3D modeling system 102 generates the semantic scene graph 316 to create a representation of a previously generated three-dimensional scene that the 3D modeling system 102 can use to easily identify the most corresponding to the requested three-dimensional scene. As described below, the semantic scene graph 316 includes a plurality of nodes and edges that indicate spatial relationships between objects in the three-dimensional scene.

[0075] Specifically, the semantic scene graph is an undirected graph including object nodes, relationship nodes, and edge labels. Object nodes represent objects in a three-dimensional scene. The 3D modeling system 102 can annotate objects in the scene with a list of per-object attributes (e.g., "old," "wooden"). Relationship nodes represent specific instances of relationships between two or more objects (e.g., "to the left of," "on each side of," "around," "on"). In addition, each edge label connects an object node to a relationship node and describes the type of connection between the corresponding object and the relationship. Each of these components is described in more detail below.

[0076] In one or more embodiments, the 3D modeling system 102 generates the semantic scene graph 316 by first assigning each base noun to an object category. Specifically, the 3D modeling system 102 can use a model database with a fixed set of object categories to map the base nouns for each entity to the corresponding object category and create object nodes for the base nouns. For example, the 3D modeling system 102 can use an equivalence set obtained from Princeton WordNet. The 3D modeling system 102 discards entities that are not mapped to categories in the model database. Additionally, the 3D modeling system 102 adds attributes and qualifiers as annotations to the corresponding object nodes.

[0077] For FIG. 3A to FIG. 3B , the 3D modeling system 102 generates an object node for each of the identified entities. Specifically, the 3D modeling system 102 uses the entity count information from the entity-command representation 302 to instantiate a new object node for each object instance. For integer counts, the 3D modeling system 102 can simply generate a number of object nodes corresponding to the integer count. For illustration, an integer count of "three" causes the 3D modeling system 102 to create three separate object nodes for the entity.

[0078] For inaccurate counts such as “some” and “many”, the 3D modeling system 102 can determine the number of inaccurate counts by examining the available three-dimensional scenes (e.g., Figure 1The 3D modeling system 102 may use the distribution to obtain a lower and upper bound on the count implied by the modifier-category pairing, and then sample uniformly from the distribution. For example, the 3D modeling system 102 may sample between the 0th and 2nd percentiles for "a few", between the 10th and 50th percentiles for "some", and between the 50th and 100th percentiles for "many". The 3D modeling system 102 may infer that a plural noun without a modifier (e.g., "there are chairs around the table") has an implicit "some" modifier. The 3D modeling system 102 replicates relationships, attributes, and qualifiers across each new instance of an object.

[0079] In addition, the 3D modeling system 102 identifies any qualifiers in the entity-command representation to determine the presence of multiple object nodes. For example, qualifiers such as "each" and "all" imply that there is more than one object node for a particular entity. In one or more embodiments, the 3D modeling system 102 leaves the qualifier on a single object node until the 3D modeling system 102 generates a three-dimensional scene using the semantic scene graph 316.

[0080] The 3D modeling system 102 also transforms the relationship information from the entity-command representation 302 into relationship nodes in the semantic scene graph 316. In particular, the 3D modeling system 102 can generate a relationship node for each of the objects based on the relationship based on the preposition or other relationship information in the entity-command relationship 302. For illustration, the 3D modeling system 102 can insert relationship nodes indicating "whether the object is above another object in the scene", "whether the object is to the left of another object in the scene", "whether the object is below another object in the scene", "whether the object is in front of another object in the scene", etc. Relationships that support more than one object are grouped together into a single relationship node in the semantic scene graph 316.

[0081] For illustration, Figure 3C Shown by targeting Figure 3BThe single relationship node of the entity-command representation 302 shown is associated with multiple object nodes. Specifically, the semantic scene graph 316 includes a first object node 318 representing a second entity 304b (corresponding to the "desk" object) from the entity-command representation. Additionally, the semantic scene graph 316 includes a plurality of object nodes 320a to 320c representing a first entity 304a (corresponding to the "book" object) from the entity-command representation. The semantic scene graph 316 includes a relationship node 322, which indicates that the corresponding objects of the object nodes 320a to 320c are "on the corresponding object of the object node 318." Additionally, as shown, since all books share the same relationship with the desk, the 3D modeling system 102 only includes one relationship node 322 in the semantic scene graph 316.

[0082] As previously described, the 3D modeling system 102 determines a count for each entity from the entity-command representation 302 . Figure 3C The diagram shows that the 3D modeling system 102 has inferred a count of objects based on the "book" entity (the first entity 304a in the entity-command representation 302). In particular, the 3D modeling system 102 analyzes the available scenes from the content database to determine a possible count range for the imprecise count "some". In this embodiment, the 3D modeling system 102 determines that "some" includes three book objects by uniformly sampling the 10th percentile to the 50th percentile of the frequency histogram of the available scenes. Accordingly, the 3D modeling system 102 generates three separate object nodes for the entity.

[0083] In addition to object nodes and relationship nodes, the semantic scene graph 316 also includes multiple edges that describe the relationship between each object node via relationship nodes. Specifically, the edge indicates the directionality of the relationship between two object nodes from the entity-command representation 302. For example, the semantic scene graph 316 includes a first edge 324 that represents the following information: the information indicates that the desk is a lower object relative to one or more other objects. The semantic scene graph 316 also includes multiple edges 326a to 326c, which represent information that the book is a higher object relative to one or more other objects. Since the relationship node 322 indicates a "on..." relationship, the combination of the first edge 324 and the multiple edges 326a to 326c with the relationship node 322 and the object nodes 318, 320a to 320c indicates to the 3D modeling system 102 that the book is above the desk in the three-dimensional scene.

[0084] In one or more embodiments, before generating a semantic scene graph for a natural language phrase, the 3D modeling system 102 first determines a canonical entity-command representation. As briefly described previously, a single conceptual request to generate a particular three-dimensional scene can be parsed in a number of different ways. For illustration, to generate a three-dimensional scene of books on a desk, a user may say a first phrase "books are piled on the desk", a second phrase "there is a pile of books on the desk", or a third phrase "stack the books on the desk". As described by FIG. 4A to FIG. 4C As shown in the entity-command representations 400 to 404 in FIG. 1 , each phrase includes a unique syntax with different nouns, verbs, prepositions, etc., thereby generating different entity-command representations. Specifically, the first entity-command representation 400 corresponds to the first phrase above, the second entity-command representation 402 corresponds to the second phrase above, and the third entity-command representation 404 corresponds to the third phrase above.

[0085] In order to create a consistent form of entity-command representation, the 3D modeling system 102 may select a particular form of entity-command representation as a canonical representation. In particular, the canonical representation allows the 3D modeling system 102 to determine a single entity-command representation to be generated for each form of concept request. In one or more embodiments, if possible, the 3D modeling system 102 selects a descriptive form of entity-command representation (first entity-command representation 400) as a canonical representation. As described herein, a descriptive form of entity-command representation is a representation in which a base noun is an object having a set of attributes that describe the base noun. If the user enters a request using a different form, the 3D modeling system 102 uses a set of pattern matching rules (e.g., similar to the pattern matching rules used to generate the entity-command representation) to transform the resulting entity-command representation into a descriptive form.

[0086] In response to generating a semantic scene graph for a natural language phrase, the 3D modeling system 102 uses the semantic scene graph to generate a three-dimensional scene. In one or more embodiments, the 3D modeling system 102 identifies one or more available scenes that are most similar to the scene requested in the natural language phrase. For example, the 3D modeling system 102 compares the semantic scene graph with the semantic scene graphs for multiple available three-dimensional scenes to identify one or more scenes similar to the requested scene. For illustration, the 3D modeling system 102 can compare the semantic scene graph of the natural language phrase with each semantic scene graph of the available scenes by comparing the object nodes, relationship nodes, and edges in the semantic scene graph of the natural language phrase with the object nodes, relationship nodes, and edges in the semantic scene graph of the available scenes. A higher number of overlapping nodes and edges indicates a closer match, while a lower number of overlapping nodes and edges indicates a lesser match.

[0087] Additionally, the 3D modeling system 102 can rank the available scenes by determining the degree of match of the corresponding semantic scene graph with the semantic scene graph of the natural language phrase. For example, the 3D modeling system 102 can assign a comparison score to each semantic scene graph of the available three-dimensional scenes based on the number and similarity of nodes and edges and the semantic scene graph of the natural language phrase. The 3D modeling system 102 can then use the comparison scores to generate a ranked list of scenes, wherein higher comparison scores are higher on the ranked list of scenes. The 3D modeling system 102 can also use a threshold score to determine whether any of the available scenes is closely aligned with the natural language phrase.

[0088] Figure 5 Diagram showing the content database based on Figure 3C The 3D modeling system 102 compares the semantic scene graph 316 with the semantic scene graphs of each scene and then selects at least one scene that best matches the expected request in the natural language phrase.

[0089] The 3D modeling system 102 selects the second scene 502 for generating a three-dimensional scene of a natural language phrase. The 3D modeling system selects the second scene 502 because the second scene 502 is most similar to the semantic scene graph 316 (i.e., the second scene 502 includes a desk, wherein there are books on the desk). In particular, even if the second scene 502 has some differences based on the semantic scene graph of the second scene 502 (i.e., as in the semantic scene graph 316, there are six books instead of three books), the second scene 502 is still closer to the requested scene than other available scenes (e.g., the book is on the desk, not the book is on the table or in the bookshelf). In addition, since the requested scene includes an inaccurate count ("some") of the referenced book entity, the 3D modeling system 102 can determine that the second scene 502 is an acceptable deviation from the semantic scene graph 316. Additionally, the 3D modeling system 102 may need to share certain entities, commands, and / or other attributes to make the scene a close match.

[0090] As briefly mentioned previously, the 3D modeling system 102 can generate a three-dimensional scene for a natural language phrase by inserting objects in a selected available three-dimensional scene (or from multiple available three-dimensional scenes) into a user's workspace. Specifically, the 3D modeling system 102 can select an object in the available three-dimensional scene based on an object identifier corresponding to the object. The 3D modeling system 102 can then copy the object and one or more attributes of the object and paste / duplicate the object in the user's workspace. The 3D modeling system 102 can perform a copy-paste operation on multiple objects in the selected available three-dimensional scene until the 3D modeling system 102 determines that the requested three-dimensional scene is complete. For illustration, the 3D modeling system 102 can copy the desk and book of the second scene 502 shown in FIG. 4 and paste the copied objects into the user's workspace at the user's client device.

[0091] In an alternative embodiment, the 3D modeling system 102 presents the selected three-dimensional scene to the user to allow the user to insert objects into the workspace. For example, the 3D modeling system 102 can cause the user's client device to open a new workspace within the client application. The user can then select one or more objects in the client application and move the objects to a pre-existing workspace or leave the objects in a new workspace to start a new project.

[0092] In one or more embodiments, the 3D modeling system 102 allows a user to enhance an existing three-dimensional scene using natural language requests. FIG. 6A to FIG. 6C The figure shows multiple 3D scenes generated continuously using natural language phrases. Fig. 6A , the 3D modeling system 102 generates a three-dimensional scene 600 including a first set of objects 602. In particular, the 3D modeling system 102 identifies a first natural language phrase including a request to generate the three-dimensional scene 600 to include one or more objects. The 3D modeling system 102 parses the first natural language phrase to generate a semantic scene graph for the first phrase, and then uses the semantic scene graph to select one or more available three-dimensional scenes including the first set of objects 602 to generate the three-dimensional scene 600.

[0093] Additionally, the 3D modeling system 102 identifies a second natural language phrase that includes a request to enhance the three-dimensional scene 600 with the second set of objects 604. The 3D modeling system 102 parses the second natural language phrase to create a semantic scene graph representing the second phrase. The 3D modeling system 102 uses the semantic scene graph of the second phrase to select one or more available three-dimensional scenes that include the second set of objects 604. Figure 6B As illustrated in , the 3D modeling system then enhances the three-dimensional scene 600 by inserting a second set of objects 604 into the three-dimensional scene.

[0094] Figure 6C The third group of objects 606 inserted into the three-dimensional scene 600 is illustrated. Specifically, the 3D modeling system 102 identifies a request in the third natural language phrase to further enhance the three-dimensional scene 600. The 3D modeling system 102 parses the third natural language phrase to create a corresponding semantic scene graph, and then uses the semantic scene graph of the third phrase to select one or more available three-dimensional scenes including the third group of objects 606. Finally, the 3D modeling system 102 enhances the three-dimensional scene 600 by inserting the third group of objects 606 into the three-dimensional scene 600.

[0095] As in FIG. 6A to FIG. 6C As shown in the example of , the 3D modeling system 102 can enhance the three-dimensional scene by inserting additional three-dimensional objects into the scene at specific locations within the scene. In one or more embodiments, the 3D modeling system 102 determines the location of the new object based on information contained in the natural language phrase. For example, in response to natural language phrases stating "insert the sofa", "put the TV in front of the sofa", and "put the coffee table between the sofa and the TV", the 3D modeling system 102 can insert the coffee table between the sofa and the TV. The 3D modeling system 102 can use additional intelligence to determine the location of objects within the scene, and if necessary, can adjust the location of objects already in the scene to accommodate the new object.

[0096] As about Figures 1 to 6C As described, 3D system 102 may therefore perform operations for processing natural language phrases to generate a three-dimensional scene. Figure 7 Detailed schematic diagram of an embodiment of the 3D system 102 described above is shown. As shown, the 3D modeling system 102 can be implemented on (multiple) computing devices 700 (e.g., on Figure 1 and below about Fig. 9 702, a client device and / or a server device as further described herein. Additionally, the 3D modeling system 102 may include, but is not limited to, a content manager 702, a communication manager 704, a natural language processor 706, a 3D scene generator 708, and a data storage manager 710. The 3D modeling system 102 may be implemented on any number of computing devices. For example, the 3D modeling system 102 may be implemented in a distributed system of server devices for generating a three-dimensional scene based on a natural language phrase. Alternatively, the 3D modeling system 102 may be implemented on a single computing device, such as a single client device running a client application that processes natural language for generating a three-dimensional scene.

[0097] In one or more embodiments, each of the components of the 3D modeling system 102 communicates with the other components using any suitable communication technology. Additionally, the components of the 3D modeling system 102 may communicate with one or more other devices, including: other computing devices of the user, server devices (e.g., cloud storage devices), license servers, or other devices / systems. It is to be appreciated that although the components of the 3D modeling system 102 are Figure 7 Although shown as being separated, any subcomponents may be combined into fewer components, such as into a single component, or separated into more components, as may be useful for a particular implementation. Figure 7 The illustrated components are described as being connected to the 3D modeling system 102 , but at least some of the components used to perform operations in conjunction with the 3D modeling system 102 described herein may be implemented on other devices within the environment.

[0098] The components of the 3D modeling system 102 may include software, hardware, or both. For example, the components of the 3D modeling system 102 may include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices (e.g., computing device(s) 700). When executed by the one or more processors, the computer-executable instructions of the 3D modeling system 102 may cause the computing device(s) 700 to perform the three-dimensional generation operations described herein. Alternatively, the components of the 3D modeling system 102 may include hardware, such as a dedicated processing device to perform a particular function or group of functions. Additionally, or alternatively, the components of the 3D modeling system 102 may include a combination of computer-executable instructions and hardware.

[0099] Furthermore, for example, the components of the 3D modeling system 102 that perform the functions described herein for the 3D modeling system 102 may be implemented as part of a standalone application, a module of an application, a plug-in for an application including a marketplace application, one or more library functions that may be called by other applications, and / or a cloud computing model. Thus, the components of the 3D modeling system 102 may be implemented as part of a standalone application on a personal computing device or a mobile device. Alternatively or additionally, the components of the 3D modeling system 102 may be implemented in any application that allows for three-dimensional content generation, including but not limited to: CREATIVE FUSE, and "ADOBE", "ADOBE FUSE", "CREATIVE CLOUD", "ADOBE FUSE", "ADOBE DIMENSION" and "ILLUSTRATOR" are registered trademarks of Adobe Inc. in the United States and / or other countries.

[0100] As mentioned, the 3D modeling system 102 includes a content manager 702 that facilitates storage and management of three-dimensional content. Specifically, the content manager 702 can access a content database (e.g., Figure 1 The content manager 702 may include a content database 106 (shown), which includes a plurality of three-dimensional scenes that have been previously created by any number of content creators. The content manager 702 allows content creators to store and share three-dimensional content across applications and devices and / or with other users. In addition, the content manager 702 may store and / or manage assets used to generate three-dimensional scenes.

[0101] The 3D modeling system 102 includes a communication manager 704 that facilitates communication between the 3D modeling system 102 and one or more computing devices and / or systems. For example, the communication manager 704 can facilitate communication with one or more client devices of a user to receive a natural language request to generate a three-dimensional scene. Additionally, the communication manager 704 can facilitate communication with the client device to provide a three-dimensional scene generated in response to the natural language request. In addition, the communication manager 704 can allow the client device to store the three-dimensional scene on the content manager 702 for later use by one or more users.

[0102] The 3D modeling system 102 also includes a natural language processor 706 that facilitates processing and analyzing natural language phrases. Specifically, the natural language processor 706 can use natural language processing techniques to generate dependency trees, entity-command representations, and semantic scene graphs of natural language phrases. The natural language processor 706 can also communicate with one or more other components (e.g., a 3D scene generator) to provide a semantic scene graph for comparison with available three-dimensional scenes in the content manager 702.

[0103] The 3D modeling system 102 also includes a 3D scene generator 708 that facilitates generating a three-dimensional scene. In particular, the 3D scene generator 708 can use the semantic scene graph of the natural language phrase from the natural language processor 706 to identify one or more available three-dimensional scenes that the 3D scene generator can use to generate a three-dimensional scene based on the natural language phrase. The 3D scene generator 708 can also perform operations within the three-dimensional environment to insert, remove, manipulate, or otherwise modify three-dimensional content in the three-dimensional content scene.

[0104] The 3D modeling system 102 also includes a data storage manager 710 (which includes non-transitory computer memory) that stores and maintains data associated with processing natural language phrases and generating three-dimensional scenes. For example, the data storage manager 710 can store semantic scene graphs for natural language phrases and three-dimensional scenes. The data storage manager 710 can also store information for accessing and searching content databases in conjunction with the content manager 702.

[0105] Now turn to Figure 8 , which shows a flow chart of a series of actions 800 for synthesizing a three-dimensional scene using natural language. Figure 8 The actions according to one embodiment are illustrated, but alternative embodiments may omit the Figure 8 Any action shown in the action, to Figure 8 Add any of the actions shown in Figure 8 Any of the actions shown in the sequence, and / or modify the Figure 8 Any action in the action. Figure 8 The actions in may be performed as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform Figure 8 In other embodiments, the system may perform Figure 8 Action in.

[0106] As shown, the series of actions 800 includes an action 802 of analyzing a natural language phrase. For example, action 802 involves using natural language processing to analyze a natural language phrase including a request to generate a three-dimensional scene to determine dependencies involving one or more entities and one or more commands in the natural language phrase. Action 802 may involve receiving a text input including a natural language phrase or a voice input including a natural language phrase. Additionally, the natural language phrase may include multiple requests to generate multiple three-dimensional scenes. Action 802 may also involve receiving multiple natural language phrases.

[0107] The series of actions 800 also includes an action 804 of generating an entity-command representation of a natural language phrase. For example, action 804 involves generating an entity-command representation of a natural language phrase using the determined dependencies between one or more entities and one or more commands. For example, action 804 may first involve generating a dependency tree, the dependency tree including multiple tokens representing words in the natural language phrase and dependency relationships corresponding to the multiple tokens. Then, action 804 may involve converting the dependency tree into: an entity list including one or more entities annotated with one or more attributes and one or more relationships corresponding to the one or more entities; and a command list including one or more command verbs operating on the one or more entities.

[0108] Action 804 may also involve identifying one or more additional natural language phrases in one or more different phrase forms that include a request to generate a three-dimensional scene. Action 804 may further involve generating canonical entity-command representations for the one or more additional natural language phrases in one or more different phrase forms.

[0109] Action 804 may also involve determining that an object node includes an inaccurate count corresponding to a basic noun. Then, action 804 may involve determining a frequency histogram for an object category by analyzing a scene database to count the number of times two or more entity instances occur in an object category. Action 804 may further involve determining a plurality of objects to be included in a three-dimensional scene for an entity by sampling the distribution of count modifiers represented for an entity-command, the distribution being determined based on a frequency histogram. For example, action 804 may involve determining a plurality of object nodes to be included in a semantic scene graph for a basic noun by sampling the distribution determined based on a frequency histogram.

[0110] Additionally, the series of actions 800 includes an action 806 of generating a three-dimensional scene. For example, action 806 involves generating a three-dimensional scene by using at least one of a plurality of available three-dimensional scenes identified by using entity-command representations based on natural language phrases. For example, action 806 may involve accessing a database including previously generated three-dimensional scenes having corresponding semantic scene graphs representing the layout of objects within the previously generated three-dimensional scenes.

[0111] Action 806 may also involve converting the entity-command representation of the natural language phrase into a semantic scene graph indicating the contextual relationship of one or more entities and one or more commands. For example, action 806 may involve mapping the base noun for an entity in the one or more entities to an object node of an object category. Action 806 may also involve mapping the relationship corresponding to the one or more entities to a relationship node, wherein the edge indicates the direction of the relationship corresponding to the one or more entities. Action 806 may involve generating an edge indicating the direction of the relationship corresponding to the one or more entities.

[0112] Action 806 may also involve adding an attribute or qualifier as an annotation to the object node of the base noun.In addition, action 806 may involve using the semantic scene graph to select a three-dimensional scene from a plurality of available three-dimensional scenes corresponding to the semantic scene graph.

[0113] Additionally, action 806 may involve comparing the semantic scene graph of the natural language phrase with the semantic scene graphs of a plurality of available three-dimensional scenes. For example, action 806 may involve ranking the plurality of available three-dimensional scenes based on the similarity of the semantic scene graph of the natural language phrase with the semantic scene graphs of the plurality of available three-dimensional scenes. Action 806 may also involve identifying a three-dimensional scene having a semantic scene graph that matches the semantic scene graph of the natural language phrase from the plurality of available three-dimensional scenes. For example, action 806 may involve selecting the three-dimensional scene with the highest ranking from the plurality of available three-dimensional scenes.

[0114] As discussed in more detail below, embodiments of the present disclosure may include or utilize a special-purpose or general-purpose computer including computer hardware (such as, for example, one or more processors and system memory). Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transient computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). Typically, a processor (e.g., a microprocessor) receives instructions from a non-transient computer-readable medium (e.g., a memory) and executes those instructions to perform one or more processes, including one or more of the processes described herein.

[0115] Computer-readable media can be any available media that can be accessed by a general or special-purpose computer system. A computer-readable medium that stores computer-executable instructions is a non-transient computer-readable storage medium (device). A computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present disclosure may include at least two distinct computer-readable media: a non-transient computer-readable storage medium (device) and a transmission medium.

[0116] Non-transitory computer-readable storage media (devices) include: RAM, ROM, EEPROM, CD-ROM, solid-state drives ("SSD") (e.g., RAM-based), flash memory, phase-change memory ("PCM"), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code components in the form of computer-executable instructions or data structures and that can be accessed by a general or special purpose computer.

[0117] "Network" is defined as one or more data links that support the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer through a network or another communication connection (hard-wired connection, wireless connection, or a combination of hard-wired connection or wireless connection), the computer properly regards the connection as a transmission medium. The transmission medium may include a network and / or a data link, which may be used to carry computer executable instructions or data structure forms and may be accessed by a general or special-purpose computer. The above combination should also be included in the scope of computer-readable media.

[0118] Further, when the program code parts of computer executable instructions or data structure form arrive various computer system components, these program code parts can be automatically transferred to non-transient computer readable storage medium (equipment) (or vice versa) from transmission medium.For example, the computer executable instructions or data structure received by network or data link can be buffered in the RAM in the network interface module (for example, " NIC "), and then finally transferred to the computer system RAM and / or the less volatile computer storage medium (equipment) at the computer system.Therefore, it should be understood that non-transient computer readable storage medium (equipment) can be included in the computer system component that also (or even mainly) utilizes transmission medium.

[0119] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions when executed at a processor. In some embodiments, computer executable instructions are executed on a general-purpose computer to convert a general-purpose computer into a special-purpose computer that implements the elements in the present disclosure. Computer executable instructions can be, for example, binary, intermediate format instructions (such as, assembly language) or even source code. Although the subject matter has been described in a language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or the actions described above. On the contrary, the described features and actions are disclosed as example forms of implementing the claims.

[0120] Those skilled in the art will appreciate that the present disclosure can be practiced in a network computing environment with many types of computer system configurations, including: personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, tablets, pagers, routers, switches, etc. The present disclosure can also be practiced in a distributed system environment, in which local computer systems and remote computer systems linked through a network (by a hardwired data link, a wireless data link, or a combination of a hardwired data link and a wireless data link) all perform tasks. In a distributed system environment, program modules can be located in local memory storage devices and remote memory storage devices.

[0121] Embodiments of the present disclosure may also be implemented in a cloud computing environment. In this specification, "cloud computing" is defined as a model for implementing on-demand network access to a shared pool of configurable computing resources. For example, cloud computing may be employed in a market to provide universal and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly provided via virtualization, and may be published with less management effort or service provider interaction, and then scaled accordingly.

[0122] The cloud computing model may consist of various characteristics, such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model may also exhibit various service models, such as, for example, software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this specification and claims, a "cloud computing environment" is an environment that employs cloud computing.

[0123] Fig. 9 A block diagram of an exemplary computing device 900 that can be configured to perform one or more of the processes described above is illustrated. It should be appreciated that one or more computing devices (such as computing device 900) can implement a multi-RNN prediction system. Fig. 9 As shown, computing device 900 may include processor 902, memory 904, storage device 906, I / O interface 908, and communication interface 910, which may be communicatively coupled via communication infrastructure 912. In some embodiments, computing device 900 may include a processor 902, memory 904, storage device 906, I / O interface 908, and communication interface 910. Fig. 9 The components shown in FIG. 1 may be fewer or more than those shown in FIG. 1 . Fig. 9 Components of computing device 900 are shown in FIG.

[0124] In one or more embodiments, the processor 902 includes hardware for executing instructions (such as those that constitute a computer program). As an example and not by way of limitation, in order to execute instructions for dynamically modifying a workflow, the processor 902 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 904, or a storage device 906 and decode and execute them. The memory 904 may be a volatile or non-volatile memory used to store data, metadata, and programs for execution by the processor(s). The storage device 906 includes a storage device (such as a hard disk, a flash disk drive, or other digital storage device) for storing data or instructions for performing the methods described herein.

[0125] The I / O interface 908 allows a user to provide input to the computing device 900, receive output from the computing device 900, and otherwise transfer data to and receive data from the computing device 900. The I / O interface 908 may include a mouse, a keypad or keyboard, a touch screen, a camera, an optical scanner, a network interface, a modem, other known I / O devices, or a combination of such I / O interfaces. The I / O interface 908 may include one or more devices for presenting output to a user, including but not limited to: a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O interface 908 is configured to provide graphics data to the display for presentation to the user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content that may serve a specific implementation.

[0126] The communication interface 910 may include hardware, software, or both. In any case, the communication interface 910 may provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device 900 and one or more other computing devices or networks. As an example and not by way of limitation, the communication interface 910 may include: a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (such as, WI-FI).

[0127] Additionally, the communication interface 910 can facilitate communication with various types of wired or wireless networks. The communication interface 910 can also facilitate communication using various communication protocols. The communication infrastructure 912 can also include hardware, software, or both that couple the components of the computing device 900 to each other. For example, the communication interface 910 can use one or more networks and / or protocols to enable multiple computing devices connected by a specific infrastructure to communicate with each other to perform one or more aspects of the process described herein. For illustration, the digital content activity management process can allow multiple devices (e.g., client devices and server devices) to use various communication networks and protocols to exchange information for sharing information, such as, electronic messages, user interaction information, engagement metrics, or activity management resources.

[0128] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. Various embodiments and aspects of the present disclosure are described with reference to the details discussed herein, and the accompanying drawings illustrate various embodiments. The above description and the accompanying drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Many specific details are described to provide a thorough understanding of the various embodiments of the present disclosure.

[0129] Without departing from the spirit or essential features of the present disclosure, the present disclosure may be embodied in other specific forms. The described embodiments should be considered in all respects only as illustrative and not restrictive. For example, the method described herein may be performed using fewer or more steps / actions, or the steps / actions may be performed in different orders. Additionally, the steps / actions described herein may be repeated or performed in parallel with each other, or repeated or performed in parallel with different instances of the same or similar steps / actions. Therefore, the scope of the present application is indicated by the appended claims rather than the foregoing description. All changes within the equivalent meaning and scope of the claims will be included within the scope of the claims.

Claims

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computer system to: analyzing a natural language phrase including a request to generate a three-dimensional scene using natural language processing to determine a dependency tree including dependencies of a plurality of entities involving the natural language phrase and one or more commands operating on the plurality of entities; generating an entity-command representation of the natural language phrase using the determined dependencies between the plurality of entities and the one or more commands operating on the plurality of entities; Converting the entity-command representation of the natural language phrase into a semantic scene graph based on the contextual relationship between the multiple entities and the one or more commands, the semantic scene graph including relative positioning of multiple objects in a scene, by: generating a plurality of object nodes representing the plurality of entities from the entity-command representation, and generating one or more relationship nodes representing spatial relationships between the plurality of object nodes; as well as generating a plurality of edges connecting the plurality of object nodes and the one or more relationship nodes, wherein the plurality of edges indicate directionality of the spatial relationship between the plurality of object nodes; as well as The three-dimensional scene is generated by using at least one three-dimensional scene from a plurality of available three-dimensional scenes according to the semantic scene graph.

2. The non-transitory computer-readable medium of claim 1 , wherein the instructions that cause the computer system to generate the three-dimensional scene cause the computer system to select a three-dimensional scene from the multiple available three-dimensional scenes based on multiple semantic scene graphs of the multiple available three-dimensional scenes, using the semantic scene graphs, wherein the multiple available three-dimensional scenes include previously generated three-dimensional scenes.

3. The non-transitory computer readable medium of claim 2, wherein the instructions causing the computer system to select the three-dimensional scene from the plurality of available three-dimensional scenes further cause the computer system to: comparing the semantic scene graph of the natural language phrase to semantic scene graphs of the plurality of available three-dimensional scenes; and A three-dimensional scene having a semantic scene graph that matches the semantic scene graph of the natural language phrase is selected from the plurality of available three-dimensional scenes.

4. The non-transitory computer-readable medium of claim 2, wherein the instructions causing the computer system to generate the three-dimensional scene by using the at least one three-dimensional scene from the plurality of available three-dimensional scenes based on the semantic scene graph further cause the computer system to access a database comprising previously generated three-dimensional scenes, the previously generated three-dimensional scenes having corresponding semantic scene graphs representing the layout of objects within the previously generated three-dimensional scenes.

5. The non-transitory computer-readable medium of claim 1 , wherein the instructions that cause the computer system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the computer system to: Mapping a base noun to an object node in the plurality of object nodes based on an object category of a base noun for an entity in the plurality of entities; and A relationship between the entity and an additional entity from the plurality of entities is mapped to a relationship node having an edge indicating a directionality of the relationship between the entity and the additional entity from the plurality of entities within the semantic scene graph.

6. The non-transitory computer-readable medium of claim 5, wherein the instructions causing the computer system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the computer system to add attributes or qualifiers of the base noun from the natural language phrase as annotations to the object node of the base noun within the semantic scene graph.

7. The non-transitory computer-readable medium of claim 1 , wherein the instructions that cause the computer system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the computer system to: determining a frequency histogram for the object class by analyzing a context database to count occurrences of two or more entity instances in an object class of an entity in the plurality of entities; and A plurality of objects to be included in the three-dimensional scene for the entity of the plurality of entities is determined by sampling a distribution of counted modifiers for the entity-command representation, the distribution being determined based on the frequency histogram.

8. The non-transitory computer-readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to: identifying a plurality of natural language phrases including a request to generate the three-dimensional scene in a plurality of different phrase forms, the plurality of natural language phrases including the analyzed natural language phrase, The instructions causing the computer system to generate the entity-command representations of the natural language phrases cause the computer system to generate canonical entity-command representations for the plurality of natural language phrases in the plurality of different phrase forms.

9. The non-transitory computer-readable medium of claim 1 , wherein the instructions that cause the computer system to generate the entity-command representation of the natural language phrase cause the computer system to: generating the dependency tree including a plurality of tokens representing words in the natural language phrase and dependency relationships corresponding to the plurality of tokens; and Transform the dependency tree into: an entity list including the plurality of entities annotated with one or more attributes and the one or more relationships corresponding to the plurality of entities; and A command list includes one or more command verbs that operate on the plurality of entities.

10. A system for synthesizing a three-dimensional scene using natural language in a digital media environment for three-dimensional computer modeling, comprising: at least one processor; as well as Non-transitory computer memory, including: a database of previously generated three-dimensional scenes; and instructions that, when executed by the at least one processor, cause the system to: identifying a natural language phrase comprising a request to generate a three-dimensional scene; Using natural language processing to analyze a natural language phrase including the request to generate a three-dimensional scene to mark a character string in the natural language phrase as an entity or a command; determining a dependency tree, the dependency tree comprising one or more dependencies involving a plurality of entities and one or more commands operating on the plurality of entities; generating an entity-command representation of the natural language phrase by performing pattern matching on the plurality of entities and the one or more commands operating on the plurality of entities based on the determined one or more dependencies; Converting the entity-command representation of the natural language phrase into a semantic scene graph based on the contextual relationship between the multiple entities and the one or more commands, the semantic scene graph including relative positioning of multiple objects in a scene, by: generating a plurality of object nodes representing the plurality of entities from the entity-command representation, and generating one or more relationship nodes representing spatial relationships between the plurality of object nodes; and generating a plurality of edges connecting the plurality of object nodes and the one or more relationship nodes, wherein the plurality of edges indicate directionality of the spatial relationship between the plurality of object nodes; and The three-dimensional scene is generated according to the semantic scene graph using at least one previously generated three-dimensional scene from the database of previously generated three-dimensional scenes.

11. The system of claim 10, wherein the instructions causing the system to generate the three-dimensional scene cause the system to select a three-dimensional scene from the database of previously generated three-dimensional scenes based on a semantic scene graph of the previously generated three-dimensional scenes using the semantic scene graph.

12. The system of claim 11, wherein the instructions causing the system to select the three-dimensional scene from the database of previously generated three-dimensional scenes further cause the system to: comparing the semantic scene graph of the natural language phrase to the semantic scene graph of the previously generated three-dimensional scene; ranking the previously generated three-dimensional scenes based on similarities between the semantic scene graph of the natural language phrase and the semantic scene graph of the previously generated three-dimensional scenes; European Space A three-dimensional scene with a highest ranking is selected from the previously generated three-dimensional scenes.

13. The system of claim 10, wherein the instructions causing the system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the system to: mapping a base noun to an object node in the plurality of object nodes based on an object category of a base noun for an entity in the plurality of entities; mapping a relationship between the entity and an additional entity of the plurality of entities to a relationship node based on the one or more dependencies; as well as An edge is generated that indicates a directionality of the relationship between the entity and the additional entity within the semantic scene graph.

14. A system according to claim 13, wherein the instructions that cause the system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the system to add attributes or qualifiers of the base noun from the natural language phrase as annotations to the object node of the base noun.

15. The system of claim 13, wherein the instructions causing the system to convert the entity-command representation of the natural language phrase into the semantic scene graph cause the system to: Based on the natural language phrase, determining that the object node includes an imprecise count corresponding to the base noun; For each of the previously generated three-dimensional scenes, determining a frequency histogram for the object class by analyzing the database of previously generated three-dimensional scenes to count the number of occurrences of two or more entity instances in the object class; and By sampling the distribution determined based on the frequency histogram, a plurality of object nodes to be included in the semantic scene graph for the base noun are determined.

16. The system of claim 10, wherein: The instructions causing the system to identify the natural language phrase cause the system to identify a plurality of natural language phrases in a plurality of different phrase forms that include a request to generate the three-dimensional scene, the plurality of natural language phrases including the identified natural language phrase, and The instructions causing the system to generate the entity-command representations of the natural language phrases cause the system to generate canonical entity-command representations for the plurality of natural language phrases in the plurality of different phrase forms.

17. The system of claim 10, wherein the instructions that cause the system to generate the entity-command representation of the natural language phrase cause the system to: generating the dependency tree including a plurality of tokens representing words in the natural language phrase and dependency relationships corresponding to the plurality of tokens; and Transform the dependency tree into: an entity list including one or more entities annotated with one or more attributes and one or more relationships corresponding to the one or more entities; and A command list comprising one or more command verbs operating on the one or more entities.

18. A computer-implemented method for synthesizing a three-dimensional scene using natural language in a digital media environment for three-dimensional computer modeling comprising: analyzing a natural language phrase including a request to generate a three-dimensional scene using natural language processing to determine a dependency tree including dependencies of a plurality of entities involving the natural language phrase and one or more commands operating on the plurality of entities; generating an entity-command representation of the natural language phrase using the determined dependencies between the plurality of entities and the one or more commands operating on the plurality of entities; Generating a semantic scene graph from the entity-command representation based on the contextual relationship between the plurality of entities and the one or more commands, the semantic scene graph indicating relative positioning of a plurality of objects in a scene, by: generating a plurality of object nodes representing the plurality of entities from the entity-command representation, and generating one or more relationship nodes representing spatial relationships between the plurality of object nodes; as well as generating a plurality of edges connecting the plurality of object nodes and the one or more relationship nodes, wherein the plurality of edges indicate directionality of the spatial relationship between the plurality of object nodes; as well as The three-dimensional scene is generated according to the semantic scene graph using at least one previously generated three-dimensional scene from a database of previously generated three-dimensional scenes.

19. The computer-implemented method of claim 18, further comprising: mapping a base noun for an entity of the plurality of entities to an object node of an object class; as well as Relationships corresponding to the one or more entities are mapped to relationship nodes having edges indicating directions of the relationships corresponding to the plurality of entities within the semantic scene graph.

20. The computer-implemented method of claim 18, wherein generating the three-dimensional scene comprises: comparing the semantic scene graph to a plurality of semantic scene graphs corresponding to the database of previously generated three-dimensional scenes; as well as A three-dimensional scene having a semantic scene graph that matches the semantic scene graph of the natural language phrase is selected from the previously generated three-dimensional scenes.

Images