Method and system for generating 3d scene based on natural language description

Abstract

The invention provides a method and system for generating a 3d scene based on natural language description, and relates to the technical field of computer graphics, and the method comprises the steps: constructing a 3d asset function library, converting 3d assets into functions controlled by parameters, and storing the functions in the 3d asset function library; constructing a knowledge base, wherein the knowledge base comprises the function and the use method of each function in the 3d asset function library and the meaning of each parameter; natural language description input by a user is converted into semantic information understood by a computer based on a large model; obtaining 3d assets needing to be constructed according to the semantic information, selecting functions in a 3d asset function library based on the large model, determining parameter values of the functions according to a knowledge base, and generating program codes; and running the program code, correspondingly calling functions in the 3d asset function library, and generating 3d assets. According to the method, a user does not need to have 3d modeling knowledge, and rapid generation and editing of a 3d scene can be realized through natural language description.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics, and relates to a method and system for generating a 3D scene based on natural language description. Background Art

[0002] In the era of the metaverse, 3D content is becoming more and more abundant, and 3D modeling is becoming more and more important. The technical methods of 3D modeling usually include direct modeling, procedural modeling, and AI modeling.

[0003] When creating complex 3D objects, direct modeling is suitable if individual simple shapes need to be manipulated and joined together. In this way, if modifications or additions are needed, the overall geometry of the model will not be affected. At the same time, when realistic images need to be rendered using CG objects, this modeling method is also very useful, as it can be more flexible in changing 3D objects without being constrained by any specific parameters.

[0004] Procedural modeling can create and edit models by introducing parameters. When new parameters are added, all objects will be automatically created and changed, and the characteristics and relationships of each element in the model will be determined. Usually, the modeling efficiency is higher, but the refinement of the modeling depends on the design of the parameters.

[0005] In recent years, AI modeling has made remarkable progress. In the field of computer graphics, advanced algorithms and technologies such as NeRF (Neural Radiance Fields) have been widely applied, making it easier and more efficient to generate highly realistic 3D scenes. The NeRF technology learns from 2D images through deep learning methods and generates continuous 3D scene models. Its working principle is self-supervised. By training data on a limited number of input views, high-quality renders can be generated with a relatively small dataset. Compared with the traditional methods of representing scenes using discrete grids or voxels, the continuous function representation of NeRF has advantages and can render from any angle, producing amazing high-quality renderings.

[0006] None of the above 3D modeling methods can meet the requirements of low-cost, fast, and high-quality 3D modeling. Specifically, direct modeling is too inefficient and usually takes several weeks to complete 3D modeling. If complex scenes are modeled, the time required will be even longer; although procedural modeling has higher efficiency compared to manual modeling, since a lot of manual operations are still required, such as selecting 3D assets, adjusting parameters, and adjusting the positions of 3D assets in the scene, etc., it still has relatively high requirements for the 3D knowledge of modelers; the quality of AI modeling is still relatively poor, and technically it cannot meet the commercial requirements in the short term.

[0007] Therefore, how to provide a method for automatically and quickly completing 3D scene modeling based on natural language has become an urgent problem to be solved. Summary of the Invention

[0008] In view of the above problems, the present invention provides a method and system for generating a 3D scene based on natural language description. Based on a large model, it can automatically and quickly complete 3D scene modeling, reduce the requirements for 3D modeling knowledge of modelers, and greatly improve the 3D modeling efficiency.

[0009] To achieve the above object, the present invention provides a method for generating a 3D scene based on natural language description, including:

[0010] Construct a 3D asset function library, convert 3D assets into functions controlled by parameters and save them in the 3D asset function library;

[0011] Construct a knowledge base, where the knowledge base includes the functions of each function in the 3D asset function library, usage methods, and the meanings of each parameter;

[0012] Convert the natural language description input by the user into semantic information understood by the computer based on a large model;

[0013] Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on a large model, and determine the parameter values of the functions according to the knowledge base to generate program code;

[0014] Run the program code, correspondingly call the functions in the 3D asset function library, and generate the 3D assets.

[0015] As a further improvement of the present invention, through the Python API interface of Blender, the code corresponding to each node in Blender is encapsulated into a function;

[0016] For 3D assets composed of nodes in Blender, construct a node graph through the nodes in Blender;

[0017] Convert each node into a line of Python statement, expose the parameters in the node in the Python statement, recursively traverse each node in the node graph, and finally convert the entire 3D asset into a Python function, and this Python function is the function of the 3D asset.

[0018] As a further improvement of the present invention,

[0019] For 3D assets composed of nodes not in Blender, directly import the 3D asset function library in the form of a Blender file, and control the parameters of the 3D asset through the Python API interface of Blender. The parameters include size and position.

[0020] As a further improvement of the present invention, the 3D asset function library contains examples of 3D modeling using each function.

[0021] The large model searches the 3D asset function library to understand the usage methods of each function.

[0022] As a further improvement of the present invention, for each parameter in the knowledge base, the data type and value range of the parameter are described.

[0023] The large model searches the knowledge base to infer the value of the parameter in the function.

[0024] As a further improvement of the present invention, the natural language description input by the user is converted into semantic information understood by the computer based on the large model, including:

[0025] The large model recognizes the input natural language description. The large model searches the knowledge base and selects the function corresponding to the natural language description according to the function of each function, the usage method, and the meaning of each parameter.

[0026] The large model expands the corresponding description for each parameter of the function to obtain semantic information understood by the computer.

[0027] As a further improvement of the present invention, based on the semantic information, the 3D asset to be constructed is obtained. The large model selects functions in the 3D asset function library and determines the parameter values of the functions according to the knowledge base, and generates program code, including:

[0028] The large model obtains the 3D asset and its features according to the semantic information, selects functions in the 3D asset function library respectively according to the 3D asset and its features, and infers the parameter values of each function according to the semantic information.

[0029] As a further improvement of the present invention, the program code is run, and the functions in the 3D asset function library are called correspondingly to generate the 3D asset, including:

[0030] Call the functions in the 3D asset function library. This function calls the functions of each node in Blender line by line, creates nodes in combination with the parameter values of each function, and generates a 3D asset.

[0031] As a further improvement of the present invention, for the generated 3D assets, natural language descriptions are input again, and various features of the 3D assets are further edited and adjusted through natural language interaction.

[0032] The present invention also provides a system for generating a 3D scene based on natural language descriptions, including: a 3D asset function library construction module, a knowledge base construction module, a natural language processing module, and a 3D scene generation module;

[0033] The 3D asset function library construction module is used for:

[0034] Construct a 3D asset function library, convert 3D assets into functions controlled by parameters, and save them to the 3D asset function library;

[0035] The knowledge base construction module is used for:

[0036] Construct a knowledge base, which includes the functions of each function in the 3D asset function library, the usage methods, and the meanings of each parameter;

[0037] The natural language processing module is used for:

[0038] Convert the natural language descriptions input by the user into semantic information understood by the computer based on a large model;

[0039] The 3D scene generation module is used for:

[0040] Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on a large model, and determine the parameter values of the functions according to the knowledge base to generate program code;

[0041] Run the program code, correspondingly call the functions in the 3D asset function library, and generate the 3D assets.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] With the help of the world knowledge and language understanding ability of the large model, as well as the advantages of procedural modeling, the present invention determines the parameters required for procedural modeling according to the natural language descriptions input by the user through the large model, and then obtains the Python code for the 3D scene generation task through the constructed 3D asset function library and knowledge base. By calling the Python API interface of Blender, the 3D scene modeling can be automatically completed. Compared with the prior art, it not only greatly reduces the threshold of 3D scene modeling operation for users, but also significantly improves the efficiency of 3D modeling. Description of the Drawings

[0044] Figure 1A flow chart of a method for generating a 3D scene based on a natural language description disclosed in one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a system for generating a 3D scene based on a natural language description disclosed in one embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of an application of a method and system for generating a 3D scene based on natural language description disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0048] The present invention is described in further detail below with reference to the accompanying drawings:

[0049] like Figure 1 As shown, the present invention provides a method for generating a 3D scene based on natural language description, comprising the steps of:

[0050] S1. Build a 3D asset function library, convert 3D assets into functions controlled by parameters and save them to the 3D asset function library;

[0051] in,

[0052] The purpose of building a 3D asset function library is to functionalize 3D assets, that is, convert 3D assets into Python functions controlled by parameters and save them. When you need to create the 3D asset, you only need to call the relevant function and set the parameters.

[0053] During this process, we functionalized some commonly used 3D assets in nature and indoors, and designed some necessary parameters for each function to adjust the shape, material, etc. of the assets.

[0054] Further,

[0055] In Blender, the node graph can be converted into Python code, and the corresponding function can be obtained after the code is abstracted. Therefore, through the Blender Python API interface, the code corresponding to each node in Blender is encapsulated into a function;

[0056] For 3D assets composed of nodes in Blender, for example, to generate a tree, it needs to be divided into three steps: The first step is to construct the skeleton of the tree, starting from a root node and using a random walk algorithm for tree growth; the second step is to cover the tree with an epidermis and add bark material; the third step is to place the branches and leaves, with their directions and colors having a certain degree of randomness to ensure that each tree is not exactly the same. We can construct a node graph through the nodes in Blender; convert each node into a line of Python statement, expose the parameters in the node in the Python statement, recursively traverse each node in the node graph, and finally convert the entire 3D asset into a Python function, and this Python function is the function of the 3D asset; when it is necessary to generate this 3D asset, running the function of this 3D asset can call the functions of each node in Blender line by line to create each node, realizing the creation and property modification of the 3D asset.

[0057] For 3D assets composed of nodes not in Blender, directly import the 3D asset function library in the form of calling a Blender file (.blend file), and control the parameters of the 3D asset through the Python API interface of Blender. The parameters include: size, position.

[0058] Specifically,

[0059] For example, for a tree asset composed entirely of Blender nodes, when generating this tree asset, after converting it into the add_tree() function, in addition to the conventional control of the tree's size, parameters such as leaf_type, flower_type, fruit_type, trunk_surface, etc. can also be edited through the Python API interface of Blender to precisely control the leaves, flowers, fruits, and trunks on the tree.

[0060] S2. Build a knowledge base, which includes the functions of each function in the 3D asset function library, usage methods, and the meanings of each parameter;

[0061] Among them,

[0062] The 3D asset function library contains some examples of using each function for 3D modeling; the large model (LLM) searches the 3D asset function library to understand the usage methods of each function, and then it can know how to select the appropriate function.

[0063] Furthermore,

[0064] For each parameter in the knowledge base, describe the data type and value range of the parameter; the large model searches the knowledge base, can understand all the parameters of each 3D asset function in detail, then determines the function parameters in the 3D scene generation module, sets the values of these parameters, and calls the function to generate the 3D asset. For example: to generate a tree, the large model first selects the add_tree() function, then determines the values of parameters such as the height, width, branch distribution, and leaf color of the tree through the description of the tree, and finally calls the function to generate a tree.

[0065] Specifically,

[0066] Taking the generation of a tree in a 3D scene as an example, after converting the tree asset into the add_tree() function, the appearance of trees, the types of trees, and the density of trees in the scene can be controlled respectively through trees_chance, max_tree_species, and tree_density.

[0067] In addition, by setting the parameters in the add_tree() function, the style of each tree asset can also be precisely controlled. The knowledge base will not only describe the meaning of each parameter in detail, but also describe the data type, value range, and one or more examples of the parameter, as shown in Table 1 specifically:

[0068] Table 1

[0069]

[0070] S3. Convert the natural language description input by the user into semantic information understood by the computer based on the large model;

[0071] Among them,

[0072] First, the user inputs a natural language description for the pre-generated 3D scene on the user operation interface. The description can be detailed or simple, but it is recommended that the user describe it as detailed as possible; then click to create a 3D scene;

[0073] Secondly,

[0074] The large model recognizes the input natural language description. The large model searches the knowledge base and selects the function corresponding to the natural language description according to the function, usage method of each function, and the meaning of each parameter;

[0075] The large model expands the corresponding description for each parameter of the function to obtain semantic information understood by the computer.

[0076] Specifically,

[0077] Generally, the descriptions provided by users are incomplete for generating 3D scenes. This module mainly uses large models to expand the text input by users, describing the scene in more detail. Since the large model knows that the add_tree() function is used to generate trees and knows what its parameters are and what each parameter corresponds to, when expanding the text, the large model will focus on expanding the descriptions corresponding to the parameters so as to obtain the values of these parameters, which is convenient for subsequent procedural generation. The reason why this can be done mainly depends on the rich world knowledge possessed by general large models and their ability to understand natural language well. The large model in the present invention can adopt GPT4, ERNIE Bot or similar general large models.

[0078] For example, when the user inputs a simple description of the scene: "There are mountains on both sides of a river", the large model will expand this description: "This is a charming landscape scenic area, located on the edge of a vast city. A broad and winding river steadily runs through the center, and on both sides are lush, towering mountains, adding various vitality and vigor here. This river is not an ordinary river. Its width is sufficient for three or four yachts to sail side by side without interfering with each other. Even in the dry season, it still appears majestic, with deep and continuous water flow. When the flood is strong in summer, the river proudly churns with waves, almost submerging the riverbanks. In spring, the new green in the mountains is as fresh as if washed, with a vivid green color that is almost dizzying. In summer, the numerous leaves in the mountains sparkle in the sun, like a green ocean. In autumn, the leaves turn golden, and the mountain tops seem to be covered with a golden blanket. In winter, a pure white snow covers the mountains, and only the branches and leaves of the trees still have a trace of color. The mountains on both sides of the river have different shapes, scattered in height, like a mysterious melody, leading people into this wonderful world. Large tracts of coniferous and broad-leaved forests grow freely on the slopes, with tall trees and lush leaves, playing a symphony belonging to nature and nurturing a rich ecosystem. Birds soar freely in the sky, swans swim quietly in the river, squirrels jump happily among the trees, and deer play in the forest. All living beings are enjoying the gift bestowed by nature. No matter from which angle you look at this river, you can feel its vitality, the beauty and magic of nature. This is a paradise for life, a gift from nature, and a haven that people yearn for." The expanded statement describes more details and is more convenient for inferring model parameters.

[0079] S4. Obtain the 3D assets to be constructed according to the semantic information, select functions from the 3D asset function library based on the large model, and determine the parameter values of the functions according to the knowledge base to generate program code;

[0080] Among them,

[0081] The large model obtains 3D assets and features according to semantic information, selects functions in the 3D asset function library respectively according to the 3D assets and features, and infers the parameter values of each function according to the semantic information.

[0082] Furthermore,

[0083] According to the expansion of the 3D scene, the large model can easily infer the parameter values of the functions, so it can directly generate Python code.

[0084] S5. Run the program code, call the functions in the 3D asset function library correspondingly, and generate 3D assets.

[0085] Among them,

[0086] Call the functions in the 3D asset function library. This function calls the functions of each node in Blender line by line, creates nodes in combination with the parameter values of each function, and generates 3D assets.

[0087] S6. For the generated 3D assets, input the natural language description again, and further edit and adjust the features of the 3D assets through natural language interaction.

[0088] Among them,

[0089] Such as: adjusting the size, changing the material, etc.;

[0090] Taking the generation of trees in the scene as an example, in the above-mentioned task of constructing a river scene described by the user, the large model can, according to the content of its expansion, know that trees need to be created and what the density of the trees should be like, so relevant parameters are set, such as trees_chance = 1, tree_density = 0.05, etc. If the user hopes to increase the density of the trees and make the trees bloom, the generated 3D model can be modified through natural language interaction. At this time, the large model can automatically modify the parameters to: tree_density = 0.07, Tree.flower_type = 'flower', realize the adjustment of 3D assets, and present the final 3D assets in the 3D scene.

[0091] Such as Figure 2 , 3 As shown, the present invention also provides a system for generating a 3D scene based on a natural language description, including: a 3D asset function library construction module, a knowledge base construction module, a user interface module, a natural language processing module, and a 3D scene generation module;

[0092] The 3D asset function library construction module is used for:

[0093] Construct a 3D asset function library, convert 3D assets into functions controlled by parameters, and save them to the 3D asset function library;

[0094] A knowledge base construction module, for:

[0095] Construct a knowledge base, which includes the functions, usage methods of each function in the 3D asset function library, and the meanings of each parameter;

[0096] A user interface module, for:

[0097] Allow users to input natural language descriptions to create and view the generated 3D scenes, and at the same time can modify the 3D scenes by inputting natural language descriptions;

[0098] A natural language processing module, for:

[0099] Receive the natural language input of the user, understand the user's intention, and convert the natural language description input by the user into semantic information understood by the computer based on a large model;

[0100] A 3D scene generation module, for:

[0101] Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on a large model, and determine the parameter values of the functions according to the knowledge base to generate program code;

[0102] Run the program code, correspondingly call the functions in the 3D asset function library, and generate 3D assets.

[0103] Embodiment:

[0104] As Figure 3 shown, applying the method and system for generating a 3D scene based on natural language description of the present invention to complete the generation of 3D assets, the process includes:

[0105] Step 1: The user inputs a natural language description in the user interface module, specifically: a dense forest;

[0106] Step 2: After receiving the natural language description, the natural language processing module uses the large model LLM to expand the user input "a dense forest", and describes in detail the appearance of the forest, the distribution density of the trees, the current season, the color of the leaves, etc.;

[0107] Step 3: The 3D scene generation module determines the involved 3D assets according to the expanded content, searches for the desired functions in the 3D asset function library, and uses the large model LLM to determine the function parameters to obtain Python code;

[0108] Step 4: The 3D scene generation module calls the Python API of Blender to generate 3D assets.

[0109] Advantages of the present invention:

[0110] With the help of the world knowledge and language understanding ability of the large model, as well as the advantages of procedural modeling, the large model determines the parameters required for procedural modeling according to the natural language description input by the user, and then obtains the Python code for the 3D scene generation task through the constructed 3D asset function library and knowledge base. By calling the Python API interface of Blender, the 3D scene modeling can be automatically completed. Compared with the prior art, it not only greatly reduces the threshold for users of 3D scene modeling operations, but also significantly improves the efficiency of 3D modeling.

[0111] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for generating a 3D scene based on natural language description, characterized in that, including: Construct a 3D asset function library, convert 3D assets into functions controlled by parameters, and save them to the 3D asset function library; Construct a knowledge base, which includes the functions of each function in the 3D asset function library, the usage methods, and the meanings of each parameter; Convert the natural language description input by the user into semantic information understood by the computer based on a large model; Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on the large model, and determine the parameter values of the functions according to the knowledge base to generate program code; Run the program code, correspondingly call the functions in the 3D asset function library, and generate the 3D assets.

2. The method for generating a 3D scene based on a natural language description according to claim 1, wherein: Through the python api interface of blender, encapsulate the code corresponding to each node in blender into a function; For 3D assets composed of nodes in blender, construct a node graph through the nodes in blender; Convert each node into a line of python statement, expose the parameters in the node in the python statement, recursively traverse each node in the node graph, and finally convert the entire 3D asset into a python function, and this python function is the function of the 3D asset.

3. The method for generating a 3D scene based on a natural language description according to claim 2, wherein: For 3D assets composed of nodes other than those in blender, directly import the 3D asset function library in the form of a blender file, and control the parameters of the 3D asset through the python api interface of blender, and the parameters include: size, position.

4. The method for generating a 3D scene based on a natural language description according to claim 1, characterized in that: The 3D asset function library contains examples of using each function for 3D modeling; The large model searches the 3D asset function library to understand the usage methods of each function.

5. The method for generating a 3D scene based on a natural language description according to claim 1, wherein: The knowledge base describes the data types and value ranges of the parameters corresponding to each parameter; The large model searches the knowledge base to infer the values of the parameters in the function.

6. The method for generating a 3D scene based on a natural language description according to claim 1, wherein: Convert the natural language description input by the user into semantic information understood by the computer based on a large model; including: The large model recognizes the input natural language description, the large model searches the knowledge base, and selects the function corresponding to the natural language description according to the function of each function, the usage method, and the meaning of each parameter; The large model expands the corresponding description for each parameter of the function to obtain semantic information understood by the computer.

7. The method for generating a 3D scene based on a natural language description according to claim 1, characterized in that: Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on the large model, and determine the parameter values of the functions according to the knowledge base to generate program code; including: The large model obtains 3D assets and features according to the semantic information, selects functions in the 3D asset function library respectively according to the 3D assets and features, and infers the parameter values of each function according to the semantic information.

8. The method for generating a 3D scene based on a natural language description according to claim 1, wherein: Run the program code, correspondingly call the functions in the 3D asset function library, and generate the 3D assets; including: Call the functions in the 3D asset function library. These functions call the functions of each node in Blender line by line, and create nodes by combining the parameter values of each function to generate 3D assets.

9. The method for generating a 3D scene based on a natural language description according to claim 1, wherein: For the generated 3D assets, input the natural language description again, and further edit and adjust the characteristics of the 3D assets through natural language interaction.

10. A system for generating a 3D scene based on a natural language description, implementing the method according to any one of claims 1 to 9, characterized in that: It includes: A 3D asset function library construction module, a knowledge base construction module, a natural language processing module, and a 3D scene generation module; The 3D asset function library construction module is used for: Construct a 3D asset function library, convert 3D assets into functions controlled by parameters, and save them to the 3D asset function library; The knowledge base construction module is used for: Construct a knowledge base, where the knowledge base includes the functions of each function in the 3D asset function library, usage methods, and the meanings of each parameter; The natural language processing module is used for: Convert the natural language description input by the user into semantic information understood by a computer based on a large model; The 3D scene generation module is used for: Obtain the 3D assets to be constructed according to the semantic information, select functions in the 3D asset function library based on a large model, and determine the parameter values of the functions according to the knowledge base to generate program code; Run the program code, correspondingly call the functions in the 3D asset function library, and generate the 3D assets.

Images