Scene generation method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a scene generation method and device, electronic equipment, a storage medium and a computer program product. The scene generation method comprises the steps of obtaining a description text for a target scene; a first plan is determined through a scheduling agent based on the description text, and the first plan comprises an object used for generating the target scene; on the basis of the object and a predefined knowledge document corresponding to the object in a preset resource library, a second plan containing subtasks is determined through a professional agent, and the subtasks correspond to the predefined knowledge document; based on the subtasks in the second plan, performing retrieval in a preset resource library through a retrieval agent to obtain objects or object operation interfaces corresponding to the subtasks; and based on the object or the object operation interface corresponding to the subtask, generating a target scene through an execution agent. The problem of poor scene generation effect is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a scene generation method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the rapid development of generative modeling technology, the automatic generation of 3D content has achieved remarkable results, covering the fields of 3D objects, virtual images and scenes, and has been widely used in the generation of 3D scenes, games, animation, movies and virtual reality. Programmatic content generation technology is an advanced technology for quickly creating high-quality 3D assets. It can generate realistic and complex 3D assets in natural scenes through adjustable parameters and rule-based systems. Methods in related technologies such as 3D-GPT and SceneCraft introduced an instruction-driven 3D modeling method, and successfully cooperated with human designers to establish a programmatic generation framework by integrating the LLM (Large Language Model) agent with the programmatic generation software.

[0003] In the related technologies, for learning-based content generation methods, due to representation methods that are incompatible with industrial processes, such as the use of 3D primitives (such as point clouds or radiation fields), there is a significant gap between them and industrial-level applications. These learning-based methods have no geometric structure and are therefore difficult to directly apply to virtual engines such as UE; for procedural generation methods, the generated 3D data is restricted by rules and has a certain degree of deviation from the real world, which severely limits flexibility and practicality, and users cannot customize the generated results; for instruction-driven 3D modeling methods, there are some significant limitations, including the inability to achieve fine 3D asset editing, the inability to make full use of existing procedural generation resources, the inability to generate high-quality and large-scale scenes, and the limited diversity of assets due to restrictions imposed by prompt expressions.

[0004] Therefore, the industry is eager to have a 3D scene generation method that can automatically generate high-quality 3D scenes, is commercially available, and can be freely edited. Summary of the invention

[0005] The scene generation method, device, electronic device, storage medium, and computer program product provided by the exemplary embodiments of the present disclosure can at least solve the above technical problems and other technical problems not mentioned above.

[0006] According to one aspect of the present disclosure, a scenario generation method is provided, the scenario generation method comprising: obtaining a description text for a target scenario; determining a first plan through a scheduling agent based on the description text, wherein the first plan includes an object for generating the target scenario; determining a second plan including subtasks through a professional agent based on the object and a predefined knowledge document corresponding to the object in a preset resource library, wherein the subtasks correspond to the predefined knowledge document; based on the subtasks in the second plan, searching in the preset resource library through a retrieval agent to obtain objects or object operation interfaces corresponding to the subtasks; and generating the target scenario through an execution agent based on the objects or object operation interfaces corresponding to the subtasks.

[0007] Optionally, the scheduling agent, the professional agent, and the retrieval agent are large language models; wherein the scenario generation method further includes: for the current tasks to be processed by the scheduling agent, the professional agent, the retrieval agent, and the execution agent respectively, prompting the large language models serving as the scheduling agent, the professional agent, the retrieval agent, and the execution agent through a preset prompt template corresponding to the current task, wherein the preset prompt template includes at least one of the following items: role information of the current agent, task objectives and constraints of the current task, knowledge related to the current task, output format of the current agent, and reference examples for the current task.

[0008] Optionally, the subtask in the second plan is searched in the preset resource library through a retrieval agent, including: instructing the retrieval agent to output a retrieval description through the preset prompt template corresponding to the subtask, wherein the retrieval description is used to describe the object or object operation interface corresponding to the subtask; and searching the preset resource library based on the embedded vector corresponding to the retrieval description.

[0009] Optionally, determining a first plan through a scheduling agent based on the description text includes: extracting a first object from the description text through the scheduling agent; inferring a second object related to the target scene through the scheduling agent to determine a first plan, wherein the first plan includes the first object and the second object.

[0010] Optionally, generating the target scene through the execution agent includes: when the object operation interface is retrieved, modifying predefined parameters of the object operation interface based on the description text; and generating the target scene through the execution agent based on the object operation interface after the predefined parameters are modified.

[0011] Optionally, modifying predefined parameters of the object operation interface based on the description text includes: modifying predefined parameters of the object operation interface regarding object style based on the description text, wherein the object style includes at least one of the following items: object material properties, object geometric structure.

[0012] Optionally, the second plan includes at least one of the subtasks, and the predefined knowledge document includes constraints when combining multiple subtasks, and the constraints include at least one of the following items: restrictions on the combination method, and the order of use of the combined basic units.

[0013] According to another aspect of the present disclosure, a scene generation device is also provided, and the scene generation device includes: a description acquisition unit, configured to: acquire a description text for a target scene; a first planning unit, configured to: determine a first plan through a scheduling agent based on the description text, wherein the first plan includes an object for generating the target scene; a second planning unit, configured to: determine a second plan including subtasks through a professional agent based on the object and a predefined knowledge document corresponding to the object in a preset resource library, wherein the subtask corresponds to the predefined knowledge document; an asset retrieval unit, configured to: based on the subtasks in the second plan, search in the preset resource library through a retrieval agent to obtain an object or object operation interface corresponding to the subtask; and a scene generation unit, configured to: generate the target scene through an execution agent based on the object or object operation interface corresponding to the subtask.

[0014] According to another aspect of an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute any of the scene generation methods described above.

[0015] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is also provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute any of the scene generation methods described above.

[0016] According to another aspect of an embodiment of the present disclosure, a system is also provided, comprising at least one computing device and at least one storage device storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute any of the scene generation methods described above.

[0017] According to another aspect of an embodiment of the present disclosure, a computer program product is further provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the scene generation method as described in any one of the above is implemented.

[0018] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0019] According to the scene generation method, device, electronic device, storage medium and computer program product disclosed in the present invention, it is possible to divide the scene generation task into a programmatic generation framework consisting of a task planning stage, a resource retrieval stage, and an action execution stage through multiple agents. Combined with a resource library including rich object resources, object operation interface resources, documents and other programmatic modeling resources, high-quality scene generation effects can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 A flowchart showing a scene generation method in an exemplary embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram showing a tree object and a corresponding part of an operation API and a knowledge document in an exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram showing a process of performing retrieval by pre-trained CLIP in an exemplary embodiment of the present disclosure;

[0024] Figure 4 , Figure 5 , Figure 6 , Figure 7 A schematic diagram showing a preset prompt template in an exemplary embodiment of the present disclosure;

[0025] Figure 8 A schematic diagram showing a 3D scene generation result including materials and a related PCG in an exemplary embodiment of the present disclosure;

[0026] Fig. 9 A schematic diagram showing generation and modification of a scene in an exemplary embodiment of the present disclosure;

[0027] Fig.10 A schematic diagram showing scene generation by PCGPlanner and PCGHub in an exemplary embodiment of the present disclosure;

[0028] Fig.11A schematic diagram showing a flow chart of scene generation performed by the PCGPlanner framework in an exemplary embodiment of the present disclosure;

[0029] Fig.12 A flowchart showing a method for performing scene generation by a programmatic generation method in an exemplary embodiment of the present disclosure is shown;

[0030] Fig.13 A schematic diagram showing visualization results of the generation quality of large-scale natural and urban scenes in an exemplary embodiment of the present disclosure;

[0031] Fig.14 A schematic diagram showing the generation effect of other scenes in an exemplary embodiment of the present disclosure;

[0032] Fig.15 A schematic diagram showing a comparison of the effects of city scene generation between the related technology and the exemplary embodiment of the present disclosure;

[0033] Fig.16 A block diagram showing a scene generation device in an exemplary embodiment of the present disclosure is shown;

[0034] Fig.17 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0037] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0038] In order to solve the above problems, the present disclosure provides a scene generation method, device, electronic device, storage medium and computer program product, which can divide the scene generation task into a programmatic generation framework consisting of a task planning stage, a resource retrieval stage, and an action execution stage through multiple agents, and combined with a resource library including rich object resources, object operation interface resources, documents and other programmatic modeling resources, high-quality scene generation effects can be obtained.

[0039] Next, we will refer to Figures 1 to 17 The scene generation method, device, electronic device, storage medium and computer program product of the present disclosure are described in detail.

[0040] Figure 1 A flowchart of a scene generation method in an exemplary embodiment of the present disclosure is shown.

[0041] Reference Figure 1 , in step 101, a description text for a target scene is obtained.

[0042] According to an exemplary embodiment of the present disclosure, a description text of a user for a target scene may be obtained. For example, the description text may be similar to the following: “generate a modern city”, “generate an autumn forest”, etc.

[0043] In step 102 , a first plan is determined by a scheduling agent based on the description text, wherein the first plan includes objects for generating a target scene.

[0044] According to an exemplary embodiment of the present disclosure, a user's description text of a target scene can be received through a dispatch agent, and information can be extracted from the description text through the dispatch agent. The dispatch agent can act as a planner that can interpret the user's intention (embodied in the description text) as a rough plan (first plan) and can list all objects (referring to 3D models of objects used for scene generation, which can also be called assets) mentioned in the input instructions (embodied in the description text).

[0045] According to an exemplary embodiment of the present disclosure, determining a first plan through a scheduling agent based on a description text may include but is not limited to the following steps: extracting a first object from the description text through a scheduling agent; inferring a second object related to a target scene through a scheduling agent to determine a first plan, wherein the first plan includes the first object and the second object.

[0046] According to an exemplary embodiment of the present disclosure, the user may omit some common background objects in the description text, and the scheduling agent can automatically add these omitted objects to the rough plan (first plan). This part can be encapsulated as an automatic selection module, which can be used to automatically infer and select appropriate parameters according to specified rules and logic when the description information is insufficient. For example, it can automatically select feasible parameter options based on other known parameters to complete the operation execution and complete the scene generation task through the object operation interface (operation API, which can also be referred to as operation) of the supplementary parameters, solving the problem of insufficient description information. .

[0047] In step 103, based on the object and the predefined knowledge document corresponding to the object in the preset resource library, a second plan including subtasks is determined by a professional agent, wherein the subtasks correspond to the predefined knowledge document.

[0048] According to an exemplary embodiment of the present disclosure, a further detailed plan (second plan) can be formulated by a specialist agent. At this stage, the specialist agent can act as a professional designer and can provide a detailed plan for the objects given in the rough plan. The specialist agent can decompose each rough plan into a detailed plan containing multiple subtasks, each of which has a corresponding executable object operation interface (API) or an object to be imported.

[0049] According to an exemplary embodiment of the present disclosure, for the construction of a preset resource library (PCGBench, or PCGHub), the present disclosure collects a large number of programmatic assets and high-quality 3D static assets and other objects. The categories of these objects cover aspects such as natural environment, architecture, biology, environmental effects, and science fiction elements. All programmatic assets can be made through Blender nodes, and an unlimited number of programmatic assets can be generated by adjusting the parameters of geometry nodes and shader nodes.

[0050] These programmatic assets can be encapsulated as Python APIs (object operation interfaces, or operation APIs), and relevant documentation can be made so that they can be called through a large language model as each agent.

[0051] For example, 135 APIs with high-level interpretability (such as the number of floors of a building) can be extracted from 1,532 original node APIs to select APIs with obvious visualization effects. By testing these selected APIs, the functional description of each API, the detailed description of each parameter, the API type, and the API name can be recorded to obtain the predefined knowledge document corresponding to each API.

[0052] The knowledge document module can be used to store the knowledge document of each operation API, and the knowledge document can be passed to the dedicated agent to prompt the dedicated agent's scheduled tasks. The knowledge document can be recorded in JSON format, and the knowledge document corresponding to each API can be set to the same architecture.

[0053] For example, a knowledge document in JSON format in a knowledge document module may have the following structure: a type field indicating the type of object generated; a description field describing the function; a function API field specifying the operation API name; and a parameter information field listing possible parameter values ​​and their detailed descriptions.

[0054] The following diagrams show examples of operating APIs, knowledge documents, and objects.

[0055] Figure 2 A schematic diagram showing a tree object and a corresponding part of an operation API and a knowledge document in an exemplary embodiment of the present disclosure.

[0056] Reference Figure 2 For example, a tree object may correspond to operation APIs such as tree_mesh_generate and tree_material_generate, and the above operation APIs may correspond to knowledge documents such as tree_mesh_doc and tree_material_doc. Specifically, the knowledge document tree_mesh_doc of the tree_mesh_generate API may contain the following information: {"type":"tree mesh","Description":"Function for generating treesmesh.","Function API":"tree_mesh_generate","Parameter Information":{"Possiblevalues":("High","Medium","Low"),...}}.

[0057] In addition, the knowledge documentation process can be accelerated by developing relevant tools to improve the efficiency and accuracy of document generation.

[0058] For a scene generation task, it may be necessary to call multiple operation APIs from different modules. Different operation APIs are likely to be called in sequence. Therefore, the correct functional arrangement between different modules and operation APIs is crucial. In order to ensure that it can handle a wide range of scene generation tasks and overcome the limitations of tokens, the two-stage iterative PCGPlanner proposed in this disclosure can be composed of a scheduling agent and a professional agent.

[0059] In terms of procedural constraints, the application of procedural methods is often limited by specific prerequisites. For example, generating tree materials requires that the tree mesh already exists, and placing street lights on both sides of the road requires the road to be generated first. This shows that all procedural methods cannot be freely combined without restrictions, and they need to be reasonably combined according to the limitations of these methods. To solve this problem, a series of subtasks that may be encountered in scene generation can be defined, and detailed usage documentation can be compiled for each subtask to clarify potential limitations. This usage documentation can be implemented through predefined knowledge documents.

[0060] According to an exemplary embodiment of the present disclosure, the second plan may include at least one subtask, and the predefined knowledge document may include constraints when combining multiple subtasks, and the constraints may include at least one of the following items: restrictions on the combination method, the order of use of the combined basic units, etc.

[0061] According to the exemplary embodiments of the present disclosure, in actual operation, fully functional programmatic generation methods (such as generating complete trees, etc.) can be preferentially used as combined basic units (objects), rather than gradually building and combining objects from basic elements (such as separate trunks and leaves, etc.) to generate 3D models in the scene. By providing a knowledge document containing constraints rather than direct restrictions, the threshold for integrating new programmatic methods can be lowered, and the flexibility of the large language model (LLM) in driving the programmatic method and the diversity of generated results are enhanced.

[0062] Return to reference Figure 1 In step 104, based on the subtask in the second plan, a search is performed in a preset resource library through a search agent to obtain an object or an object operation interface corresponding to the subtask.

[0063] According to an exemplary embodiment of the present disclosure, according to the detailed plan (second plan) given by the professional agent, a suitable operation API can be searched in PCGHub to execute or import an object. In order to accurately perform retrieval, the present disclosure proposes a retrieval agent that can identify whether the target of a subtask type is an object or an operation API.

[0064] According to an exemplary embodiment of the present disclosure, the operation API may be retrieved by executing the agent module.

[0065] According to an exemplary embodiment of the present disclosure, based on the subtasks in the second plan, searching in a preset resource library is performed through a retrieval agent, which may include but is not limited to the following steps: instructing the retrieval agent to output a retrieval description through a preset prompt template corresponding to the subtask, wherein the retrieval description is used to describe the object or object operation interface corresponding to the subtask; searching in the preset resource library based on the embedded vector corresponding to the retrieval description.

[0066] According to an exemplary embodiment of the present disclosure, through a preset prompt template for pre-retrieval prompt, the retrieval agent can be instructed to output a retrieval description as a search index of an object or operation API. The retrieval description can be encoded into an embedded vector for retrieval.

[0067] According to an exemplary embodiment of the present disclosure, pre-trained CLIP (Contrastive Language-Image Pre-training) may be used to search for operation APIs through text-to-text retrieval, and to search for objects such as 3D models through text-to-image retrieval.

[0068] Figure 3 A schematic diagram showing a process of performing retrieval through a pre-trained CLIP in an exemplary embodiment of the present disclosure.

[0069] Reference Figure 3 According to an exemplary embodiment of the present disclosure, the rendered image 301 of each object in the PCGHub of the present disclosure can be encoded into a normalized 768-dimensional vector, i.e., an image embedding, by a pre-trained CLIP, and then can be compared with the text embedding of the input description 302. One of the five results in the image embedding that are most similar to the text embedding can be randomly selected as a retrieval result and imported into the Blender scene. For example, the best retrieval matching result can be selected by calculating the cosine distance between the embedding vectors as a similarity evaluation.

[0070] Return to reference Figure 1 , in step 105, based on the object or object operation interface corresponding to the subtask, a target scene is generated by executing the agent.

[0071] According to an exemplary embodiment of the present disclosure, in order to simulate the complex arrangement and layout logic in nature, a wide range of Scatter Generators with various functions are collected. These generators have the following functions: (1) Diversified distribution patterns, supporting the layout of objects of various statistical distribution types, including Poisson distribution, uniform distribution, normal distribution, and exponential distribution; (2) Terrain-adaptive distribution, selective distribution based on factors such as terrain height, slope, and edge width to adapt to different terrain conditions; (3) Logical distribution, supporting distribution patterns based on specific logic, such as arranging plants outside buildings on flat ground instead of inside, or arranging street lights on both sides of the road, etc.

[0072] According to an exemplary embodiment of the present disclosure, in order to realize the interactive workflow in Blender, the present disclosure deploys an executor in the Blender environment. All agents can send action execution commands to the executor through the local backend server. The present disclosure uses structured encapsulation to convert PCG plug-ins into executable action functions. This allows the executor to flexibly initialize all actions. In order to allow each agent to accurately control the action, each action function (operation API) can be structured and recorded in JSON format. For example, you can refer to Figure 2 The action functions are structured and recorded in JSON format above.

[0073] According to an exemplary embodiment of the present disclosure, in the action execution phase, each operation API in PCGHub can be defined in the form of a Python function and can have multiple predefined parameters. For example, Figure 2 As shown in the middle. In the case where parameters need to be selected to adapt to the task corresponding to the current description text, the corresponding preset prompt template can be used to instruct the execution agent to automatically select a feasible parameter from the options based on other parameters. The executor can finally use the inferred parameter execution operation API in the Blender Python environment to gradually add new objects and modify scene items until all tasks are completed.

[0074] According to an exemplary embodiment of the present disclosure, the scheduling agent, professional agent, retrieval agent, and execution agent mentioned in the present disclosure can all be large language models; wherein, the scenario generation method may also include but is not limited to the following steps: for the current tasks to be processed by the scheduling agent, professional agent, retrieval agent, and execution agent respectively, prompting the large language model serving as the scheduling agent, professional agent, retrieval agent, and execution agent through a preset prompt template corresponding to the current task, wherein the preset prompt template may include at least one of the following items: role information of the current agent, task objectives and constraints of the current task, knowledge related to the current task, output format of the current agent, and reference examples for the current task.

[0075] According to an exemplary embodiment of the present disclosure, in order to realize the object (asset) generation driven by a large language model (LLM), the PCGPlanner agent hierarchy of the present disclosure can be introduced. The hierarchy can be composed of multiple agents, such as scheduling agents, professional agents, retrieval agents, execution agents, etc., to automatically generate high-quality, large-scale scenes based on the user's description text.

[0076] The PCGPlanner of the present disclosure may mainly include three main stages: task planning, asset retrieval, and operation execution. In the task planning stage, an agent for sequential planning may be introduced to process user queries (description texts) and output a detailed list of subtasks, such as a scheduling agent and a professional agent. Then, in order to accurately connect the subtasks with the preset resource library of the present disclosure, the object operation interface and objects mentioned in the task list may be retrieved from the preset resource library through a retrieval agent. Finally, the subtasks of program generation or object operation, etc. may be processed by an execution agent using an annotated operation API.

[0077] In order to enable LLM to have the ability to model scenarios like an expert, the present disclosure may introduce a system prompt template (preset prompt template) to prompt the large language model as each agent to complete the subtasks in the scenario modeling task. For each agent, the system prompt template may have a similar structure, for example, the system prompt template may include at least one of the following items: Role, Task, Document, Format, Examples.

[0078] Figure 4 , Figure 5 , Figure 6 , Figure 7 A schematic diagram showing a preset prompt template in an exemplary embodiment of the present disclosure.

[0079] See also Figure 4 , Figure 5 , Figure 6 , Figure 7 According to an exemplary embodiment of the present disclosure, the meanings of each item in the preset prompt template may be as follows:

[0080] Role: Each agent is assigned a specific role that describes its responsibilities in the scenario generation process;

[0081] Task: Explains the objectives of the task in detail, and also explains the various constraints that will be encountered when performing these tasks;

[0082] Document: At each step, the agent receives a knowledge document related to the task, which can be used as a knowledge reserve;

[0083] Format: represents the output format of each agent. In order to accurately transmit information between agents, the output format of each agent can be strictly defined;

[0084] Examples: To help agents follow the format, you can display multiple examples in each prompt for agents to refer to.

[0085] See also Figure 4 , Figure 5 , Figure 7 When it is necessary to select parameters in the operation API, a code example can also be introduced in the preset prompt template, so that the execution agent can be instructed to automatically select a feasible parameter through the corresponding preset prompt template to obtain an executable operation API.

[0086] According to an exemplary embodiment of the present disclosure, the following table shows the indicator results of ablation experiments on different prompt components in a preset prompt template for a generated tree object, taking the indicators ER (Executability Rate) and SR (Success Rate) as examples.

[0087]

[0088] According to an exemplary embodiment of the present disclosure, the following table shows the performance of different large language models (e.g., Llama2-7B, Llama2-13B, Mistral, Gemma-2B, Gemma-7B, GPT-3.5-turbo, GPT-4, etc.) in natural scene generation and urban scene generation.

[0089]

[0090] According to an exemplary embodiment of the present disclosure, generating a target scene by executing an agent may include but is not limited to the following steps: when an object operation interface is retrieved, modifying the predefined parameters of the object operation interface based on the description text; generating a target scene by executing the agent based on the object operation interface after the predefined parameters are modified.

[0091] According to an exemplary embodiment of the present disclosure, a Python function for each operation API can be defined in PCGHub, which can have multiple predefined parameters. After obtaining the operation API, the encapsulated operation execution module can customize these initial parameters according to the description text of the task, so as to generate a target scene through the execution agent based on the object operation interface after the predefined parameters are modified.

[0092] According to an exemplary embodiment of the present disclosure, the operation execution module may further include a parameter definition module for accurately defining and interpreting each parameter of the operation API to ensure the accuracy and completeness of the parameters.

[0093] According to an exemplary embodiment of the present disclosure, the agent retrieval operation API (α) can be executed, and the preset prompt template (P specialist ) and subtasks (scenario items sj), the retrieved operation API is customized to generate a modified version α* of the operation API, which is expressed as: α*←Agent retrieval (α,P specialist ,s j ).

[0094] The following table shows the pseudo code of the sceneχ algorithm workflow of the scene generation method used in the exemplary embodiment of the present disclosure.

[0095]

[0096]

[0097] Referring to the above table, we can record the user query as q, the knowledge document as D, and the API collection as The operation API retrieved from is α, the model (object) retrieved from the asset (object) set Γ is γ, and P i represents the prompt of each agent, P dispatch represents the preset prompt template corresponding to the scheduling agent, P specialist Indicates the preset prompt template corresponding to the professional agent, P retrieval Indicates the preset prompt template corresponding to the retrieval agent, End_flag = False.

[0098] The output is the modified operation API α*.

[0099] When a user query is received, the multi-agent framework starts processing; if the end condition is not reached, the agent can be dispatched first. dispatch In the corresponding preset prompt template P dispatch Under the prompt, the input q is disassembled into a rough plan. For each object in the rough plan, a professional agent can be used to specialist In the corresponding preset prompt template P specialist Under the prompt, make a detailed plan for each object. For each subtask in the detailed plan, you can retrieve the agent retrieval In the corresponding preset prompt template P retrieval , retrieve the operation API α or model γ in the preset resource library through CLIP; when the operation API α is retrieved, you can retrieve it through the retrieval agent Agent retrieval In the preset prompt template P specialist , for the current subtask s j Modify the subtask s j The corresponding operation API α*, otherwise, when the retrieved model is model γ, directly import the model γ into the scene project; when all subtasks in the detailed plan are completed, the execution of this algorithm ends.

[0100] According to an exemplary embodiment of the present disclosure, modifying the predefined parameters of the object operation interface based on the description text may include but is not limited to the following steps: modifying the predefined parameters of the object operation interface regarding the object style based on the description text, wherein the object style includes at least one of the following items: object material properties, object geometry.

[0101] According to exemplary embodiments of the present disclosure, in terms of stylization methods, the present disclosure proposes two methods for stylizing 3D objects.

[0102] The first method is to adjust the material properties of an object (such as reflectivity, roughness, and transparency) to achieve a specific visual effect through procedural material generation technology; for example, this method can be used to simulate the texture of a muddy ground or a metallic glossy surface.

[0103] Figure 8 A schematic diagram showing a 3D scene generation result including materials and a related PCG in an exemplary embodiment of the present disclosure.

[0104] Reference Figure 8 , the textures of forests, bushes, grass, rivers, etc. in various scenes can be simulated through procedural material generation technology.

[0105] The second method is to achieve stylization by modifying the geometric structure of the object, including but not limited to adjusting the geometric data of vertices, edges and faces to change the appearance of the object; for example, this method can be used to simulate rotten trees and leaves with water droplets.

[0106] Combining these two methods can significantly enrich the visual expression of 3D objects and provide diverse style effects.

[0107] According to an exemplary embodiment of the present disclosure, a Blender Python environment execution module can be encapsulated so that a Python script can be executed in Blender through the Blender Python environment execution module, so as to execute an operation API in the Blender Python environment, and implement programmatic generation and modification according to a description text corresponding to user updated requirements, such as gradually adding new objects and modifying items (objects) in the scene, dynamically adjusting objects and parameters in the scene, etc., until all tasks are completed.

[0108] Fig. 9 A schematic diagram showing generation and modification of a scene in an exemplary embodiment of the present disclosure.

[0109] Reference Fig. 9 According to an exemplary embodiment of the present disclosure, a corresponding target scene can be generated according to the description text 901, and objects in the generated scene can be modified accordingly according to the description text 902 corresponding to the modification instruction to update the generated scene.

[0110] Fig.10 A schematic diagram showing scene generation by PCGPlanner and PCGHub in an exemplary embodiment of the present disclosure.

[0111] Reference Fig.10 According to an exemplary embodiment of the present disclosure, PCGHub includes various PCG models (including objects such as 3D models) and assets (including objects such as operation APIs), which can easily integrate user PCG and assets. Formatting and packaging are simple, with smooth operation processes, 500+ program modules, and 10,000+ high-quality assets. After obtaining the user's description text (for example, generating beautiful cherry blossoms / generating a modern city / generating an autumn forest / generating a woods at noon, etc.), the scene generation task is performed, and the corresponding scene generation result can be obtained after performing scene decomposition, terrain generation, object generation or retrieval, object arrangement and other processes based on the description text. The scene model generated by the scene generation method of the exemplary embodiment of the present disclosure has a delicate geometric structure, realistic material textures, and natural lighting, and can be seamlessly deployed in industrial pipelines.

[0112] Fig.11A schematic diagram showing the flow of scene generation performed by the PCGPlanner framework in an exemplary embodiment of the present disclosure.

[0113] Reference Fig.11 According to an exemplary embodiment of the present disclosure, the PCG Planner framework includes three basic components: mission planning, asset retrieval, and action execution. The framework enables LLMs as agents to plan missions in complex scenarios, perform operations using multiple APIs, and facilitate large-scale scenario generation.

[0114] The task planning component includes a scheduling agent and a professional agent. The scheduling agent is used to generate a rough plan containing objects under the corresponding prompt template, and the professional agent is used to further generate a detailed plan based on the rough plan under the corresponding prompt template.

[0115] After obtaining the detailed plan, the asset retrieval component can perform retrieval based on it. The retrieval agent can encode the description text of the target scene (Text Encoder) to obtain the text vector and then obtain the retrieval results of 3D model assets (objects) based on text-to-image retrieval, or obtain the retrieval results of the operation API (object operation interface) based on text-to-text retrieval.

[0116] After obtaining the search results, the action execution component can obtain the target scenario by executing the agent under the corresponding prompt template by importing objects or modifying the object operation interface to operate objects.

[0117] Fig.12 A flowchart showing a method for performing scene generation by a programmatic generation method in an exemplary embodiment of the present disclosure is shown.

[0118] According to an exemplary embodiment of the present disclosure, pictures, text / dialogues, structured data, etc. or a combination thereof for describing the target scene can be obtained. The description can be preliminarily processed by the data feature processing module to extract information in the description based on the large language model task parsing module. After that, the clarity of the task can be judged based on the information. If the task is not clear enough, the language model can ask questions about fuzzy items until the received task is clear enough. After that, the tasks can be assigned to obtain the target scene through the collaborative operation of each module. For example, the tasks can be assigned to three major blocks: terrain information processing, asset information processing, and scene object placement. Both terrain information processing and asset information processing can be used by the PCG super miserable generator to generate terrain and assets respectively. For the placement of scene objects, the objects in the scene can be placed and corrected according to the relationship and description between the objects. After that, the environment can be integrated based on the generated terrain, assets, and the placement relationship between objects to obtain a 3D digital scene.

[0119] Fig.13 A schematic diagram showing visualization results of the generation quality of large-scale natural and urban scenes in an exemplary embodiment of the present disclosure.

[0120] Reference Fig.13 According to an exemplary embodiment of the present disclosure, when the description text is "In early summer, under the bright sunshine, the forest is lush with vegetation", "Cherry blossoms are in full bloom, the grass is green, and the river flows peacefully", and "Bright lights illuminate this modern city, showing a distinctive contemporary style", the effect of scene generation can be as follows: Fig.13 As shown in respectively.

[0121] Fig.14 A schematic diagram showing generation effects of other scenes in an exemplary embodiment of the present disclosure.

[0122] Reference Fig.14 According to an exemplary embodiment of the present disclosure, after obtaining the corresponding description text, a variety of corresponding scene effect diagrams can be generated.

[0123] Fig.15 A schematic diagram showing a comparison of the effects of city scene generation of the related technology and the exemplary embodiment of the present disclosure is shown.

[0124] Reference Fig.15 According to the exemplary embodiments of the present invention, with respect to the generation effect of urban scenes, in the related technologies, the scene effects generated by frameworks such as Persistent Nature, Scene Dreamer, InfiniCity, and CityDreamer all have the problem of unreasonable geometric shapes to varying degrees, while the scene generation method of the present invention performs realistically and excellently when generating large-scale urban scenes.

[0125] According to an exemplary embodiment of the present disclosure, the following Table 1 shows the results of the average score (AS) and the expert average score (AES) of each model and Sceneχ of the present disclosure, and the following Table 2 evaluates the similarity between the prompt rendering results of Sceneχ of the present disclosure and the rendering results of each model.

[0126]

[0127] According to exemplary embodiments of the present disclosure, the following table shows a comparison of the time required for generating natural scenes and cities at different terrain scales between the architecture in the related art and the architecture of the present disclosure.

[0128]

[0129] According to an exemplary embodiment of the present disclosure, a system named SceneX is proposed, which realizes high-quality scene generation of 3D natural scenes and 3D cities by using Blender executable Python scripts. SceneX may include two main components: PCGHub and PCGPlanner.

[0130] PCGHub is a large-scale PCG repository that contains rich procedural modeling resources, a large number of static assets, and related pre-processing PCG API code and documentation. It can be used as a resource library to generate high-quality 3D scenes and evaluate SceneX generation performance.

[0131] Based on the PCGHub, this paper proposes a PCGPlanner framework for automatically generating 3D scenes using PCG and predefined models. It can be composed of a mission planning phase, an asset retrieval phase, and an action execution phase guided by custom instructions. Through the collaboration of PCGHub and PCGPlanner agents, SceneX is able to efficiently and controllably generate a variety of high-quality scenes.

[0132] According to an exemplary embodiment of the present disclosure, a base and pipeline for a diverse procedurally generated world can be constructed through a large language model (LLM) and agent technology. Specifically, the following may be included:

[0133] (1) Intelligent agent interaction construction scene:

[0134] The agent built with the Large Language Model (LLM) generates a variety of virtual scenes through high-quality, low-threshold interactions. The agent can automatically generate corresponding scenes and assets based on the user's natural language description, greatly reducing the threshold for programmatic generation and improving the quality and diversity of generation.

[0135] (2) Seamlessly bridge PCGHub and downstream ecosystem:

[0136] PCGPlanner can achieve seamless bridging with the PCGHub platform. Through standardized interfaces and data formats, it ensures that the generated scenes and assets can be smoothly integrated into PCGHub and its downstream systems, achieving programmatically generated interoperability and data sharing.

[0137] (3) Fully automated process:

[0138] Establish a fully automated process covering scene generation, asset creation, and ground editing and adjustment. This process includes all steps from preliminary design to final generation, which can ensure the automation and efficiency of the generation process, thereby reducing manual intervention and improving the accuracy and consistency of generation.

[0139] Fig.16 A block diagram of a scene generating device in an exemplary embodiment of the present disclosure is shown.

[0140] Reference Fig.16 , an exemplary embodiment of the present disclosure also provides a scene generation device 1600, which may include but is not limited to a description acquisition unit 1601, a first planning unit 1602, a second planning unit 1603, an asset retrieval unit 1604, and a scene generation unit 1605.

[0141] The description acquisition unit 1601 may acquire a description text for a target scene.

[0142] The first planning unit 1602 may determine a first plan through a scheduling agent based on the description text, wherein the first plan includes an object for generating a target scene.

[0143] The second planning unit 1603 may determine a second plan including subtasks through a professional agent based on the object and a predefined knowledge document corresponding to the object in a preset resource library, wherein the subtasks correspond to the predefined knowledge document.

[0144] The asset retrieval unit 1604 may search in a preset resource library through a retrieval agent based on the subtask in the second plan to obtain an object or an object operation interface corresponding to the subtask.

[0145] The scene generation unit 1605 may generate a target scene by executing an agent based on an object or an object operation interface corresponding to a subtask.

[0146] It is understandable that in the exemplary embodiment of the above-mentioned scene generation device 1600, the specific implementation process is roughly the same as the exemplary embodiment of the above-mentioned scene generation method, and will not be repeated here. The scene generation device 1600 can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these devices may correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. In addition, one or more functions implemented by these devices can also be uniformly executed by components in physical entity devices (for example, processors, clients or servers, etc.).

[0147] Fig.17 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0148] Reference Fig.17 The electronic device 1700 includes at least one memory 1701 and at least one processor 1702, wherein the at least one memory 1701 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 1702, a scene generation method according to an exemplary embodiment of the present disclosure is executed.

[0149] As an example, the electronic device 1700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1700 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0150] In the electronic device 1700, the processor 1702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0151] The processor 1702 may execute instructions or codes stored in the memory 1701, wherein the memory 1701 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0152] The memory 1701 may be integrated with the processor 1702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1701 and the processor 1702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1702 can read files stored in the memory.

[0153] In addition, the electronic device 1700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1700 may be connected to each other via a bus and / or a network.

[0154] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the above-mentioned scene generation method.

[0155] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers. It should be noted that the instructions can also be used to execute additional steps in addition to the above steps or to perform more specific processing when executing the above steps. These additional steps and further processing contents have been mentioned in the description of the relevant methods, so they will not be repeated here to avoid repetition.

[0156] Another embodiment of the present disclosure relates to a system including at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the above-mentioned scene generation method.

[0157] It should be noted that the system according to the exemplary embodiments of the present disclosure can completely rely on the execution of computer programs or instructions to realize corresponding functions, that is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., lib library) to realize the corresponding functions.

[0158] On the other hand, when the above-mentioned system is implemented in software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that at least one processor or at least one computing device can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0159] According to an exemplary embodiment of the present disclosure, the storage device may be integrated with the computing device, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage device may include an independent device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage device and the computing device may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the computing device can read instructions stored in the storage device.

[0160] Another embodiment of the present disclosure relates to a computer program product, including a computer program / instruction, which implements any of the above scene generation methods when executed by a processor.

[0161] According to the scene generation method, device, electronic device, storage medium and computer program product provided by the present invention, it is possible to divide the scene generation task into a programmatic generation framework consisting of a task planning stage, a resource retrieval stage, and an action execution stage through multiple agents. Combined with a resource library including rich object resources, object operation interface resources, documents and other programmatic modeling resources, high-quality scene generation effects can be obtained.

[0162] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A scene generation method, characterized in that: The scene generation method comprises: Get the description text for the target scene; Determining a first plan through a scheduling agent based on the description text, wherein the first plan includes objects for generating the target scene; Based on the object and a predefined knowledge document corresponding to the object in a preset resource library, determining a second plan including subtasks through a professional agent, wherein the subtasks correspond to the predefined knowledge document; Based on the subtask in the second plan, searching the preset resource library through a search agent to obtain an object or object operation interface corresponding to the subtask; The target scene is generated by executing an agent based on the object or object operation interface corresponding to the subtask.

2. The scene generation method according to claim 1, characterized in that: The scheduling agent, the professional agent, and the search agent are large language models; Wherein, the scene generation method further includes: For the current tasks to be processed by the scheduling agent, the professional agent, the retrieval agent and the execution agent respectively, the large language model serving as the scheduling agent, the professional agent, the retrieval agent and the execution agent is prompted by a preset prompt template corresponding to the current task, wherein the preset prompt template includes at least one of the following items: role information of the current agent, task objectives and constraints of the current task, knowledge related to the current task, output format of the current agent, and reference examples for the current task.

3. The scene generation method according to claim 2, characterized in that: The searching in the preset resource library by a search agent based on the subtask in the second plan includes: Instructing the search agent to output a search description through the preset prompt template corresponding to the subtask, wherein the search description is used to describe the object or object operation interface corresponding to the subtask; Based on the embedding vector corresponding to the search description, a search is performed in the preset resource library.

4. The scene generation method according to claim 1, characterized in that: The determining the first plan by a scheduling agent based on the description text includes: extracting a first object from the description text by the scheduling agent; A second object related to the target scene is inferred by the scheduling agent to determine a first plan, wherein the first plan includes the first object and the second object.

5. The scene generation method according to claim 1, characterized in that: The generating the target scene by executing the agent includes: In case the object operation interface is retrieved, modifying predefined parameters of the object operation interface based on the description text; Based on the object operation interface after the predefined parameters are modified, the target scene is generated by the execution agent.

6. The scene generation method according to claim 5, characterized in that: The modifying of the predefined parameters of the object operation interface based on the description text includes: The predefined parameters of the object operation interface regarding the object style are modified based on the description text, wherein the object style includes at least one of the following items: object material attributes and object geometric structure.

7. The scene generation method according to claim 1, characterized in that: The second plan includes at least one of the subtasks, and the predefined knowledge document includes constraints for combining multiple subtasks, and the constraints include at least one of the following items: restrictions on the combination method and the order of use of the combined basic units.

8. A scene generation device, characterized in that: The scene generating device comprises: The description acquisition unit is configured to: acquire a description text for a target scene; A first planning unit is configured to: determine a first plan through a scheduling agent based on the description text, wherein the first plan includes an object for generating the target scene; A second planning unit is configured to: determine a second plan including subtasks through a professional agent based on the object and a predefined knowledge document corresponding to the object in a preset resource library, wherein the subtasks correspond to the predefined knowledge document; An asset retrieval unit is configured to: based on the subtask in the second plan, search the preset resource library through a retrieval agent to obtain an object or object operation interface corresponding to the subtask; The scene generation unit is configured to generate the target scene by executing an agent based on the object or object operation interface corresponding to the subtask.

9. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the scene generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to execute the scene generation method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the scene generation method according to any one of claims 1 to 7 is implemented.

Images