Text-to-3D scene synthesis method and device
Through the text-to-3D scene synthesis method, text descriptions are obtained, assets are retrieved, scene decomposed, and scene maps are built, which solves the problem of insufficient control in traditional 3D scene generation, and realizes efficient and accurate 3D scene generation, reducing costs and training cycles. .
Patent Information
- Application Number
- CN202411044243.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
AI Technical Summary
Traditional manual 3D scenes are costly, the training cycle of algorithm training and synthesis of 3D scenes is long, the control force is weak, and dynamic adjustment cannot be achieved. Moreover, the control ability of visual multimodal generation 3D scenes is insufficient, so the generation effect cannot be controlled more accurately.
By obtaining text descriptions, the corresponding assets are retrieved from the 3D object library, the scene decomposition is performed, and the scene diagram is constructed based on the decomposition results, and the 3D scene is finally synthesized. The method includes preprocessing text descriptions, extracting keywords, retrieving assets using text similarity algorithms, decomposing assets, building scene maps, and optimizing scene layout through visual feedback loops to ensure that the generated 3D scene meets the expected requirements.
It improves the production efficiency and accuracy of 3D scenes, enhances the controllability of scene assets and their layout relationships, significantly reduces the time and cost of manual adjustment and correction, and ensures the quality and consistency of generated 3D scenes.
Smart Images

Figure CN118918264A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image rendering, and in particular to a method and device for synthesizing text into a 3D scene. Background Art
[0002] Traditional artificial 3D scenes can be manually produced to precisely control the layout of 3D assets, thereby producing 3D scenes that are relatively in line with demand. Algorithmic synthesis of 3D scenes is achieved by organizing training data, writing neural network structures, and training model parameters to enable it to have the ability to generate 3D scenes. However, the cost of traditional artificial production of 3D scenes is high, the training cycle of algorithm training to synthesize 3D scenes is long, the control is weak, and dynamic adjustment cannot be achieved. Both visual multimodal generation of 3D scenes and 3D scenes generated by ordinary algorithms have the problem of insufficient control capabilities and cannot accurately control the generation effect.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for synthesizing text into a 3D scene, so as to at least solve the technical problem that the scene generation effect cannot be controlled more accurately.
[0005] According to one aspect of an embodiment of the present invention, a method for text-to-3D scene synthesis is provided, comprising: obtaining a text description, and retrieving corresponding assets from a 3D object library based on the text description; performing scene decomposition based on the assets, and constructing a scene graph based on the result of the scene decomposition, and then synthesizing a 3D scene based on the scene graph.
[0006] According to another aspect of an embodiment of the present invention, a text-to-3D scene synthesis device is also provided, including: an acquisition module, configured to acquire a text description, and retrieve corresponding assets from a 3D object library based on the text description; a synthesis module, configured to perform scene decomposition based on the assets, and construct a scene graph based on the result of the scene decomposition, and then synthesize a 3D scene based on the scene graph.
[0007] In the embodiment of the present invention, a text description is obtained, and corresponding assets are retrieved from a 3D object library based on the text description; scene decomposition is performed based on the assets, and a scene graph is constructed based on the result of the scene decomposition, and then a 3D scene is synthesized based on the scene graph. The above solution solves the technical problem of being unable to accurately control the scene generation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0009] Figure 1 is a flowchart of a method for synthesizing text into a 3D scene according to an embodiment of the present invention;
[0010] Figure 2 is another text to 3D scene synthesis method according to an embodiment of the present invention;
[0011] Figure 3 It is another text to 3D scene synthesis method according to an embodiment of the present invention;
[0012] Figure 4 is a structural schematic diagram of a text-to-3D scene synthesis device according to an embodiment of the present invention;
[0013] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0014] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0016] According to an embodiment of the present invention, an embodiment of a text to 3D scene synthesis method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0017] Figure 1 is a text to 3D scene synthesis method according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:
[0018] Step S102: obtaining a text description, and retrieving a corresponding asset from a 3D object library based on the text description.
[0019] Receive text descriptions input by users, including natural language text, to describe a scene and / or character. Perform preprocessing on the text descriptions, such as word segmentation, part-of-speech tagging, and named entity recognition. Extract keywords or key phrases from the text to determine the core content of the description.
[0020] Pre-index the assets in the 3D object library, categorize and label them based on keywords or metadata. Based on the extracted keywords, query the 3D object library to retrieve assets that match the description. The search can include matching of characters and scenes.
[0021] Use text similarity algorithms (such as cosine similarity, Jaccard similarity) to calculate the similarity between the text description and the asset tag or description. Based on the similarity score, filter out irrelevant assets and sort the assets with higher relevance. Return the retrieved assets to the user, including the persona and the corresponding scene.
[0022] Step S104: performing scene decomposition based on the asset, constructing a scene graph based on the result of the scene decomposition, and then synthesizing a 3D scene based on the scene graph.
[0023] First, perform scene decomposition. Decompose the assets into scene assets and layout relationship graphs; construct a relationship graph based on the decomposed scene assets and layout relationship graphs, and connect the assets with the corresponding spatial relationship nodes in the relationship graph to obtain the result of the scene decomposition. This application not only improves the efficiency and accuracy of scene construction, but also enhances the controllability of scene assets and their layout relationships, so that the generated 3D scene better meets the expected requirements.
[0024] Then, a scene graph is constructed based on the result of the scene decomposition, and then a 3D scene is synthesized based on the scene graph. The code of the 3D creation tool is generated according to the scene graph, and the 3D creation tool is used to render the 3D scene based on the code. This embodiment generates the code of the 3D creation tool and uses the tool for rendering, which not only improves the production efficiency and accuracy of the 3D scene, but also ensures that the layout and details of the scene meet expectations, and significantly reduces the time and cost of manual adjustment and correction.
[0025] In some embodiments, after synthesizing the 3D scene based on the scene graph, check whether the 3D scene meets the requirements; if the 3D scene does not meet the requirements, evaluate the 3D scene, identify the constraints in the 3D scene that do not meet the requirements, and optimize the scene layout according to the identification results to perform self-improvement. This application not only improves the accuracy and quality of the 3D scene, but also enhances the self-improvement ability of the system, so that the finally generated 3D scene is more in line with expectations and needs.
[0026] In some embodiments, after identifying the constraints that do not meet the requirements in the 3D scene, the knowledge learned in the self-improvement process can also be used to build a spatial skill library, which includes constraint functions for describing the relationship constraints on the assets. For example, a scoring function can be used to judge whether the relationship constraints of the assets are correct, wherein the input of the scoring function is an asset layout list of the layout matrix representing the assets connected by the relationship and the relationship parameters, and the output of the scoring function is a real number between 0 and 1 that describes the degree of satisfaction of the relationship constraints. In this way, the layout of the 3D scene can be more accurately evaluated and optimized, and the quality and consistency of the generated scene can be improved.
[0027] Figure 2 is another text to 3D scene synthesis method according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:
[0028] Step S202, asset retrieval.
[0029] First, according to the input text description, a CLIP-based retriever is used to retrieve relevant assets from the 3D object library, including characters and corresponding scenes. CLIP-based retriever is a retriever based on the CLIP (Contrastive Language-Image Pre-training) model. It uses the image and text understanding capabilities of the CLIP model to retrieve assets related to the input text description from a large-scale 3D object library.
[0030] Step S204, scene decomposition.
[0031] Decompose the requirements into scene assets and their layout relationship graphs, use LLM-Constructor to build the relationship graph, and connect the assets with the corresponding spatial relationship nodes. LLM-Constructor is a LLM-Coder, which is also a code generator based on a large model. The main function of LLM-Constructor in this application is to generate Blender's python code based on the text description and asset list to build the scene graph and human body movements. Blender is a 3D creation tool.
[0032] Step S206, scene layout optimization in the feedback loop.
[0033] Use LLM-Coder to generate Blender code based on the scene graph and use Blender to render. If the rendering result does not meet the requirements, use LLM-VReviewer to evaluate and provide feedback optimization based on the evaluation results. LLM-Coder is a large model-based code generator that can modify and optimize the code based on the results of LLM-VReviewer. LLM-VReviewer is a visual code auditor that identifies whether each asset in the scene meets the constraints of the text requirements and points out where modifications are needed.
[0034] Step S208, feedback loop skill library.
[0035] Using the knowledge learned during the self-improvement process, a spatial skill library is built to generate high-quality 3D scenes faster when processing new input queries. Among them, a scoring function is needed to determine whether the relationship constraints of the assets are correct.
[0036] The visual consistency scoring function is described in detail below. The scoring function ScoreRelation is used to evaluate whether the relationship constraints between character assets and scene assets are satisfied. It is the core tool for optimizing scene layout.
[0037] The input of the scoring function includes the asset layout list and relationship parameters. Among them, the asset layout list layout(asset) is a spatial relationship matrix connected by the relationship relation. The layout matrix layout(asset) describes the position and posture of the asset asset in 3D space. Relationship parameters are parameters related to the relationship, such as distance, direction, etc. These parameters define the specific requirements of the relationship. The output of the scoring function is a real number between 0 and 1, which describes the degree of satisfaction of the relationship constraint, 1 means complete satisfaction, and 0 means complete dissatisfaction.
[0038] Scoring functions are used to check asset-to-asset consistency. Check if the relative position, size, and orientation between assets are as expected. For example, are objects placed in the correct position, with appropriate overlaps or gaps. You can also check the consistency of character poses and expressions.
[0039] In some embodiments, the visual consistency score may be calculated using the following formula:
[0040] Visual consistency score = asset-to-asset consistency weight × asset-to-asset consistency score + overall visual quality weight × overall visual quality score.
[0041] The consistency score between assets is determined by evaluating the matching degree of relative position, size and direction between assets. The formula can be as follows:
[0042]
[0043] Where N is the number of asset pairs, a i and a j is the i-th pair of assets. α, β, and γ are the weights of position score, size score, and direction score, respectively. i ,a j ) is asset a i and a j SizeScore(a i ,a j ) is asset a i and a j DirScore(a i ,a j ) is asset a i and a j direction score.
[0044] in, d actual (a i ,a j ) is asset a i and a j The actual distance. expected (a i ,a j ) is asset a i and a j expected distance.
[0045] Among them, s actua l is the actual size of the asset. s expecte d is the expected size of the asset. θ actual (a i,a j ) is asset a i and a j The actual direction angle difference. θ expected (a i ,a j ) is asset a i and a j The expected direction angle difference.
[0046] By comparing the actual distance with the expected distance and the actual size with the expected size, this solution can more comprehensively evaluate the visual consistency between assets, thereby improving the visual quality and consistency of the synthesis results. The overall visual quality weight and score further optimizes the evaluation of the global visual effect, making the score more accurate and comprehensive.
[0047] in, Q i is the score of the i-th quality evaluation item, and n is the number of quality evaluation items.
[0048] This embodiment uses a multimodal language model to model and construct 3D relationships, automatically learns and updates constraint functions to adapt to different scenes and relationships, thereby improving the accuracy and rationality of scene layout. The scene layout is iteratively improved through a visual feedback loop to ensure that the generated scene meets the requirements of the text description. In addition, this embodiment introduces a visual consistency scoring function to evaluate the state of a person and whether the relationship between the person and the scene is consistent in visual effect.
[0049] Figure 3 is another text to 3D scene synthesis method according to an embodiment of the present invention, such as Figure 3 As shown, the method comprises the following steps:
[0050] Step S302: obtaining a text description, and retrieving corresponding assets from a 3D object library based on the text description, wherein the assets include characters and corresponding scenes.
[0051] Receive text descriptions input by users, including natural language text, to describe a scene and / or character. Perform preprocessing on the text descriptions, such as word segmentation, part-of-speech tagging, and named entity recognition. Extract keywords or key phrases from the text to determine the core content of the description.
[0052] Pre-index the assets in the 3D object library, categorize and label them based on keywords or metadata. Based on the extracted keywords, query the 3D object library to retrieve assets that match the description. The search can include matching of characters and scenes.
[0053] Use text similarity algorithms (such as cosine similarity, Jaccard similarity) to calculate the similarity between the text description and the asset tag or description. Based on the similarity score, filter out irrelevant assets and sort the assets with higher relevance. Return the retrieved assets to the user, including the persona and the corresponding scene.
[0054] Step S304, decomposing the assets into character layout, character posture, scene assets and layout relationship diagram, constructing a relationship diagram based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and connecting the assets with corresponding spatial relationship nodes in the relationship diagram to obtain human body movements.
[0055] For example, a method for generating human motion includes the following steps:
[0056] 1) Decompose assets.
[0057] Decompose 3D assets into four parts: character layout, character pose, scene assets, and layout relationship diagram. Character layout is used to describe the position and direction of each character in the scene, which can be expressed in coordinates and direction angles. Character pose is used to describe the character's skeletal pose, that is, the position and angle of each joint of the character at a specific moment. Scene assets are static or dynamic objects in the scene, such as buildings, furniture, trees, etc. The layout relationship diagram is used to describe the spatial relationship between characters, objects, and scenes.
[0058] 2) Build a relationship diagram.
[0059] The nodes of the relationship graph represent individual assets, and the edges represent the relationships between them (such as "located in" or "adjacent to"). Nodes represent characters, objects, or elements in a scene, and edges represent the relationships between these elements. The specific steps are as follows: Create nodes: Create a node for each asset (character, scene, etc.) and attach descriptive information (such as the character's role, the type of scene). Create edges: Create edges based on the spatial relationship between assets. For example, character A is located in scene B, and you can add an edge from character A to scene B to represent the "located in" relationship.
[0060] 3) Connect assets to spatial relationship nodes in the relationship graph.
[0061] Connect each asset to the corresponding node in the relationship graph to establish the spatial relationship between characters, poses, and scenes. The specific steps include: 1) Mark node position and direction. Add position information and direction information to each node. For example, a node can represent the specific position and orientation of a character in a scene. 2) Mark pose information. Add skeletal pose information to the character node to describe the position and angle of each joint of the character at a specific moment.
[0062] 4) Synthesize human body movements.
[0063] The action of the character is derived through the information of nodes and edges in the relationship graph. The specific steps are as follows: 1) Analyze the character's posture. Determine the character's current action (such as standing, walking, running) based on the character's skeletal posture information. 2) Analyze the spatial relationship: Determine the character's motion trajectory and relative position in the scene based on the spatial relationship between the character and the scene. For example, if a character moves from one position to another, its motion path and direction can be determined based on the edge information in the relationship graph. 3) Synthesize human body movements. Combine the analyzed posture and spatial relationship information to generate detailed human body movements.
[0064] 5) Synthesize 3D scenes.
[0065] The steps for synthesizing a 3D scene are as follows Figure 1 and Figure 2 The corresponding embodiments are described and will not be repeated here.
[0066] Through the above steps, the 3D assets can be decomposed into character layout, character posture, scene assets and layout relationship diagram, and a relationship diagram can be built based on these elements to obtain human body movements.
[0067] In some embodiments, after obtaining the human body movement, it is also possible to check whether the human body movement meets the requirements; if the human body movement does not meet the requirements, the human body movement is evaluated, the constraints that do not meet the requirements in the human body movement are identified, and the human body movement is optimized according to the identification result to perform self-improvement.
[0068] After identifying the unsatisfied constraints in the human motion, a spatial skill library is constructed using the knowledge learned in the self-improvement process, wherein the spatial skill library includes constraint functions for describing the relational constraints on the assets.
[0069] In some embodiments, a visual consistency scoring function is used to determine whether the relationship constraints of the assets are correct, wherein the visual consistency scoring function evaluates whether the assets and assets, character postures and expressions, the spatial relationships between people and scenes, and the spatial relationships between people in the corresponding scenes meet the visual consistency description by rendering the results in a single-person or multi-person interactive state.
[0070] For example, a visual consistency scoring function is used to determine whether the relationship constraints of the assets are correct: check whether the relative position, size, and direction between the assets are consistent with expectations; evaluate whether the character's posture and expression are natural and match the scene and action; determine whether the character's interaction with objects and the environment in the scene is reasonable; observe whether the relative position and interaction between the characters are as expected; and determine whether the overall visual effect of the rendering result is as expected.
[0071] The input of the visual consistency scoring function includes an asset layout list, relationship parameters and character action parameters, wherein the asset layout list is a spatial relationship matrix connected by relationships; the relationship parameters are parameters related to relationships; and the character action parameters are used to describe the actions of each character in the current scene.
[0072] The embodiment of the present invention further provides a method for synthesizing text into a 3D scene, the method comprising the following steps:
[0073] 1) Asset retrieval agent.
[0074] An agent is an LLM-based agent that can autonomously perform tasks, interact with the environment and make decisions.
[0075] First, according to the input text description, a CLIP-based retriever is used to retrieve relevant assets from the 3D object library, including characters and corresponding scenes. CLIP-based retriever is a retriever based on the CLIP (Contrastive Language-Image Pre-training) model. It uses the image and text understanding capabilities of the CLIP model to retrieve assets related to the input text description from a large-scale 3D object library.
[0076] 2) Script decomposition agent.
[0077] Decompose the requirements into character layout, character posture, scene assets and their layout relationship diagram, use LLM-Constructor to build the relationship diagram, and connect the assets with the corresponding spatial relationship nodes. LLM-Constructor is a LLM-Coder, which is also a code generator based on a large model. The main function of LLM-Constructor in this application is to generate Blender's python code based on the text description and asset list to build the scene graph and human body movements. Blender is a 3D creation tool.
[0078] 3) Scene layout optimization agent in the feedback loop.
[0079] Use LLM-Coder to generate Blender code based on the scene graph and human body movements, and use Blender to render. If the rendering result does not meet the requirements, use LLM-VReviewer to evaluate and provide feedback optimization based on the evaluation results. LLM-Coder is a code generator based on large models, which can modify and optimize the code based on the results of LLM-VReviewer. LLM-VReviewer is a visual code auditor that identifies whether each asset in the scene meets the constraints required by the text and points out where modifications are needed.
[0080] 4) Feedback loop skill agent.
[0081] Using the knowledge learned in the self-improvement process, a spatial skill library is built to generate high-quality character actions faster when processing new input queries. Among them, a scoring function is needed to determine whether the relationship constraints of the assets are correct. This application introduces a visual consistency scoring function to evaluate whether the assets and assets, character postures and expressions, the spatial relationship between people and scenes, and the spatial relationship between people in the scene meet the visual consistency description by rendering the results in a single or multi-person interactive state.
[0082] The visual consistency scoring function is described in detail below. The scoring function ScoreRelation is used to evaluate whether the relationship constraints between character assets and scene assets are satisfied. It is the core tool for optimizing scene layout.
[0083] The input of the scoring function includes the asset layout list, relationship parameters and character action parameters. Among them, the asset layout list layout(asset) is a spatial relationship matrix connected by the relationship relation. The layout matrix layout(asset) describes the position and posture of the asset asset in 3D space. Relationship parameters are parameters related to the relationship, such as distance, direction, etc. These parameters define the specific requirements of the relationship. Character action parameters describe the actions of each character in the current scene. The output of the scoring function is a real number between 0 and 1, which describes the degree of satisfaction of the relationship constraint, 1 means complete satisfaction, and 0 means complete dissatisfaction.
[0084] Scoring functions are used to check asset-to-asset consistency. Check whether the relative position, size, and orientation between assets are as expected. For example, are objects placed in the right place, and are there appropriate overlaps or gaps. You can also check the consistency of character poses and expressions. Evaluate whether the character's poses and expressions are natural and match the scene and action. For example, whether the character is standing, sitting, or performing a specific action, and whether the expression is consistent with the situation. You can also check the consistency of the spatial relationship between people and scenes. Determine whether the interaction between the character and the objects and environment in the scene is reasonable. For example, whether the character can move freely in the scene, and whether it collides or is blocked by objects. You can also check the consistency of the spatial relationship between people. Observe whether the relative position and interaction between characters are as expected. For example, whether the characters are having a conversation, collaborating, or keeping an appropriate distance. You can also check the overall visual quality. Consider the overall visual effect of the rendering result, including color, lighting, material, etc. Evaluate whether the image is clear, realistic, and has a good artistic effect.
[0085] In some embodiments, the visual consistency score may be calculated using the following formula:
[0086] Visual consistency score = consistency weight between assets × consistency score between assets + consistency weight of character posture and expression × consistency score of character posture and expression + consistency weight of spatial relationship between person and scene × consistency score of spatial relationship between person and scene + consistency weight of spatial relationship between people × consistency score of spatial relationship between people + overall visual quality weight × overall visual quality score.
[0087] The calculation method of the consistency score between assets is as described above and will not be repeated here.
[0088] in, θ actual is the actual posture angle of the character, θ expected is the desired pose angle of the character. d actual is the actual distance between the character and the object in the scene, d expected is the expected distance between the character and objects in the scene. d actual1 is the actual distance between the characters. expected1 is the desired distance between characters. Q i is the score of the i-th quality evaluation item, and n is the number of quality evaluation items.
[0089] In this application, the scoring function can verify whether the position, size and orientation of the asset are as expected, evaluate the naturalness of the character's posture and expression, determine whether the interaction between the character and the objects and environment in the scene is reasonable, and observe whether the interaction between the characters is as expected. At the same time, the overall visual effect of the rendering result, such as color, lighting and material, is also considered to ensure that the image is clear, realistic and has good artistic effect. Through the above evaluation formula, the calculation results are more accurate, which greatly improves the rationality and visual appeal of the scene layout.
[0090] This embodiment uses a multimodal language model to model and construct 3D relationships, automatically learns and updates constraint functions to adapt to different scenes and relationships, thereby improving the accuracy and rationality of scene layout. The scene layout is iteratively improved through a visual feedback loop to ensure that the generated scene meets the requirements of the text description. In addition, this embodiment introduces a visual consistency scoring function to evaluate the state of a person and whether the relationship between the person and the scene is consistent in visual effect.
[0091] The present application also provides a text to 3D scene synthesis device, such as Figure 4 As shown, the device includes: an acquisition module 42 is configured to acquire a text description and retrieve corresponding assets from a 3D object library based on the text description; a synthesis module 44 is configured to perform scene decomposition based on the assets, and construct a scene graph based on the result of the scene decomposition, and then synthesize a 3D scene based on the scene graph.
[0092] It should be noted that the text-to-3D scene synthesis device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the text-to-3D scene synthesis device provided in the above embodiment and the text-to-3D scene synthesis method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0093] Figure 5 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0094] like Figure 5As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0095] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0096] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0097] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0098] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0099] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0100] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.
[0101] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method embodiment.
[0102] As another aspect, the present application further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.
[0103] As another aspect, the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.
[0104] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0105] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0110] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for synthesizing text into 3D scenes, characterized in that: include: Obtaining a text description, and retrieving a corresponding asset from a 3D object library based on the text description; A scene is decomposed based on the asset, a scene graph is constructed based on the result of the scene decomposition, and then a 3D scene is synthesized based on the scene graph.
2. The method according to claim 1, characterized in that Perform scenario decomposition based on the assets, including: Decomposing the assets into scene assets and layout relationship diagrams; A relationship graph is constructed based on the decomposed scene assets and the layout relationship graph, and the assets are connected with corresponding spatial relationship nodes in the relationship graph to obtain the result of the scene decomposition.
3. The method according to claim 1, characterized in that Synthesizing a 3D scene based on the scene graph includes: The code of a 3D creation tool is generated according to the scene graph, and the 3D creation tool is used to render a 3D scene based on the code.
4. The method according to any one of claims 1 to 3, characterized in that After synthesizing the 3D scene based on the scene graph, the method further includes: Check whether the 3D scene meets the requirements; In the case that the 3D scene does not meet the requirements, the 3D scene is evaluated, the constraints that do not meet the requirements in the 3D scene are identified, and the scene layout is optimized according to the identification results to perform self-improvement.
5. The method according to claim 4, characterized in that After identifying the unsatisfactory constraints in the 3D scene, the method further includes: using the knowledge learned in the self-improvement process to construct a spatial skill library, wherein the spatial skill library includes constraint functions for describing the relationship constraints on the assets.
6. The method according to claim 5, characterized in that A scoring function is used to determine whether the relationship constraints of the assets are correct, wherein the input of the scoring function is an asset layout list and relationship parameters representing a layout matrix of the assets connected by the relationships, and the output of the scoring function is a real number between 0 and 1 that describes the degree to which the relationship constraints are satisfied.
7. A text to 3D scene synthesis device, characterized in that: include: An acquisition module is configured to acquire a text description and retrieve a corresponding asset from a 3D object library based on the text description; The synthesis module is configured to perform scene decomposition based on the asset, construct a scene graph based on the result of the scene decomposition, and then synthesize a 3D scene based on the scene graph.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.
9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Physically fine-grained controllable animation synthesis method and system for robot large model
WO2026102708A1
