Text-to-human motion synthesis method and device

By retrieving and decomposing assets from the 3D object library, building a relationship diagram to generate human movements, and using visual consistency scoring function optimization, the problem of poor character action generation ability in traditional methods is solved, realism and vision in specific scenarios are achieved Consistent effect.

CN118918227APending Publication Date: 2024-11-08SHIYOU UNIVERSE (CHENGDU) TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411044244.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When traditionally artificially crafted characters' movements, the generation ability is poor, especially in specific scenes such as martial arts movies, the movements do not conform to physical laws, but the existing technology has not effectively solved this problem.

Method used

By obtaining text descriptions, the characters and scene assets are retrieved from the 3D object library, decompose into character layout, posture and scene assets, build relationship diagrams, and connect spatial relationship nodes to generate human movements, and use the visual consistency scoring function to optimize the movements to conform to Visual effects.

Benefits of technology

It improves the ability to generate characters' actions, ensures that the actions are realistic and consistent in a specific scene, and solves the problem that the actions do not conform to physical laws in traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918227A_ABST
    Figure CN118918227A_ABST
Patent Text Reader

Abstract

The invention discloses a text-to-human motion synthesis method and device. The method comprises the steps that text description is acquired, corresponding assets are retrieved from a 3D object library based on the text description, and the assets comprise figures and corresponding scenes; the assets are decomposed into figure layout, figure postures, scene assets and a layout relation graph, a relation graph is constructed based on the decomposed figure layout, figure postures, scene assets and the layout relation graph, the assets are connected with corresponding spatial relation nodes in the relation graph, and human body actions are obtained. According to the invention, the technical problem of poor character action generation capability is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image rendering, and in particular to a method and device for synthesizing text into human motion. Background Art

[0002] When making character movements manually, the layout of 3D assets can be precisely controlled by manual production, so as to produce character movements that are relatively in line with the needs. Physical simulation is used to realize the interaction between people and scenes, and the algorithm synthesizes character movements. By organizing training data, writing neural network structures, and training model parameters, it has the ability to generate character movements.

[0003] After the introduction of physical simulation, the rendering effect will be closer to reality, but the functions of physical simulation are usually approximate, complex, and expensive to render. At the same time, in certain scenarios, traditional physical rules may fail. For example, in some martial arts movie scenes, people's actions obviously do not conform to the laws of physics. However, in such scenarios, the final rendering effect will not be considered a problem because of violating the laws of physics. On the contrary, such rules will be accepted.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiment of the present invention provides a method and device for synthesizing text into human motion, so as to at least solve the technical problem of poor generation capability of human motion.

[0006] According to one aspect of an embodiment of the present invention, a text-to-human motion synthesis method is provided, comprising: obtaining a text description, retrieving corresponding assets from a 3D object library based on the text description, the assets comprising characters and corresponding scenes; decomposing the assets into character layout, character posture, scene assets and a layout relationship graph, constructing a relationship graph based on the decomposed character layout, character posture, scene assets and layout relationship graph, and connecting the assets with corresponding spatial relationship nodes in the relationship graph to obtain human motion.

[0007] According to another aspect of an embodiment of the present invention, a text-to-human motion synthesis device is also provided, including: an acquisition module, configured to acquire a text description, and retrieve corresponding assets from a 3D object library based on the text description, wherein the assets include characters and corresponding scenes; an action synthesis module, configured to decompose the assets into character layout, character posture, scene assets and a layout relationship diagram, construct a relationship diagram based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and connect the assets with corresponding spatial relationship nodes in the relationship diagram to obtain human motion.

[0008] In an embodiment of the present invention, a text description is obtained, and corresponding assets are retrieved from a 3D object library based on the text description, wherein the assets include characters and corresponding scenes; the assets are decomposed into character layout, character posture, scene assets, and a layout relationship diagram, and a relationship diagram is constructed based on the decomposed character layout, character posture, scene assets, and layout relationship diagram, and the assets are connected to corresponding spatial relationship nodes in the relationship diagram to obtain human body movements. Through the above solution, the technical problem of poor generation capability of character movements is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0010] Figure 1 is a flowchart of a method for synthesizing text into human motion according to an embodiment of the present invention;

[0011] Figure 2 is a flow chart of a method for generating human body motion according to an embodiment of the present invention;

[0012] Figure 3 is a flowchart of another text-to-human action synthesis method according to an embodiment of the present invention;

[0013] Figure 4 is a structural schematic diagram of a text-to-human motion synthesis device according to an embodiment of the present invention;

[0014] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] According to an embodiment of the present invention, an embodiment of a text-to-human motion synthesis method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0018] Figure 1 is a text-to-human action synthesis method according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0019] Step S102: obtaining a text description, and retrieving corresponding assets from a 3D object library based on the text description, wherein the assets include characters and corresponding scenes.

[0020] Receive text descriptions input by users, including natural language text, to describe a scene and / or character. Perform preprocessing on the text descriptions, such as word segmentation, part-of-speech tagging, and named entity recognition. Extract keywords or key phrases from the text to determine the core content of the description.

[0021] Pre-index the assets in the 3D object library, categorize and label them based on keywords or metadata. Based on the extracted keywords, query the 3D object library to retrieve assets that match the description. The search can include matching of characters and scenes.

[0022] Use text similarity algorithms (such as cosine similarity, Jaccard similarity) to calculate the similarity between the text description and the asset tag or description. Based on the similarity score, filter out irrelevant assets and sort the assets with higher relevance. Return the retrieved assets to the user, including the persona and the corresponding scene.

[0023] Step S104, decomposing the assets into character layout, character posture, scene assets and layout relationship diagram, constructing a relationship diagram based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and connecting the assets with corresponding spatial relationship nodes in the relationship diagram to obtain human body movements.

[0024] like Figure 2 As shown, the method for generating human body movements comprises the following steps:

[0025] Step S1042, decomposing assets.

[0026] Decompose 3D assets into four parts: character layout, character pose, scene assets, and layout relationship diagram. Character layout is used to describe the position and direction of each character in the scene, which can be expressed in coordinates and direction angles. Character pose is used to describe the character's skeletal pose, that is, the position and angle of each joint of the character at a specific moment. Scene assets are static or dynamic objects in the scene, such as buildings, furniture, trees, etc. The layout relationship diagram is used to describe the spatial relationship between characters, objects, and scenes.

[0027] Step S1044, constructing a relationship diagram.

[0028] The nodes of the relationship graph represent individual assets, and the edges represent the relationships between them (such as "located in" or "adjacent to"). Nodes represent characters, objects, or elements in a scene, and edges represent the relationships between these elements. The specific steps are as follows: Create nodes: Create a node for each asset (character, scene, etc.) and attach descriptive information (such as the character's role, the type of scene). Create edges: Create edges based on the spatial relationship between assets. For example, character A is located in scene B, and you can add an edge from character A to scene B to represent the "located in" relationship.

[0029] Step S1046, connecting the asset with the spatial relationship node in the relationship graph.

[0030] Connect each asset to the corresponding node in the relationship graph to establish the spatial relationship between characters, poses, and scenes. The specific steps include: 1) Mark node position and direction. Add position information and direction information to each node. For example, a node can represent the specific position and orientation of a character in a scene. 2) Mark pose information. Add skeletal pose information to the character node to describe the position and angle of each joint of the character at a specific moment.

[0031] Step S1048, synthesizing human body movements.

[0032] The action of the character is derived through the information of nodes and edges in the relationship graph. The specific steps are as follows: 1) Analyze the character's posture. Determine the character's current action (such as standing, walking, running) based on the character's skeletal posture information. 2) Analyze the spatial relationship: Determine the character's motion trajectory and relative position in the scene based on the spatial relationship between the character and the scene. For example, if a character moves from one position to another, its motion path and direction can be determined based on the edge information in the relationship graph. 3) Synthesize human body movements. Combine the analyzed posture and spatial relationship information to generate detailed human body movements.

[0033] Through the above steps, the 3D assets can be decomposed into character layout, character posture, scene assets and layout relationship diagram, and a relationship diagram can be built based on these elements to obtain human body movements.

[0034] In some embodiments, after obtaining the human body movement, it is also possible to check whether the human body movement meets the requirements; if the human body movement does not meet the requirements, the human body movement is evaluated, the constraints that do not meet the requirements in the human body movement are identified, and the human body movement is optimized according to the identification result to perform self-improvement.

[0035] After identifying the unsatisfied constraints in the human motion, a spatial skill library is constructed using the knowledge learned in the self-improvement process, wherein the spatial skill library includes constraint functions for describing the relational constraints on the assets.

[0036] In some embodiments, a visual consistency scoring function is used to determine whether the relationship constraints of the assets are correct, wherein the visual consistency scoring function evaluates whether the assets and assets, character postures and expressions, the spatial relationships between people and scenes, and the spatial relationships between people in the corresponding scenes meet the visual consistency description by rendering the results in a single-person or multi-person interactive state.

[0037] For example, a visual consistency scoring function is used to determine whether the relationship constraints of the assets are correct: check whether the relative position, size, and direction between the assets are consistent with expectations; evaluate whether the character's posture and expression are natural and match the scene and action; determine whether the character's interaction with objects and the environment in the scene is reasonable; observe whether the relative position and interaction between the characters are as expected; and determine whether the overall visual effect of the rendering result is as expected.

[0038] The input of the visual consistency scoring function includes an asset layout list, relationship parameters and character action parameters, wherein the asset layout list is a spatial relationship matrix connected by relationships; the relationship parameters are parameters related to relationships; and the character action parameters are used to describe the actions of each character in the current scene.

[0039] Figure 3 is another text-to-human action synthesis method according to an embodiment of the present invention. Figure 3 As shown, the method comprises the following steps:

[0040] Step S302, asset retrieval agent.

[0041] An agent is an LLM-based agent that can autonomously perform tasks, interact with the environment and make decisions.

[0042] First, according to the input text description, a CLIP-based retriever is used to retrieve relevant assets from the 3D object library, including characters and corresponding scenes. CLIP-based retriever is a retriever based on the CLIP (Contrastive Language-Image Pre-training) model. It uses the image and text understanding capabilities of the CLIP model to retrieve assets related to the input text description from a large-scale 3D object library.

[0043] Step S304, script decomposition agent.

[0044] Decompose the requirements into character layout, character posture, scene assets and their layout relationship diagram, use LLM-Constructor to build the relationship diagram, and connect the assets with the corresponding spatial relationship nodes. LLM-Constructor is a LLM-Coder, which is also a code generator based on a large model. The main function of LLM-Constructor in this application is to generate Blender's python code based on the text description and asset list to build the scene graph and human body movements. Blender is a 3D creation tool.

[0045] Step S306, scene layout optimization agent in the feedback loop.

[0046] Use LLM-Coder to generate Blender code based on the scene graph and human body movements, and use Blender to render. If the rendering result does not meet the requirements, use LLM-VReviewer to evaluate and provide feedback optimization based on the evaluation results. LLM-Coder is a code generator based on large models, which can modify and optimize the code based on the results of LLM-VReviewer. LLM-VReviewer is a visual code auditor that identifies whether each asset in the scene meets the constraints required by the text and points out where modifications are needed.

[0047] Step S308, feedback loop skill agent.

[0048] Using the knowledge learned in the self-improvement process, a spatial skill library is built to generate high-quality character actions faster when processing new input queries. Among them, a scoring function is needed to determine whether the relationship constraints of the assets are correct. This application introduces a visual consistency scoring function to evaluate whether the assets and assets, character postures and expressions, the spatial relationship between people and scenes, and the spatial relationship between people in the scene meet the visual consistency description by rendering the results in a single or multi-person interactive state.

[0049] The visual consistency scoring function is described in detail below. The scoring function ScoreRelation is used to evaluate whether the relationship constraints between character assets and scene assets are satisfied. It is the core tool for optimizing scene layout.

[0050] The input of the scoring function includes the asset layout list, relationship parameters and character action parameters. Among them, the asset layout list layout(asset) is a spatial relationship matrix connected by the relationship relation. The layout matrix layout(asset) describes the position and posture of the asset asset in 3D space. Relationship parameters are parameters related to the relationship, such as distance, direction, etc. These parameters define the specific requirements of the relationship. Character action parameters describe the actions of each character in the current scene. The output of the scoring function is a real number between 0 and 1, which describes the degree of satisfaction of the relationship constraint, 1 means complete satisfaction, and 0 means complete dissatisfaction.

[0051] Scoring functions are used to check asset-to-asset consistency. Check whether the relative position, size, and orientation between assets are as expected. For example, are objects placed in the right place, and are there appropriate overlaps or gaps. You can also check the consistency of character poses and expressions. Evaluate whether the character's poses and expressions are natural and match the scene and action. For example, whether the character is standing, sitting, or performing a specific action, and whether the expression is consistent with the situation. You can also check the consistency of the spatial relationship between people and scenes. Determine whether the interaction between the character and the objects and environment in the scene is reasonable. For example, whether the character can move freely in the scene, and whether it collides or is blocked by objects. You can also check the consistency of the spatial relationship between people. Observe whether the relative position and interaction between characters are as expected. For example, whether the characters are having a conversation, collaborating, or keeping an appropriate distance. You can also check the overall visual quality. Consider the overall visual effect of the rendering result, including color, lighting, material, etc. Evaluate whether the image is clear, realistic, and has a good artistic effect.

[0052] In some embodiments, the visual consistency score may be calculated using the following formula:

[0053] Visual consistency score = consistency weight between assets × consistency score between assets + consistency weight of character posture and expression × consistency score of character posture and expression + consistency weight of spatial relationship between person and scene × consistency score of spatial relationship between person and scene + consistency weight of spatial relationship between people × consistency score of spatial relationship between people + overall visual quality weight × overall visual quality score.

[0054] The consistency score between assets is determined by evaluating the matching degree of relative position, size and direction between assets. The formula can be as follows:

[0055]

[0056] Where N is the number of asset pairs, a i and a j is the i-th pair of assets. α, β, and γ are the weights of position score, size score, and direction score, respectively. i ,a j ) is asset a i and a j SizeScore(a i ,a j ) is asset a i and a j DirScore(a i ,a j ) is asset a i and a j direction score.

[0057] in, d actual (a i ,a j ) is asset a i and a j The actual distance. expected (a i ,a j ) is asset a i and a j expected distance.

[0058] Among them, s actua l is the actual size of the asset. s expecte d is the expected size of the asset. θ actual (a i ,a j ) is asset a i and a j The actual direction angle difference. θ expected (ai ,a j ) is asset a i and a j The expected direction angle difference.

[0059] in, θ actua l is the actual posture angle of the character, θ expected is the desired pose angle of the character. d actual is the actual distance between the character and the object in the scene, d expected is the expected distance between the character and objects in the scene. d actua l1 is the actual distance between the characters. expected1 is the desired distance between characters. Q i is the score of the i-th quality evaluation item, and n is the number of quality evaluation items.

[0060] In this application, the scoring function can verify whether the position, size and orientation of the asset are as expected, evaluate the naturalness of the character's posture and expression, determine whether the interaction between the character and the objects and environment in the scene is reasonable, and observe whether the interaction between the characters is as expected. At the same time, the overall visual effect of the rendering result, such as color, lighting and material, is also considered to ensure that the image is clear, realistic and has good artistic effect. Through the above evaluation formula, the calculation results are more accurate, which greatly improves the rationality and visual appeal of the scene layout.

[0061] This embodiment uses a multimodal language model to model and construct 3D relationships, automatically learns and updates constraint functions to adapt to different scenes and relationships, thereby improving the accuracy and rationality of scene layout. The scene layout is iteratively improved through a visual feedback loop to ensure that the generated scene meets the requirements of the text description. In addition, this embodiment introduces a visual consistency scoring function to evaluate the state of a person and whether the relationship between the person and the scene is consistent in visual effect.

[0062] The present application also provides a text to human motion synthesis device, such as Figure 4 As shown, it includes: an acquisition module 42, which is configured to acquire a text description, and retrieve corresponding assets from a 3D object library based on the text description, wherein the assets include characters and corresponding scenes; an action synthesis module 44, which is configured to decompose the assets into character layout, character posture, scene assets and layout relationship diagram, construct a relationship diagram based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and connect the assets with corresponding spatial relationship nodes in the relationship diagram to obtain human body actions.

[0063] It should be noted that the text-to-human motion synthesis device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual object rendering device provided in the above embodiment and the virtual object rendering method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0064] Figure 5 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0065] like Figure 5 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0066] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0067] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the method and apparatus of the present application are executed. In some embodiments, the electronic device may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0068] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0069] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0070] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0071] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0072] The computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement each step of the method embodiment.

[0073] As another aspect, the present application further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0074] As another aspect, the present application further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0075] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0076] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0079] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0081] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text-to-human action synthesis method, characterized in that: include: Obtaining a text description, and retrieving a corresponding asset from a 3D object library based on the text description, wherein the asset includes a character and a corresponding scene; The assets are decomposed into character layout, character posture, scene assets and layout relationship diagram, a relationship diagram is constructed based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and the assets are connected with corresponding spatial relationship nodes in the relationship diagram to obtain human body movements.

2. The method according to claim 1, characterized in that: After obtaining the human body motion, the method further includes: Check whether the human body movements meet the requirements; In the case that the human body motion does not meet the requirements, the human body motion is evaluated, the constraints that do not meet the requirements in the human body motion are identified, and the human body motion is optimized according to the identification result to perform self-improvement.

3. The method according to claim 2, characterized in that After identifying the unsatisfactory constraints in the human body motion, the method further comprises: using the knowledge learned in the self-improvement process to construct a spatial skill library, wherein the spatial skill library comprises constraint functions for describing the relational constraints on the assets.

4. The method according to claim 3, characterized in that A visual consistency scoring function is used to determine whether the relationship constraints of the assets are correct, wherein the visual consistency scoring function evaluates whether the assets and assets, the character's posture and expression, the spatial relationship between people and the scene, and the spatial relationship between people in the corresponding scene meet the visual consistency description by rendering the results in a single-person or multi-person interactive state.

5. The method according to claim 4, characterized in that A visual consistency scoring function is used to judge whether the relationship constraints of the asset are correct: Check that the relative positions, sizes, and orientations of assets are as expected; Evaluate whether the character's posture and expression are natural and match the scene and action; Determine whether the interaction between the characters and the objects and environment in the scene is reasonable; Observe whether the relative positions and interactions between characters are as expected; Determine whether the overall visual effect of the rendering results meets expectations.

6. The method according to claim 4, characterized in that The input of the visual consistency scoring function includes an asset layout list, relationship parameters and character action parameters, wherein the asset layout list is a spatial relationship matrix connected by relationships; the relationship parameters are parameters related to relationships; and the character action parameters are used to describe the actions of each character in the current scene.

7. A text to human motion synthesis device, characterized in that: include: An acquisition module is configured to acquire a text description and retrieve a corresponding asset from a 3D object library based on the text description, wherein the asset includes a character and a corresponding scene; The action synthesis module is configured to decompose the assets into character layout, character posture, scene assets and layout relationship diagram, construct a relationship diagram based on the decomposed character layout, character posture, scene assets and layout relationship diagram, and connect the assets with corresponding spatial relationship nodes in the relationship diagram to obtain human body actions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Physically fine-grained controllable animation synthesis method and system for robot large model

    WO2026102708A1