Arbitrary skeleton motion generation method and system based on multimodal large language model

Abstract

The present invention discloses a method and system for generating arbitrary skeleton motion based on a multimodal large language model, which relates to the field of computer technology and solves the technical problem that it is difficult to ensure the naturalness and coherence of motion, and the accuracy and efficiency are low in text-driven skeleton animation. The method includes: understanding a given skeleton structure based on a multimodal large language model; refining a given abstract motion description into a specific motion description of each joint of a key frame; comparing the motion description of each key frame with a standard posture to generate a key frame; visualizing the key frame, evaluating and feeding back the visualization result; generating the remaining frames according to the information of the key frame and the motion description; visualizing all frames of the skeleton structure, and then evaluating and feeding back; binding the object surface and the skeleton structure to generate skeleton motion. The present invention improves the accuracy and efficiency of generating arbitrary skeleton motion by reasoning and refining the abstract motion description through a multimodal large language model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for generating arbitrary skeleton motion based on a multimodal large language model. Background Art

[0002] In recent years, with the development of computer technology, large language models (LLMs) have made significant progress. By expanding the scale of data and models, these LLMs have demonstrated extraordinary emergent capabilities, often including instruction following, contextual learning, and thought chaining. Although LLMs have demonstrated amazing zero-shot / few-shot reasoning performance on most natural language processing (NLP) tasks, they are essentially "blind" to vision because they can only understand discrete texts. As a result, multimodal large language models have emerged, which refer to LLM-based models, large-scale neural network models that have the ability to receive, reason, and output multimodal information (such as text, images, videos, etc.).

[0003] At the same time as the emergence of multimodal large-scale language models, there is also the text-driven arbitrary skeleton motion generation technology, which uses a piece of motion description information describing a virtual object to generate the virtual object motion. The text-driven arbitrary skeleton motion generation is a problem of aligning text with structured motion. It requires the model to understand the differences and uniformities between different skeletons, and use a unified structural representation for different skeletons to drive the timing.

[0004] There are two main solutions in the existing technology. One is to avoid driving the skeleton and directly drive the surface, first predict the key points of the object's surface and the weights of the other surface points with respect to these key points, then drive these key points through text, and then obtain the positions of the other surface points based on the driven key points and weights. The other is to train a generative model with a limited data set of text-skeleton motion pairs, directly obtain the corresponding skeleton motion through the text as a condition, and then obtain the moving surface through the bone binding technology.

[0005] However, the error of the solution driven by surface key points is large, especially the structure between different frames will have serious inconsistency problems, and the coherence of movement is poor. The solution of using a limited dataset of text-skeleton motion pairs to train the generative model is limited by the dataset and cannot generate the movement of skeletons outside the dataset distribution. In other words, it is difficult to generate arbitrary skeleton movements that have never appeared before, and it is difficult to obtain data of non-human skeletons. A new method for generating arbitrary skeleton movements is urgently needed.

[0006] In the process of implementing the present invention, it is found that there are at least the following problems in the prior art:

[0007] It is difficult to ensure the naturalness and continuity of skeleton movements that have never appeared based on text-driven arbitrary skeleton animation, and the accuracy and efficiency of generating non-human skeleton movements are low. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for generating arbitrary skeleton motion based on a multimodal large language model, so as to solve the technical problems existing in the prior art that it is difficult to ensure the naturalness and coherence of skeleton motion based on text-driven arbitrary skeleton animation, and the accuracy and efficiency of generating non-human skeleton motion are low. The many technical effects that can be produced by the preferred technical solution among the many technical solutions provided by the present invention are described in detail below.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The present invention provides a method for generating arbitrary skeleton motion based on a multimodal large language model, comprising the following steps: S100: understanding a given skeleton structure based on a multimodal large language model; S200: the multimodal large language model refines a given abstract motion description into a specific motion description of each joint of a key frame; S300: comparing the motion description of each key frame with a standard posture to generate a key frame; S400: visualizing the key frame, and evaluating the visualization result through the multimodal large language model; S500: the multimodal large language model generates remaining frames according to the information of the key frame and the motion description; S600: visualizing all frames of the skeleton structure, and the multimodal large language model evaluates the visualization of all frames; S700: binding the object surface and the skeleton structure through the bone binding technology to generate skeleton motion.

[0011] Preferably, the steps S400 and S500 further include: S450: adjusting the joint XYZ rotation axis angles in the key frame according to the evaluation result of the multimodal large language model.

[0012] Preferably, the steps S600 and S700 further include: S650: adjusting the joint XYZ rotation axis angles in non-key frames according to the visual evaluation results of all frames.

[0013] Preferably, the S100 step specifically includes: S110: extracting the joint hierarchy structure and the position information of each joint relative to its parent node in the HIERARCHY information of the skeleton through a multimodal large language model, and rendering the image corresponding to the standard posture of the skeleton through Blender software; S120: combining the species description and the joint hierarchy structure of the skeleton, the multimodal large language model infers the physical meaning of the three motion axes of each joint; S130: the multimodal large language model aligns the three motion axes of each joint with the three rotation axes XYZ in the Blender software; S140: the multimodal large language model describes and memorizes the standard posture in natural language.

[0014] Preferably, in step S200, the multimodal large language model parses each key frame to include detailed movements of the joints, and provides detailed action descriptions of the three movement axes.

[0015] Preferably, in step S300, the keyframes are generated by inferring the specific XYZ rotation axis angles of all joints in each keyframe.

[0016] Preferably, in step S400, the evaluation content includes the naturalness of the joint movement posture and the rationality of the overall movement, and the multimodal large language model also points out areas that need improvement after outputting the evaluation results.

[0017] Preferably, in step S500, the multimodal large language model infers transition frames between key frames to ensure the continuity and naturalness of the skeleton movement, and the XYZ rotation axis angles of all joints will be concretized in each frame.

[0018] Preferably, in the step S600, all frames are visualized by Blender software, and the evaluation content of the multimodal large language model includes checking the coherence of the entire motion sequence, the rationality of the posture of each frame and the overall motion effect, and giving specific modification suggestions.

[0019] A system for generating arbitrary skeleton motion based on a multimodal large language model, wherein the system is used to run any of the above methods for generating arbitrary skeleton motion based on a multimodal large language model.

[0020] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0021] This application uses a multimodal large language model to reason and refine abstract motion descriptions, and refines abstract motion descriptions into specific motion descriptions for each key frame. Compared with existing text-driven skeleton motion generation technology solutions, it is easier to achieve alignment and understanding of skeleton motion, and thus easier to generate non-human skeleton motion that has never appeared. This application first generates the joint XYZ rotation axis angles of the key frames, and then generates the joint XYZ rotation axis angles of all frames to ensure the naturalness and continuity of the motion. On the basis of ensuring accuracy, it effectively simplifies the process of generating arbitrary skeleton animations and effectively improves the accuracy and efficiency of generating arbitrary skeleton motions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0023] Figure 1 It is a flowchart of a method for generating arbitrary skeleton motion based on a multimodal large language model according to a first embodiment of the present invention;

[0024] Figure 2 It is a flowchart of step S100 in the method for generating arbitrary skeleton motion based on a multimodal large language model in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0026] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0027] In order to illustrate the technical solution of the present invention, a specific embodiment is used below for description, and only the parts related to the embodiment of the present invention are shown.

[0028] Embodiment 1:

[0029] like Figure 1As shown, the present invention provides a method for generating arbitrary skeleton motion based on a multimodal large language model, including the following steps. S100: Based on the multimodal large language model, the given skeleton structure is understood, that is, the physical meaning of the joint motion of the skeleton structure is aligned with the physical meaning of the XYZ axis rotation in the Blender software. The XYZ axis is the name of three orthogonal axes, which are often used to describe rotation and displacement in 3D space. Blender software is a free open source three-dimensional graphics and image software that provides a series of animation short film production solutions from modeling, animation, materials, rendering, to audio processing, video editing, etc., which is convenient for users to implement skeleton motion operations. S200: The multimodal large language model refines the given abstract motion description into a specific motion description of each joint of the key frame. The key frame is a frame used to define the key moment in the animation in animation production. The key frame contains the main actions and postures of the animation. S300: Compare the motion description of each key frame with the standard posture, that is, infer the specific XYZ rotation axis angles of all joints in each key frame, and generate key frames. By generating key frames, it is convenient for this embodiment to quickly generate the final skeleton animation. S400: Visualize the keyframes and evaluate the visualization results through the multimodal large language model. Due to the illusion of the multimodal large language model itself, the keyframes directly inferred may have defects. Therefore, the keyframes generated by visualization are evaluated by the multimodal large language model through the Blender software. S500: The multimodal large language model generates the remaining frames according to the information of the keyframes and the motion description. S600: Visualize all frames of the skeleton structure, and the multimodal large language model evaluates the visualization of all frames. S700: Bind the object surface and the skeleton structure through the skeleton binding technology to generate skeleton motion. In 3D animation, the skeleton binding technology binds the object surface to the skeleton so that the skeleton motion can drive the surface deformation and movement of the object. This embodiment infers and refines the abstract motion description through the multimodal large language model, and refines the abstract motion description into a specific motion description for each keyframe. Compared with the existing text-driven skeleton motion generation technology, it solves the problem of alignment and difficulty in understanding different skeleton motions, thereby making it easier to generate non-human skeleton motions that have never appeared. In this embodiment, the joint XYZ rotation axis angles of the key frame are first generated, and then the joint XYZ rotation axis angles of all frames are generated to ensure the naturalness and continuity of the movement. On the basis of ensuring accuracy, the process of generating arbitrary skeleton animation is effectively simplified, and the accuracy and efficiency of generating arbitrary skeleton movement are effectively improved.

[0030] As an optional implementation, Figure 1As shown, between steps S400 and S500, the following is also included: S450: adjusting the angles of the joint XYZ rotation axes in the keyframe according to the evaluation results of the multimodal large language model. The adjusted keyframe can be visualized and evaluated again, and this process is repeated until the evaluation results are satisfactory. Through multiple iterations, the motion data of the keyframe is ensured to be accurate and natural. This can achieve a deep understanding and analysis of the skeleton structure, which is convenient for further improvement of the skeleton motion.

[0031] As an optional implementation, Figure 1 As shown, between steps S600 and S700, the following steps are also included: S650: adjusting the angles of the joint XYZ rotation axes in the non-key frames according to the visual evaluation results of all frames. Visualizing and evaluating the adjusted frames again can ensure that each frame of the final motion sequence meets expectations. Through multiple iterations, a complete and natural skeleton motion sequence is finally generated. This can achieve a deep understanding and analysis of the skeleton structure, which is convenient for further improvement of the skeleton motion.

[0032] As an optional implementation, Figure 2 As shown, the step S100 specifically includes: S110: extracting the joint hierarchy structure and the position information of each joint relative to its parent node in the HIERARCHY (HIERARCHY is the abbreviation of the joint hierarchy information, which is used to describe the subordinate relationship between the joints in the skeleton) information of the skeleton through the multimodal large language model, and rendering the image corresponding to the standard posture of the skeleton through the Blender software. S120: Combining the species description and joint hierarchy of the skeleton, the multimodal large language model infers the physical meaning of the three motion axes of each joint. For example, the physical meaning of the three motion axes of the right thigh joint of a person is to raise the leg forward / backward, to raise the leg to the right / left, and to rotate outward / inward. S130: The multimodal large language model aligns the three motion axes of each joint with the three rotation axes of XYZ in the Blender software. S140: The multimodal large language model describes and memorizes the standard posture in natural language. At this time, the rotation angles of the XYZ axes of all joints are 0.

[0033] As an optional implementation, in step S200, the multimodal large language model parses the detailed movement of the joints contained in each key frame and provides detailed action descriptions of the three movement axes. For example, when "a person is running" and the model is required to generate 100 frames of movement, the model will divide "a person is running" into several key frames distributed over 100 frames, such as the first frame is a standing state with relaxed hands, the tenth frame is a ready state with clenched fists, raised arms and bent legs, the fifteenth frame is a state where the left leg steps forward, the right leg bends backward, the right arm is raised forward, and the left arm is raised backward, and so on.

[0034] As an optional implementation, in step S300, key frames are generated by inferring specific XYZ rotation axis angles of all joints of each key frame, so that key frames can be generated quickly.

[0035] As an optional implementation, in step S400, the evaluation content includes the naturalness of the joint movement posture and the rationality of the overall movement. After the multimodal large language model outputs the evaluation results, it also points out the areas that need improvement. Thus, the multimodal large language model realizes the function of prompt engineering, that is, by providing specific prompts or guidance, the multimodal large language model can perform specific tasks, thereby achieving a deep understanding and analysis of the skeleton structure, which is convenient for further improvement of the skeleton movement.

[0036] As an optional implementation, in step S500, the multimodal large language model infers transition frames between key frames to ensure the continuity and naturalness of the skeleton movement. The XYZ rotation axis angles of all joints will be concretized in each frame, thereby effectively ensuring the continuity and naturalness of the movement.

[0037] As an optional implementation, in step S600, all frames (including key frames and transition frames) are visualized by Blender software, and the evaluation of the multimodal large language model includes checking the coherence of the entire motion sequence, the rationality of the posture of each frame, and the overall motion effect, and giving specific modification suggestions. The specific modification suggestions facilitate key frame adjustment to achieve the continuity and naturalness of the skeleton movement.

[0038] The embodiment is only a specific example and does not represent only one implementation mode of the present invention.

[0039] Embodiment 2:

[0040] A system for generating arbitrary skeletal motion based on a multimodal large language model, the skeletal motion generation system is used to run the method for generating arbitrary skeletal motion based on a multimodal large language model of Example 1. This embodiment uses a multimodal large language model to infer and refine the abstract motion description, and refines the abstract motion description into a specific motion description for each key frame. Compared with the existing text-driven skeletal motion generation technical solution, it solves the problem of different skeletal motion alignment and difficulty in understanding. First, the joint XYZ rotation axis angles of the key frame are generated, and then the joint XYZ rotation axis angles of all frames are generated to ensure the naturalness and continuity of the motion. On the basis of ensuring accuracy, the process of generating arbitrary skeletal animation is effectively simplified, and the accuracy and efficiency of generating skeletal motion are effectively improved.

[0041] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A method for generating arbitrary skeleton motion based on a multimodal large language model, characterized in that: The following steps are involved: S100: Understanding a given skeleton structure based on a multimodal large language model; S200: The multimodal large language model refines the given abstract motion description into a specific motion description of each joint of the key frame; S300: Compare the motion description of each key frame with the standard posture to generate a key frame; S400: Visualize the key frames and evaluate the visualization results through a multimodal large language model; S500: The multimodal large language model generates remaining frames according to the information of the key frames and the motion description; S600: Visualize all frames of the skeleton structure, and the multimodal large language model evaluates the visualization of all frames; S700: Bind the object surface and the skeleton structure through the skeleton binding technology to generate skeleton movement; The S100 step specifically includes: S110: extracting the joint hierarchy structure and the position information of each joint relative to its parent node in the HIERARCHY information of the skeleton through the multimodal large language model, and rendering the image corresponding to the standard posture of the skeleton through the Blender software; S120: combining the species description and the joint hierarchy structure of the skeleton, the multimodal large language model infers the physical meaning of the three motion axes of each joint; S130: the multimodal large language model aligns the three motion axes of each joint with the three rotation axes XYZ in the Blender software; S140: the multimodal large language model describes and memorizes the standard posture in natural language.

2. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: The steps S400 and S500 also include: S450: According to the evaluation result of the multimodal large language model, adjust the joint XYZ rotation axis angle in the key frame.

3. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: The steps S600 and S700 also include: S650: According to the visual evaluation results of all frames, the joint XYZ rotation axis angles in the non-key frames are adjusted.

4. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: In the step S200, the multimodal large language model parses the detailed movements of the joints contained in each key frame and provides detailed action descriptions of the three movement axes.

5. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: In step S300 , a key frame is generated by inferring the specific XYZ rotation axis angles of all joints in each key frame.

6. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: In the step S400, the evaluation content includes the naturalness of the joint movement posture and the rationality of the overall movement. After the multimodal large language model outputs the evaluation results, it also points out the areas that need improvement.

7. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: In step S500, the multimodal large language model infers transition frames between key frames to ensure the continuity and naturalness of the skeleton movement, and the XYZ rotation axis angles of all joints will be concretized in each frame.

8. The method for generating arbitrary skeleton motion based on a multimodal large language model according to claim 1, characterized in that: In the step S600, all frames are visualized by Blender software, and the evaluation content of the multimodal large language model includes checking the coherence of the entire motion sequence, the rationality of the posture of each frame, and the overall motion effect, and giving specific modification suggestions.

9. A system for generating arbitrary skeleton motion based on a multimodal large language model, characterized in that: The skeleton motion generation system is used to run any skeleton motion generation method based on a multimodal large language model described in any one of claims 1 to 8.

Images