Human body action sequence generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of human body action generation. The invention discloses a human body action sequence generation method and device, equipment and a storage medium. The generation quality of human body actions can be improved, and the generalization ability of a model can be improved. The human body action sequence generation method comprises the following steps: acquiring text information and a scene image; and inputting the text information and the scene image into an action planning model for task decomposition and action generation processing, and outputting a human body action sequence.

Description

Technical Field

[0001] This application relates to the technical field of human motion generation. More specifically, this application relates to a method, apparatus, device, and storage medium for generating a human motion sequence. Background Art

[0002] The traditional method for generating a human motion sequence is to obtain a text signal; input the text signal into an action generation model to generate an action sequence corresponding to the text signal; wherein, the action generation model includes a text encoder and an action decoder; the text encoder is used to encode the text signal; the latent space representation output by the text encoder conforms to a unified probability distribution; the action decoder is used to decode the feature vector output by the text encoder into an action sequence. However, the action generation model has limited ability to understand the surrounding environment information of the human body, resulting in low quality of generated human actions and poor generalization ability of the model. Summary of the Invention

[0003] The objective of the embodiments of this application is to provide a method, apparatus, device, and storage medium for generating a human motion sequence, which can improve the quality of generated human actions and the generalization ability of the model. The embodiments of this application are mainly implemented through the following technical solutions: In the first aspect of the embodiments of this application, a method for generating a human motion sequence is provided, including: Obtain text information and a scene image; Input the text information and the scene image into an action planning model for task decomposition and action generation processing, and output a human motion sequence.

[0004] According to an embodiment of this application, the step of inputting the text information and the scene image into an action planning model for task decomposition and action generation processing, and outputting a human motion sequence includes: Input the text information into the large language model of the action planning model for task decomposition processing to obtain multiple action description information; Input the scene image and the multiple action description information into the 3D scene perception model of the action planning model for trajectory planning processing to obtain the trajectories of human key points corresponding to each action description information; Input the multiple action description information and the trajectories of all human key points into the action generation model of the action planning model for action generation processing to obtain the human motion sequence.

[0005] According to an embodiment of this application, the step of inputting the text information into the large language model of the action planning model for task decomposition processing to obtain multiple action description information includes: Input the text information into the task planner of the large language model for task decomposition to obtain multiple subtasks; Input the multiple subtasks into the subtask executor of the large language model for action recognition processing to obtain the multiple action description information.

[0006] According to an embodiment of the present application, the steps of inputting the scene image and the multiple action description information into the 3D scene perception model of the action planning model for trajectory planning to obtain the trajectories of human key points corresponding to each action description information include: Input the scene image into the query object module of the 3D scene perception model for recognition processing to obtain point cloud data, target positions, and starting positions; Input the point cloud data into the target map generation module of the 3D scene perception model for extraction processing to generate a target map; Input the point cloud data into the obstacle avoidance map generation module of the 3D scene perception model for extraction processing to generate an obstacle avoidance map; Input the target positions, the starting positions, the target map, the obstacle avoidance map, and the multiple action description information into the trajectory generation module of the 3D scene perception model for trajectory planning to generate the trajectories of human key points corresponding to each action description information.

[0007] According to an embodiment of the present application, the steps of inputting the multiple action description information and the trajectories of all human key points into the action generation model of the action planning model for action generation processing to obtain the human action sequence include: Use the action interpolation method of the human motion diffusion model in the action generation model to perform action generation processing on the target action description information and the trajectories of human key points corresponding to the target action description information to obtain a sub-action sequence, where the target action description information is any one of the multiple action description information; Merge all the sub-action sequences to obtain the human action sequence.

[0008] According to an embodiment of the present application, the method for generating the human action sequence further includes: Optimize the action interpolation method using a gradient guidance model.

[0009] According to an embodiment of the present application, the method for generating the human action sequence further includes the training steps of the action planning model, and the training steps of the action planning model include: Obtain a training data set and a true label set, where each piece of training data has a one-to-one correspondence with one of the true labels in the true label set, and each piece of training data includes text training information and a scene training image; Input the target training data into the original action planning model for task decomposition and action generation processing, and output a predicted action sequence, where the target training data is any piece of training data in the training data set; Calculate a first loss function based on the predicted action sequence and the true label corresponding to the target training data; Adjust the model parameters of the original action planning model based on the first loss function to obtain the action planning model.

[0010] In the second aspect of the embodiments of the present application, a device for generating a human action sequence is provided, including: An acquisition module, configured to acquire text information and a scene image; An output module, configured to input the text information and the scene image into an action planning model for task decomposition and action generation processing, and output a human action sequence.

[0011] In the third aspect of the embodiments of the present application, a terminal device is provided, including: a processor and a memory, where the memory is used to store a computer program, and the processor is configured to call and run the computer program stored in the memory to execute the steps of the method for generating a human action sequence provided in the first aspect of the embodiments of the present application.

[0012] In the fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, where the computer-readable storage medium is used to store a computer program, and the computer program causes a computer to execute the steps of the method for generating a human action sequence provided in the first aspect of the embodiments of the present application.

[0013] The beneficial effects of the embodiments of the present application include: In the embodiments of the present application, text information and scene images are used in combination, so that the action planning model can understand the environmental information around the human body. Specifically, in the embodiments of the present application, text information and a scene image are acquired; the text information and the scene image are input into an action planning model for task decomposition and action generation processing, and a human action sequence is output. Thus, compared with the prior art, the embodiments of the present application can enhance the ability of the action planning model to understand environmental information, thereby improving the quality of human action generation and the generalization ability of the model. Description of the Drawings

[0014] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a flowchart of the method for generating a human motion sequence of the present application in some embodiments; Figure 2 It is a schematic diagram of the principle of the method for generating a human motion sequence of the present application in some embodiments; Figure 3 It is a reference diagram of the target position of the present application in some embodiments; Figure 4 It is a reference diagram of the target map of the present application in some embodiments; Figure 5 It is a reference diagram of the obstacle avoidance map of the present application in some embodiments; Figure 6 It is a reference diagram of the trajectory of the human key points of the present application in some embodiments; Figure 7 It is a principle block diagram of the device for generating a human motion sequence of the present application in some embodiments; Figure 8 It is a principle block diagram of the terminal device of the present application in some embodiments. Detailed Embodiments

[0016] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will provide a detailed description of the specific embodiments of the present application with reference to the drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein. Those skilled in the art can make similar improvements without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0017] It should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present application, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0018] The terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or designs. Rather, the use of terms such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0019] The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that comprises a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0020] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more of the related listed items.

[0021] The following further describes the specific embodiments of this application in conjunction with the accompanying drawings.

[0022] Reference Figure 1 shown is a flowchart of a method for generating a human motion sequence provided in the first aspect of the embodiments of this application. In Figure 1 it, the method for generating a human motion sequence includes: S1. Obtain text information and a scene image.

[0023] The text information is a user instruction, which is action description information in text form. Exemplarily, the text information may be "a person is hungry", and reference can be made to Figure 2 the "text description" in

[0024] The scene image is a 3D (3 Dimensions) scene image, and reference can be made to Figure 2 the "input scene" in

[0025] S2. Input the text information and the scene image into an action planning model for task decomposition and action generation processing, and output a human motion sequence.

[0026] The action planning model includes a large language model, a 3D (3 Dimensions) scene perception model, and an action generation model. Among them, the large language model is responsible for decomposing complex actions into combinations of simple actions and generating human key point control information for subtasks; the 3D scene perception model is responsible for perceiving the passable parts in the scene image and the positions where the targets of action interaction are located; the action generation model is used to generate a full-body action sequence for each subtask based on the information provided by the large language model and the 3D scene perception model, and synthesize them into a complete output of the final action sequence.

[0027] In other embodiments, the action planning model can use a VLM (Vision-Language Model) model to replace it.

[0028] Further, the S2 step includes: S21. Input the text information into the large language model of the action planning model for task decomposition processing to obtain multiple action description information.

[0029] The large language model can be implemented by an LMP (Language Model Program). Each LMP is responsible for a unique function (such as task decomposition, text expansion, or calling a perception module), and each LMP can call other LMPs to serve the task. Further, the S21 step includes: S211. Input the text information into the task planner of the large language model for task decomposition processing to obtain multiple subtasks.

[0030] In the S211 step, first, the text information is used as a complex or abstract target task, and then the task planner decomposes the target task into multiple subtasks. The multiple subtasks are simple subtasks relative to the target task and can be called a subtask sequence. The multiple subtasks can be walking to a certain place, interacting with an object, or standing still and doing a certain action.

[0031] Exemplarily, referring to Figure 2 as shown, the text information (i.e., the "text description" in Figure 2 ) is "a person is hungry", then the multiple subtasks corresponding to the text information are three subtasks. Specifically, the first subtask is "a person walks towards edible food", referring to a1 in Figure 2 ; the second subtask is "a person bends down to pick up the edible food", referring to a2 in Figure 2 ; the third subtask is "a person is eating the food in his hand", referring to a3 in Figure 2 .

[0032] S212. Input the multiple subtasks into the subtask executor of the large language model for action recognition processing to obtain the multiple action description information.

[0033] It should be understood that the subtask executor performs action recognition processing on each subtask to obtain action description information corresponding to each subtask.

[0034] Exemplarily, referring to Figure 2 as shown, when the first subtask a1 is "a person walks towards the edible food", the action description information corresponding to the first subtask is "a person walks", referring to Figure 2 b1 in; when the second subtask a2 is "a person bends down to pick up the edible food", the action description information corresponding to the second subtask is "a person bends down to pick up", referring to Figure 2 b2 in; when the third subtask a3 is "a person is eating the food in his hand", the action description information corresponding to the third subtask is "a person is eating", referring to Figure 2 b3 in.

[0035] In other embodiments, the large language model can be replaced by an LLM Agent (Large Language Model Agent, large language model agent).

[0036] S22. Input the scene image and the multiple action description information into the 3D scene perception model of the action planning model for trajectory planning processing to obtain the trajectories of the human body key points corresponding to each action description information.

[0037] Further, the step S22 includes: S221. Input the scene image into the query object module of the 3D scene perception model for recognition processing to obtain point cloud data, a target position (which can be understood as the Figure 2 "target query" in) and a starting position.

[0038] The query object module is the parse_query_obj function. The parse_query_obj function is used to obtain the point cloud data, the target position and the starting position from the scene image. Exemplarily, when the text information is "a person is hungry", the target position is the position where the food is located, and the position where the food is located can be referred to Figure 3 the label c in.

[0039] The target position can be the position of an unconventional object.

[0040] Further, the query object module references the OpenScene method that supports open vocabulary queries (the OpenScene method is an open-source project for 3D scene understanding. It adopts a zero-shot learning method and performs a series of novel 3D scene understanding tasks through open vocabulary queries) to obtain the object category to which each point in the point cloud data belongs. Then, the query object module also references the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to filter out the generated noise and obtain the target position (i.e., the position of the object).

[0041] It should be understood that based on the open vocabulary query feature of OpenScene, in addition to querying the target position, it can also query the position of object parts. For example, when interacting with a door, it is necessary to locate the position of the doorknob instead of the entire door, which can make the generated actions more realistic.

[0042] In other embodiments, the steps for the query object module to obtain the target position include: S2211. Determine the positive prompt word for the target position.

[0043] S2212. Determine the negative prompt word for the target position.

[0044] S2213. Search for the target position in the scene image based on the positive prompt word and the negative prompt word.

[0045] S222. Input the point cloud data into the target map generation module of the 3D scene perception model for extraction processing to generate a target map.

[0046] The target map generation module is the get_target_map function. The get_target_map function is used to obtain the target map from the point cloud data. The target map can be referred to as shown in Figure 4 shown.

[0047] Further, the step S222 includes: S2221. Set a first empty map.

[0048] S2222. Based on the point cloud data, perform geometric analysis on the target position to obtain the final position.

[0049] S2223. Modify the first empty map based on the final position to obtain the target map.

[0050] Further, the step S2222 includes: S22221. Obtain the maximum coordinate point and the minimum coordinate point of the target position in the point cloud data.

[0051] S22222. Obtain the first center coordinate point of the target position in the point cloud data based on the maximum coordinate point and the minimum coordinate point.

[0052] S22223. Move the first center coordinate point along the Z-axis direction by a second preset value, and then add the Z-axis coordinate value of the moved first center coordinate point to the Z-axis coordinate value of the maximum coordinate point to obtain a second center coordinate point. The second preset value is 20. In other embodiments, the specific value of the second preset value can be set by those skilled in the art according to actual needs.

[0053] S22224. Use the second center coordinate point as the final position.

[0054] Both the first center coordinate point and the second center coordinate point are three-dimensional coordinate points.

[0055] S223. Input the point cloud data into the obstacle avoidance map generation module of the 3D scene perception model for extraction processing to generate an obstacle avoidance map.

[0056] The obstacle avoidance map generation module is the get_avoidance_map function. The get_avoidance_map function is used to obtain an obstacle avoidance map from the point cloud data. The obstacle avoidance map can be referred to Figure 5 as shown.

[0057] It should be understood that both the obstacle avoidance map and the target map are voxel maps.

[0058] Further, the step S223 includes: S2231. Set a second empty map.

[0059] S2232. Preprocess the second empty map based on the point cloud data to obtain a map to be processed.

[0060] The step S2232 refers to adding all visible obstacles in the point cloud data to the second empty map to form the map to be processed.

[0061] S2233. Obtain the third center coordinate point of the target position.

[0062] The third center coordinate point is a three-dimensional coordinate point.

[0063] S2234. Obtain the positions of key obstacles to be avoided based on the third center coordinate point, the map to be processed, and a third preset value.

[0064] S2235. Generate the obstacle avoidance map based on the positions of the key obstacles.

[0065] S224. Input the target position, the starting position, the target map, the obstacle avoidance map, and the multiple action description information into the trajectory generation module of the 3D scene perception model for trajectory planning, and generate the trajectories of the human key points corresponding to each action description information. The step S224 can be understood as Figure 2 the "path planning" step in

[0066] The trajectory generation module is the specify_pelvis_trajectory function. The specify_pelvis_trajectory function is used to generate a smooth trajectory of the human key points.

[0067] The trajectories of the human key points can be referred to Figure 6 as shown by the label d in

[0068] Further, the step S224 includes: S2241. The trajectory generation module adds the values of each corresponding position in the target map and the obstacle avoidance map to obtain the passing cost of each position.

[0069] It should be understood that the value of the position where the interaction target is located in the target map is smaller, while the value of the position where the obstacle is located in the obstacle avoidance map is larger.

[0070] The value of each corresponding position in the obstacle avoidance map can be the x-axis and y-axis coordinates of each corresponding position in the obstacle avoidance map in the world coordinate system, and the z-axis coordinate is set to 0.5.

[0071] In other embodiments, the obtaining method of the value of each corresponding position in the target map can be the same as the obtaining method of the value of each corresponding position in the obstacle avoidance map.

[0072] S2242. The trajectory generation module performs trajectory planning based on the target position, the starting position, the passing cost of all positions, and the multiple action description information, and generates the trajectories of the human key points corresponding to each action description information.

[0073] Further, the trajectory generation module can adopt the optimize method to implement the step S2242.

[0074] Further, the optimize method can be a method in the SciPy library of Python (Python is a programming language).

[0075] The human key points refer to the root node of the human body and key points such as hands.

[0076] The trajectory of the human key points can be understood as the control information of each joint point of the human body. The control information of each joint point of the human body can be expressed as , where is the control information of each joint point of the human body, is the set of real numbers, is the number of each joint point of the human body.

[0077] Further, before the step of S2241, the step of S224 further includes: Judging whether the obstacle avoidance map is a null value. In the case where the obstacle avoidance map is a null value, the default voxel is used as the obstacle avoidance map. Further, after the step of S2242, the step of S224 further includes: S2243. Perform operations of removing zero elements and keeping the Z-axis constant at a first preset value on the trajectories of all human key points, and then use the processed trajectories of all human key points to execute step S23.

[0078] The step of S2243 can convert the generated trajectory into the data format required by the motion generation model.

[0079] In other embodiments, the 3D scene perception model can be replaced by a Neural Radiance Fields (NeRF).

[0080] S23. Input the multiple action description information and the trajectories of all human key points into the action generation model of the action planning model for action generation processing to obtain the human action sequence.

[0081] The action generation model is the generate_motion function. The generate_motion function is a function for generating human actions in Blender (Blender is an open-source 3D graphics software).

[0082] Further, the step of S23 includes: S231. Use the action interpolation method of the Human Motion Diffusion Model (MDM) in the action generation model to perform action generation processing on the target action description information and the trajectory of the human key points corresponding to the target action description information to obtain a sub-action sequence, where the target action description information is any one of the multiple action description information.

[0083] To enable seamless connection between two adjacent sub - action sequences, the human motion diffusion model replaces the first - frame action of the next sub - action sequence with the last - frame action of the current sub - action sequence.

[0084] S232. Merge all the sub - action sequences to obtain the human action sequence.

[0085] In the embodiment of this application, the action interpolation method is used to calculate the transformation matrix from the first frame to the last frame in each sub - action sequence, and the transformation matrix corresponding to each sub - action sequence is applied to the target action sequence generated later; after all the transformation matrices are applied to the target action sequence, the target action sequence is used as the human action sequence. Thus, all the sub - action sequences are merged into a complete long - action sequence.

[0086] The human action sequence is a complex human action sequence.

[0087] In other embodiments, the action generation model can use reinforcement learning methods to achieve similar functions.

[0088] In the embodiment of this application, text information and scene images are combined, enabling the action planning model to understand the environmental information around the human body. Thus, compared with the prior art, the embodiment of this application can enhance the action planning model's ability to understand environmental information, thereby improving the quality of human action generation and the generalization ability of the model.

[0089] In terms of generating long - term complex human actions, the embodiment of this application combines the task - decomposition ability of the large - language model. By decomposing complex actions into multiple sub - tasks and combining the 3D scene perception module (i.e., the 3D scene perception model) to automatically identify objects and scene positions related to the task, the embodiment enhances the action planning model's ability to understand environmental information. The embodiment of this application overcomes the problems of scarce training data and insufficient scene information in traditional methods and can generate more natural and accurate long - term complex action sequences. The embodiment of this application can be used for batch automated generation of human actions in industries such as games and animations.

[0090] In some embodiments, the text information can be expressed as , the scene image can be expressed as , the human action sequence can be expressed as , where represents a point cloud with RGB colors composed of points, is the set of real numbers, and

[0091] The embodiments of this application use the SMP-X model to represent human postures and shapes. The SMPL-X human mesh is parameterized by , where is the set of real numbers, is the global translation, is the continuous representation of the global orientation, is the body shape parameter, is the joint axis-angle rotation, is the number of human key points (which can also be understood as the number of human joints), is the differentiable linear skinning parameter. Usually, during implementation, the complete SMPL-X shape parameters are not directly generated, but the world coordinates of 22 human key points are generated , and then they are lifted to the complete parameters of SMPL-X. This task process can be expressed by the following formula: , is the th human key point, is the th frame action in the human action sequence.

[0092] In some embodiments, the method for generating the human action sequence further includes: S3. Optimize the action interpolation method using a gradient-guided model.

[0093] The embodiments of this application also calculate the distance between the generated joint points and the joint point control information, and calculate the sub-action sequence generated by the gradient-guided model that conforms to both the action description information and the spatial control information (i.e., the control information of each human joint point).

[0094] In some embodiments, the method for generating the human action sequence further includes the training step of an action planning model, and the training step of the action planning model includes: S24. Obtain a training data set and a real label set, where each training data has a one-to-one correspondence with one of the real labels in the real label set, and each training data includes text training information and scene training images.

[0095] S25. Input the target training data into the original action planning model for task decomposition and action generation processing, and output a predicted action sequence, where the target training data is any one of the training data in the training data set.

[0096] The original action planning model includes an original large language model, an original 3D scene perception model, and an original action generation model.

[0097] S26. Calculate a first loss function based on the predicted action sequence and the true label corresponding to the target training data.

[0098] The calculation formula of the first loss function is: ; where is the first loss function; is the second loss function; is the third loss function.

[0099] The calculation formula of the second loss function is: ; where is the L2 norm; is the predicted action sequence; is the true label corresponding to the target training data; is the number of frames of the predicted action sequence.

[0100] The calculation formula of the third loss function is: ; where is the L2 norm; is the speed of the predicted action sequence; is the speed of the true label corresponding to the target training data; is the number of frames of the predicted action sequence.

[0101] S27. Adjust the model parameters of the original action planning model based on the first loss function to obtain the action planning model.

[0102] Refer to Figure 7 shown, which is the principle block diagram of a human action sequence generation device provided in the second aspect of the embodiments of the present application. In Figure 7 , the human action sequence generation device 100 includes: An acquisition module 101, configured to acquire text information and a scene image; An output module 102, configured to input the text information and the scene image into an action planning model for task decomposition and action generation processing, and output a human action sequence.

[0103] The third aspect of the embodiments of the present application provides a terminal device, and the principle block diagram of the terminal device can be as Figure 8As shown. The terminal device includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected via a system bus. Among them, the processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating a human motion sequence is implemented. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor is pre-set inside the terminal device to detect the operating temperature of the internal device.

[0104] Those skilled in the art can understand that Figure 8 the block diagram of the principle shown only shows the block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0105] In some embodiments, the present application provides a terminal device, which includes a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the steps of the method for generating a human motion sequence provided in the first aspect of the present application.

[0106] In the fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium is used to store a computer program, and the computer program causes a computer to execute the steps of the method for generating a human motion sequence provided in the first aspect of the present application.

[0107] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0108] Without changing the basic principle of the present application, the technical features of the above embodiments can be combined. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0109] The above embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for generating a human body motion sequence, characterized in that: include: Get text information and scene images; The text information and the scene image are input into an action planning model for task decomposition and action generation processing, and a human action sequence is output.

2. The method for generating a human body motion sequence according to claim 1, characterized in that: The steps of inputting the text information and the scene image into the action planning model for task decomposition and action generation processing, and outputting a human action sequence include: Inputting the text information into the large language model of the action planning model for task decomposition processing to obtain multiple action description information; Inputting the scene image and the plurality of action description information into the 3D scene perception model of the action planning model for trajectory planning processing to obtain the trajectory of the key points of the human body corresponding to each action description information; The plurality of action description information and the trajectories of all human body key points are input into the action generation model of the action planning model for action generation processing to obtain the human body action sequence.

3. The method for generating a human body motion sequence according to claim 2, characterized in that: The step of inputting the text information into the large language model of the action planning model for task decomposition processing to obtain a plurality of action description information comprises: Inputting the text information into the task planner of the large language model for task decomposition processing to obtain multiple subtasks; The multiple subtasks are input into the subtask executor of the large language model for action recognition processing to obtain the multiple action description information.

4. The method for generating a human body motion sequence according to claim 2, characterized in that: The step of inputting the scene image and the plurality of action description information into the 3D scene perception model of the action planning model for trajectory planning processing, and obtaining the trajectory of the key points of the human body corresponding to each action description information comprises: Inputting the scene image into the query object module of the 3D scene perception model for recognition processing to obtain point cloud data, target position and starting position; Inputting the point cloud data into the target map generation module of the 3D scene perception model for extraction processing to generate a target map; Inputting the point cloud data into the obstacle avoidance map generation module of the 3D scene perception model for extraction and processing to generate an obstacle avoidance map; The target position, the starting position, the target map, the obstacle avoidance map and the multiple action description information are input into the trajectory generation module of the 3D scene perception model for trajectory planning, so as to generate the trajectory of the key points of the human body corresponding to each action description information.

5. The method for generating a human body motion sequence according to claim 2, characterized in that: The steps of inputting the plurality of action description information and the trajectories of all human key points into the action generation model of the action planning model for action generation processing, and obtaining the human action sequence include: Using the action interpolation method of the human motion diffusion model in the action generation model, target action description information and the trajectory of the human body key points corresponding to the target action description information are subjected to action generation processing to obtain a sub-action sequence, wherein the target action description information is any one of the multiple action description information; All sub-action sequences are combined to obtain the human body action sequence.

6. The method for generating a human body motion sequence according to claim 5, characterized in that: The method for generating the human body motion sequence also includes: A gradient-guided model is used to optimize the motion interpolation method.

7. The method for generating a human body motion sequence according to claim 1, characterized in that: The method for generating a human motion sequence further includes a step of training a motion planning model, and the step of training the motion planning model includes: Obtaining a training data set and a real label set, wherein each training data has a one-to-one correspondence with one of the real labels in the real label set, and each training data includes text training information and scene training images; Inputting target training data into the original action planning model for task decomposition and action generation processing, and outputting a predicted action sequence, wherein the target training data is any training data in the training data set; Calculate a first loss function based on the predicted action sequence and the true label corresponding to the target training data; The model parameters of the original action planning model are adjusted based on the first loss function to obtain the action planning model.

8. A device for generating a human body motion sequence, characterized in that: include: An acquisition module, used to acquire text information and scene images; The output module is used to input the text information and the scene image into the action planning model to perform task decomposition and action generation processing, and output a human action sequence.

9. A terminal device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the steps of the method for generating a human body motion sequence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the steps of the method for generating a human body motion sequence as described in any one of claims 1 to 7.

Images