Method and system for generating intelligent motion of body and electronic equipment

Abstract

The invention provides an intelligent motion generation method, an intelligent motion generation system, electronic equipment and a computer readable storage medium, in the intelligent motion generation method, motion feature data are mapped and aligned to a virtual human skeleton structure, a preliminary motion sequence is obtained, and in addition, the motion feature data are mapped and aligned to the virtual human skeleton structure; and according to the virtual human skeleton structure and the preliminary action sequence, optimizing the action of the virtual human skeleton structure, and outputting an optimized action sequence, so that after the preliminary action sequence is obtained according to a redirection strategy based on the frame sequence of the video of the human body motion, the motion of the virtual human skeleton structure is optimized. And an optimized action sequence obtained by optimizing the preliminary action sequence can be more matched with the body shape of the virtual human.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D generation technology, and in particular to an embodied intelligent motion generation method, an embodied intelligent motion generation system, an electronic device, and a computer-readable storage medium. Background Technology

[0002] The motion generation of embodied intelligent agents relies on motion redirection strategies based on reference videos. Its core goal is to transfer human motion patterns in natural scenes to virtual characters or physical robots, thereby achieving high-fidelity, interactive motion performance.

[0003] However, the practicality and stability of action generation by embodied agents in complex tasks are still limited. For example, the generated actions violate human kinematics constraints, the actions are abrupt due to a lack of smooth transitions, the training data is highly dependent, and it is difficult to adapt to multi-role interactions and physical constraints. Summary of the Invention

[0004] This invention provides an embodied intelligent motion generation method, an embodied intelligent motion generation system, an electronic device, and a computer-readable storage medium to improve the practicality and stability of embodied intelligent agent motion generation in complex tasks.

[0005] According to a first aspect of the present invention, the present invention provides an embodied intelligent motion generation method, comprising: Provides frame sequences of videos showing human movement; Based on the frame sequence, obtain action feature data; Based on the aforementioned motion feature data, a virtual human image is obtained using a generative visual language model. Obtain the virtual human skeleton structure based on the virtual human image; The motion feature data is mapped and aligned to the virtual human skeleton structure to obtain a preliminary motion sequence; Based on the virtual human skeleton structure and the preliminary action sequence, the actions of the virtual human skeleton structure are optimized, and an optimized action sequence is output. Based on the optimized action sequence and the virtual human image, a synthetic image sequence is obtained; Based on the synthesized image sequence, control commands are obtained; According to the control instructions, the target intelligent agent is controlled to perform the specified action.

[0006] Optionally, based on the virtual human skeleton structure and the initial action sequence, the method for optimizing the actions of the virtual human skeleton structure and outputting the optimized action sequence includes: Based on the virtual human skeleton structure and the preliminary action sequence, the physical rationality of the joint space of the virtual human skeleton structure is verified and corrected to output the first action sequence; Based on time continuity modeling, the action switching points in the first action sequence are compensated to output the second action sequence; Based on the second action sequence and the virtual human skeleton structure, local feature reconstruction is performed on the detailed parts of the virtual human skeleton structure to output the optimized action sequence.

[0007] Optionally, the method for optimizing the movements of the virtual human skeleton structure and outputting the optimized movement sequence based on the virtual human skeleton structure and the preliminary movement sequence further includes: Based on the frame sequence, environmental semantic information is obtained; Based on the environmental semantic information, the optimized action sequence is adjusted and a third action sequence is output to adjust the action trajectory of the virtual human skeleton structure.

[0008] Optionally, before obtaining the motion feature data based on the frame sequence, the method further includes: Obtain the structural parameters of the target agent; Align the coordinate system of the video frame sequence with the coordinate system of the target agent, and perform pose normalization processing on the target agent and the human pose in the video.

[0009] Optionally, the method for mapping and aligning the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence includes: Based on the motion feature data and the virtual human skeleton structure, an initial motion sequence is obtained through affine transformation; A preliminary action sequence is obtained through nonlinear deformation mapping based on the original preliminary action sequence.

[0010] Optionally, the method for obtaining motion feature data based on the frame sequence includes: Based on the frame sequence, obtain motion capture data; Initial motion feature data are obtained by interpolating motion capture data using a thin plate spline deformation function. The initial action feature data is corrected by using a structural prior constraint function to obtain action feature data.

[0011] Optionally, the method for obtaining the synthesized image sequence based on the optimized action sequence and the virtual human image includes: Semantic extraction is performed on the virtual human image, and vectors are embedded to obtain an image feature sequence; Based on the optimized action sequence and the image feature sequence, an optical flow map and a motion mask are generated; Based on the optical flow map and the motion mask, the synthesized image sequence is obtained by synthesizing the virtual human image using an image synthesizer.

[0012] According to a second aspect of the present invention, an embodied intelligent motion generation system is provided, comprising: The motion feature data acquisition module is used to acquire motion feature data from the frame sequence of human motion videos. The virtual human image acquisition module is used to acquire virtual human images based on the motion feature data and through a generative visual language large model. The virtual human skeleton structure acquisition module is used to acquire the virtual human skeleton structure based on the virtual human image; The redirection module is used to map and align the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence. The post-redirection optimization module is used to optimize the movements of the virtual human skeleton structure based on the virtual human skeleton structure and the initial movement sequence, and output the optimized movement sequence. The data synthesis module is used to obtain a synthesized image sequence based on the optimized action sequence and the virtual human image; An action mapping module is used to obtain control commands based on the synthesized image sequence; An intelligent agent controller is used to control a target intelligent agent to perform a specified action according to the control instructions.

[0013] Optional, the post-redirect optimization modules include: The local constraint repair submodule is used to perform physical rationality verification and correction on the joint space of the virtual human skeleton structure based on the virtual human skeleton structure and the preliminary action sequence, so as to output the first action sequence. The transition smoothing submodule is used for time continuity modeling to compensate for action switching points in the first action sequence in order to output the second action sequence. The detail enhancement submodule is used to reconstruct local features of the detailed parts of the virtual human skeleton structure based on the second action sequence and the virtual human skeleton structure, so as to output an optimized action sequence.

[0014] Optionally, the post-redirection optimization module further includes: a scene-aware correction submodule, used to obtain environmental semantic information based on the frame sequence, and the scene-aware correction submodule is also used to adjust the optimized action sequence based on the environmental semantic information and output a third action sequence to adjust the action trajectory of the virtual human skeleton structure.

[0015] Optionally, the action feature data acquisition module includes: The motion capture submodule is used to acquire motion capture data based on the frame sequence; The key point detector is used to interpolate motion capture data using a thin-plate spline deformation function to obtain initial motion feature data. The human body structure prior submodule is used to correct the initial motion feature data and obtain motion feature data.

[0016] Optionally, the motion feature data acquisition module further includes a preprocessing submodule, used to acquire the structural parameters of the target agent, and also used to align the coordinate system of the frame sequence of the video with the coordinate system of the target agent. The preprocessing module is also used to perform pose normalization processing on the target agent and the human pose in the video.

[0017] Optionally, the data synthesis module includes: An image encoder is used to extract semantics from the virtual human image and embed vectors to obtain an image feature sequence; A motion generator is used to generate an optical flow map and a motion mask based on the optimized action sequence and the image feature sequence; An image synthesizer is used to synthesize the synthesized image sequence based on the optical flow map and the motion mask, on the basis of the virtual human image.

[0018] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that the processor, when executing the computer program, implements the above-described embodied intelligent motion generation method.

[0019] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, characterized in that the computer program, when executed by a processor, implements the above-described embodied intelligent motion generation method.

[0020] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects: In the embodied intelligent motion generation method of the present invention, since the motion feature data is mapped and aligned to the virtual human skeleton structure to obtain a preliminary motion sequence, and the motion of the virtual human skeleton structure is optimized based on the virtual human skeleton structure and the preliminary motion sequence to output an optimized motion sequence, the optimized motion sequence obtained after optimizing the preliminary motion sequence in the frame sequence of the video based on human motion can better match the body shape of the virtual human.

[0021] Furthermore, by performing physical rationality checks and corrections on the joint space of the virtual human skeleton structure based on the initial action sequence to output the first action sequence, distortion problems such as limb overlap and joint angle overshooting in the virtual human skeleton structure's movements generated from the initial action sequence are resolved. Next, by compensating for action transition points in the first action sequence based on temporal continuity modeling to output the second action sequence, the abrupt transitions caused by jumps and breaks in the virtual human skeleton structure's movements generated from the first action sequence are eliminated, thereby improving the naturalness of the virtual human's rhythmic movements. Then, by reconstructing local features of the detailed parts of the virtual human skeleton structure to output the optimized action sequence, the movement details of the generated virtual human skeleton structure are restored, making the movements of the generated virtual human skeleton structure more realistic. In summary, optimizing the movements of the virtual human skeleton structure includes correcting the virtual human's posture in the temporal dimension, enhancing the continuity of the virtual human's movements, and enhancing the local features of the virtual human, making the movements of the virtual human generated by the optimized action sequence more natural and realistic, thus making the movements of the target intelligent agent corresponding to the optimized action sequence more natural and realistic.

[0022] Furthermore, since the third action sequence is output after adjusting and optimizing the action sequence based on environmental semantic information, the action trajectory of the virtual human skeleton structure is adjusted, thus enabling the actions of the virtual human skeleton structure corresponding to the third action sequence to meet real-world logic such as obstacle avoidance and collaborative synchronization.

[0023] Furthermore, since the process of obtaining action feature data based on the frame sequence includes: acquiring the structural parameters of the target agent; aligning the coordinate system of the video frame sequence with the coordinate system of the target agent; and performing pose normalization processing on the target agent and the human posture in the video, problems such as action jumps and posture imbalances caused by differences in body shape or changes in perspective are effectively eliminated. This makes the generated embodied agent's movements more consistent with human kinematics. In addition, since pose normalization processing on the target agent and the human posture in the video is completed before obtaining action feature data, the learning burden of redundant motion actions on the generative visual language model during the training phase is reduced, thereby reducing computer resource consumption and improving the response speed of embodied intelligent motion generation. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the embodied intelligent motion generation method provided in the embodiments of the present invention; Figure 2 This is a flowchart of a method for acquiring motion feature data provided in an embodiment of the present invention; Figure 3 This is a flowchart of a method for obtaining a preliminary action sequence provided in an embodiment of the present invention; Figure 4 This is a flowchart of a method for obtaining an optimized action sequence provided in an embodiment of the present invention; Figure 5 This is a flowchart of a method for obtaining a synthetic image sequence provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the embodied intelligent motion generation system provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the redirection optimization module provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the motion feature data acquisition module provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the data synthesis module provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] As mentioned in the background section, the practicality and stability of action generation for embodied intelligent agents in complex tasks remain limited.

[0027] In view of this, the present invention provides an embodied intelligent motion generation method, comprising: providing a frame sequence of a video of human motion; acquiring motion feature data based on the frame sequence; acquiring a virtual human image based on the motion feature data using a generative visual language large model; acquiring a virtual human skeleton structure based on the virtual human image; mapping and aligning the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence; optimizing the motion of the virtual human skeleton structure based on the virtual human skeleton structure and the preliminary motion sequence, and outputting an optimized motion sequence; acquiring a synthesized image sequence based on the optimized motion sequence and the virtual human image; acquiring control instructions based on the synthesized image sequence; and controlling a target intelligent agent to perform a specified action based on the control instructions.

[0028] In a video frame sequence based on human motion, after obtaining a preliminary motion sequence according to a redirection strategy, the optimized motion sequence obtained by optimizing the preliminary motion sequence can better match the body shape of the virtual human.

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0032] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Please refer to Figure 1 This invention provides a method for generating embodied intelligent motion, comprising: Step S1: Provide a frame sequence of the video of human motion; Step S2: Obtain motion feature data based on the frame sequence; Step S3: Based on the motion feature data, obtain the virtual human image through a generative visual language model; Step S4: Obtain the virtual human skeleton structure based on the virtual human image; Step S5: Map and align the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence; Step S6: Based on the virtual human skeleton structure and the preliminary action sequence, optimize the actions of the virtual human skeleton structure and output the optimized action sequence; Step S7: Obtain a synthesized image sequence based on the optimized action sequence and the virtual human image; Step S8: Obtain control commands based on the synthesized image sequence; Step S9: Control the target intelligent agent to perform the specified action according to the control command.

[0034] The following provides a detailed explanation of each step in the motion generation method described above.

[0035] For step S1, the frame sequence is a set of frames arranged in the temporal order of the human motion video. Each frame is an independent image, which includes spatial information of the scene, such as the position of the human body, the human body's movements, and environmental semantic information.

[0036] In this embodiment, before step S2, the method further includes: obtaining the structural parameters of the target intelligent agent; aligning the coordinate system of the video frame sequence with the coordinate system of the target intelligent agent; and performing pose normalization processing on the target intelligent agent and the human pose in the video.

[0037] The structural parameters of the target AI agent can include, for example, bone length, joint degrees of freedom, and initial posture. Furthermore, coordinate system unification and posture normalization effectively eliminate problems such as motion jumps and posture imbalances caused by differences in body shape or perspective between the human body in the video and the target AI agent, thus making the generated embodied AI agent's movements more consistent with human kinematics. In addition, since posture normalization between the target AI agent and the human body in the video is performed before acquiring motion feature data, the learning burden of redundant motion actions on the generative visual language model during the training phase is reduced, thereby reducing computer resource consumption and improving the response speed of embodied AI motion generation.

[0038] For step S2, please refer to Figure 2 Methods for obtaining motion feature data based on frame sequences include: Step S21: Obtain motion capture data based on the frame sequence; Specifically, this step is used to locate and separate the human subject from each frame of image to eliminate background interference. The human subject may include, for example, the human body or the object to be manipulated. Therefore, motion capture data is used to characterize the relevant motion features of the human body.

[0039] Step S22: Interpolate the motion capture data using the thin plate spline deformation function to obtain initial motion feature data; Among them, motion capture data often suffers from problems such as missing data, low resolution, and inconsistent formats. However, the interpolation processing of thin plate spline deformation function can transform imperfect motion capture data into high-quality initial motion feature data with continuous, smooth, and standardized characteristics by filling in missing data, improving accuracy, and standardizing formats.

[0040] Step S23: The initial motion feature data is corrected using a structural prior constraint function to obtain motion feature data. Specifically, the structural prior constraint function is generated based on human anatomical invariants (e.g., bone length and limb symmetry) and is used to correct the global skeleton proportions in the initial motion feature data to ensure that the generated motion feature sequence corresponds to actions that conform to human structure.

[0041] For step S5, please refer to... Figure 3 Methods for mapping and aligning motion feature data to a virtual human skeleton structure to obtain preliminary motion sequences include: Step S51: Based on the motion feature data and the virtual human skeleton structure, obtain the original preliminary motion sequence through affine transformation; Step S52: Obtain a preliminary action sequence through nonlinear deformation mapping using the original preliminary action sequence.

[0042] Among them, the combination of affine transformation and nonlinear deformation mapping can achieve efficient pose alignment in scenes with different body shapes and styles. Moreover, this method does not require manual annotation or strict data pairing, which improves the versatility and robustness of the action generation method in multi-agent or multi-task environments.

[0043] For step S6, please refer to Figure 4 Based on the virtual human skeleton structure and the initial action sequence, the methods for optimizing the actions of the virtual human skeleton structure and outputting the optimized action sequence include: Step S61: Based on the virtual human skeleton structure and the preliminary action sequence, perform physical rationality verification and correction on the joint space of the virtual human skeleton structure to output the first action sequence. Specifically, the physical rationality verification and correction of the joint space of the virtual human skeleton structure in this step includes: based on the spatial relative positions of each limb component of the virtual human skeleton structure in the current posture, detecting and repairing local geometric conflicts in the movement of the virtual human skeleton structure, such as instantaneous clipping distortions such as hands penetrating the torso and legs crossing each other.

[0044] Step S62: Based on time continuity modeling, compensate for the action switching points in the first action sequence to output the second action sequence; This step eliminates the abrupt transitions caused by jumps and breaks in the movements of the virtual human skeleton structure generated in the first action sequence, thereby improving the naturalness of the rhythm of the virtual human's movements.

[0045] Step S63: Based on the second action sequence and the virtual human skeleton structure, perform local feature reconstruction on the detailed parts of the virtual human skeleton structure to output an optimized action sequence.

[0046] This step allows the motion details of the generated virtual human skeleton structure to be restored, thus making the motion of the generated virtual human skeleton structure more realistic.

[0047] Furthermore, after step S63, the following is also included: Step S64: Obtain environmental semantic information based on the frame sequence; Step S65: Based on the environmental semantic information, adjust and optimize the action sequence and output the third action sequence to adjust the action trajectory of the virtual human skeleton structure.

[0048] Step S65 enables the virtual human skeleton structure corresponding to the third action sequence to perform actions that satisfy real-world logic such as obstacle avoidance and collaborative synchronization.

[0049] For step S7, please refer to Figure 5 Methods for obtaining synthetic image sequences based on optimized action sequences and virtual human images include: Step S71: Semantic extraction is performed on the virtual human image, and vectors are embedded to obtain the image feature sequence; Step S72: Generate an optical flow map and a motion mask based on the optimized action sequence and image feature sequence; Step S73: Based on the optical flow map and motion mask, a synthetic image sequence is obtained by synthesizing the virtual human image using an image synthesizer.

[0050] In the embodied intelligent motion generation method provided in this embodiment of the invention, since the motion feature data is mapped and aligned to the virtual human skeleton structure to obtain a preliminary motion sequence, and the motion of the virtual human skeleton structure is optimized based on the virtual human skeleton structure and the preliminary motion sequence to output an optimized motion sequence, the optimized motion sequence obtained after optimizing the preliminary motion sequence in the frame sequence of the video based on human motion can better match the body shape of the virtual human.

[0051] Furthermore, by performing physical rationality checks and corrections on the joint space of the virtual human skeleton structure based on the initial action sequence to output the first action sequence, distortion problems such as limb overlap and joint angle overshooting in the virtual human skeleton structure's movements generated from the initial action sequence are resolved. Next, by compensating for action transition points in the first action sequence based on temporal continuity modeling to output the second action sequence, the abrupt transitions caused by jumps and breaks in the virtual human skeleton structure's movements generated from the first action sequence are eliminated, thereby improving the naturalness of the virtual human's rhythmic movements. Then, by reconstructing local features of the detailed parts of the virtual human skeleton structure to output the optimized action sequence, the movement details of the generated virtual human skeleton structure are restored, making the movements of the generated virtual human skeleton structure more realistic. In summary, optimizing the movements of the virtual human skeleton structure includes correcting the virtual human's posture in the temporal dimension, enhancing the continuity of the virtual human's movements, and enhancing the local features of the virtual human, making the movements of the virtual human generated by the optimized action sequence more natural and realistic, thus making the movements of the target intelligent agent corresponding to the optimized action sequence more natural and realistic.

[0052] Furthermore, since the third action sequence is output after adjusting and optimizing the action sequence based on environmental semantic information, the action trajectory of the virtual human skeleton structure is adjusted, thus enabling the actions of the virtual human skeleton structure corresponding to the third action sequence to meet real-world logic such as obstacle avoidance and collaborative synchronization.

[0053] Furthermore, since the process of obtaining action feature data based on the frame sequence includes: acquiring the structural parameters of the target agent; aligning the coordinate system of the video frame sequence with the coordinate system of the target agent; and performing pose normalization processing on the target agent and the human posture in the video, problems such as action jumps and posture imbalances caused by differences in body shape or changes in perspective are effectively eliminated. This makes the generated embodied agent's movements more consistent with human kinematics. In addition, since pose normalization processing on the target agent and the human posture in the video is completed before obtaining action feature data, the learning burden of redundant motion actions on the generative visual language model during the training phase is reduced, thereby reducing computer resource consumption and improving the response speed of embodied intelligent motion generation.

[0054] Please refer to Figure 6 The present invention also provides an embodied intelligent motion generation system, including: a motion feature data acquisition module, a virtual human image acquisition module, a virtual human skeleton structure acquisition module, a redirection module, a post-redirection optimization module, a data synthesis module, a motion mapping module, and an intelligent agent controller.

[0055] The motion feature data acquisition module is used to acquire motion feature data from the frame sequence of human motion videos.

[0056] The virtual human image acquisition module is used to acquire virtual human images based on motion feature data and a generative visual language model.

[0057] The virtual human skeleton structure acquisition module is used to acquire the virtual human skeleton structure based on the virtual human image.

[0058] The redirection module is used to map and align motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence.

[0059] The post-redirection optimization module is used to optimize the actions of the virtual human skeleton structure based on the virtual human skeleton structure and the initial action sequence, and output the optimized action sequence.

[0060] The data synthesis module is used to obtain a synthesized image sequence based on the optimized action sequence and the virtual human image.

[0061] The motion mapping module is used to obtain control commands based on the synthesized image sequence.

[0062] The intelligent agent controller is used to control the target intelligent agent to perform specified actions according to control instructions.

[0063] In this embodiment, please refer to Figure 7 The post-redirect optimization module includes a local constraint repair submodule, a transition smoothing submodule, and a detail enhancement submodule. The local constraint repair submodule performs physical rationality verification and correction on the joint space of the virtual human skeleton structure based on the virtual human skeleton structure and the initial action sequence, to output a first action sequence. The transition smoothing submodule compensates for action switching points in the first action sequence based on time continuity modeling, to output a second action sequence. The detail enhancement submodule reconstructs local features of the detailed parts of the virtual human skeleton structure based on the second action sequence and the virtual human skeleton structure, to output an optimized action sequence.

[0064] Furthermore, the post-redirection optimization module also includes a scene-aware correction submodule. This submodule acquires environmental semantic information based on the frame sequence and adjusts and optimizes the action sequence based on this information before outputting a third action sequence to adjust the motion trajectory of the virtual human skeleton structure.

[0065] In this embodiment, please refer to Figure 8The motion feature data acquisition module includes a motion capture submodule, a keypoint detector, and a human structure prior submodule. The motion capture submodule acquires motion capture data based on frame sequences; the keypoint detector interpolates the motion capture data using a thin-plate spline deformation function to obtain initial motion feature data; and the human structure prior submodule corrects the initial motion feature data to obtain the final motion feature data.

[0066] Furthermore, the motion feature data acquisition module also includes a preprocessing submodule. This preprocessing submodule is used to acquire the structural parameters of the target agent, align the coordinate system of the video frame sequence with the coordinate system of the target agent, and perform pose normalization processing on the target agent and the human pose in the video.

[0067] In this embodiment, please refer to Figure 9 The data synthesis module includes an image encoder, a motion generator, and an image synthesizer. The image encoder extracts semantics from the virtual human image and embeds vectors to obtain an image feature sequence. The motion generator generates optical flow maps and motion masks based on the optimized action sequence and image feature sequence. The image synthesizer synthesizes a synthetic image sequence based on the optical flow map and motion mask, using these as the basis for the virtual human image.

[0068] Please refer to Figure 10 The present invention also provides an electronic device, which includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the above-described embodied intelligent motion generation method.

[0069] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described embodied intelligent motion generation method.

[0070] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for generating embodied intelligent motion, characterized in that, include: Provides frame sequences of videos showing human movement; Based on the frame sequence, obtain action feature data; Based on the aforementioned motion feature data, a virtual human image is obtained using a generative visual language model. Obtain the virtual human skeleton structure based on the virtual human image; The motion feature data is mapped and aligned to the virtual human skeleton structure to obtain a preliminary motion sequence; Based on the virtual human skeleton structure and the preliminary action sequence, the actions of the virtual human skeleton structure are optimized, and an optimized action sequence is output. Based on the optimized action sequence and the virtual human image, a synthetic image sequence is obtained; Based on the synthesized image sequence, control commands are obtained; According to the control instructions, the target intelligent agent is controlled to perform the specified action.

2. The embodied intelligent motion generation method according to claim 1, characterized in that, The method for optimizing the movements of the virtual human skeleton structure and outputting the optimized movement sequence based on the virtual human skeleton structure and the preliminary movement sequence includes: Based on the virtual human skeleton structure and the preliminary action sequence, the physical rationality of the joint space of the virtual human skeleton structure is verified and corrected to output the first action sequence; Based on time continuity modeling, the action switching points in the first action sequence are compensated to output the second action sequence; Based on the second action sequence and the virtual human skeleton structure, local feature reconstruction is performed on the detailed parts of the virtual human skeleton structure to output the optimized action sequence.

3. The embodied intelligent motion generation method according to claim 2, characterized in that, The method for optimizing the movements of the virtual human skeleton structure and outputting the optimized movement sequence based on the virtual human skeleton structure and the preliminary movement sequence further includes: Based on the frame sequence, environmental semantic information is obtained; Based on the environmental semantic information, the optimized action sequence is adjusted and a third action sequence is output to adjust the action trajectory of the virtual human skeleton structure.

4. The embodied intelligent motion generation method according to claim 1, characterized in that, Before obtaining the action feature data based on the frame sequence, the process further includes: Obtain the structural parameters of the target agent; Align the coordinate system of the video frame sequence with the coordinate system of the target agent, and perform pose normalization processing on the target agent and the human pose in the video.

5. The embodied intelligent motion generation method according to claim 1, characterized in that, The method for mapping and aligning the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence includes: Based on the motion feature data and the virtual human skeleton structure, an initial motion sequence is obtained through affine transformation; A preliminary action sequence is obtained through nonlinear deformation mapping based on the original preliminary action sequence.

6. The embodied intelligent motion generation method according to claim 1, characterized in that, The method for obtaining action feature data based on the frame sequence includes: Based on the frame sequence, obtain motion capture data; Initial motion feature data are obtained by interpolating motion capture data using a thin plate spline deformation function. The initial action feature data is corrected by using a structural prior constraint function to obtain action feature data.

7. The embodied intelligent motion generation method according to claim 1, characterized in that, The method for obtaining a synthetic image sequence based on the optimized action sequence and the virtual human image includes: Semantic extraction is performed on the virtual human image, and vectors are embedded to obtain an image feature sequence; Based on the optimized action sequence and the image feature sequence, an optical flow map and a motion mask are generated; Based on the optical flow map and the motion mask, the synthesized image sequence is obtained by synthesizing the virtual human image using an image synthesizer.

8. A body-integrated intelligent motion generation system, characterized in that, include: The motion feature data acquisition module is used to acquire motion feature data from the frame sequence of human motion videos. The virtual human image acquisition module is used to acquire virtual human images based on the motion feature data and through a generative visual language large model. The virtual human skeleton structure acquisition module is used to acquire the virtual human skeleton structure based on the virtual human image; The redirection module is used to map and align the motion feature data to the virtual human skeleton structure to obtain a preliminary motion sequence. The post-redirection optimization module is used to optimize the movements of the virtual human skeleton structure based on the virtual human skeleton structure and the initial movement sequence, and output the optimized movement sequence. The data synthesis module is used to obtain a synthesized image sequence based on the optimized action sequence and the virtual human image; An action mapping module is used to obtain control commands based on the synthesized image sequence; An intelligent agent controller is used to control a target intelligent agent to perform a specified action according to the control instructions.

9. The embodied intelligent motion generation system according to claim 8, characterized in that, The optimization module after redirection includes: The local constraint repair submodule is used to perform physical rationality verification and correction on the joint space of the virtual human skeleton structure based on the virtual human skeleton structure and the preliminary action sequence, so as to output the first action sequence. The transition smoothing submodule is used for time continuity modeling to compensate for action switching points in the first action sequence in order to output the second action sequence. The detail enhancement submodule is used to reconstruct local features of the detailed parts of the virtual human skeleton structure based on the second action sequence and the virtual human skeleton structure, so as to output an optimized action sequence.

10. The embodied intelligent motion generation system according to claim 9, characterized in that, The post-redirect optimization module further includes a scene-aware correction submodule, which is used to obtain environmental semantic information based on the frame sequence. The scene-aware correction submodule is also used to adjust the optimized action sequence based on the environmental semantic information and output a third action sequence to adjust the action trajectory of the virtual human skeleton structure.

11. The embodied intelligent motion generation system according to claim 8, characterized in that, The action feature data acquisition module includes: The motion capture submodule is used to acquire motion capture data based on the frame sequence; The key point detector is used to interpolate motion capture data using a thin-plate spline deformation function to obtain initial motion feature data. The human body structure prior submodule is used to correct the initial motion feature data and obtain motion feature data.

12. The embodied intelligent motion generation system according to claim 11, characterized in that, The motion feature data acquisition module further includes a preprocessing submodule, which is used to acquire the structural parameters of the target agent and to align the coordinate system of the video frame sequence with the coordinate system of the target agent. The preprocessing module is also used to perform pose normalization processing on the target agent and the human pose in the video.

13. The embodied intelligent motion generation system according to claim 8, characterized in that, The data synthesis module includes: An image encoder is used to extract semantics from the virtual human image and embed vectors to obtain an image feature sequence; A motion generator is used to generate an optical flow map and a motion mask based on the optimized action sequence and the image feature sequence; An image synthesizer is used to synthesize the synthesized image sequence based on the optical flow map and the motion mask, on the basis of the virtual human image.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the embodied intelligent motion generation method as described in any one of claims 1-7.

15. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the embodied intelligent motion generation method as described in any one of claims 1-7.

Images