An AIGC video generation method and system based on motion capture data driving

Abstract

The application discloses an AIGC video generation method and system based on motion capture data driving, and belongs to the technical field of artificial intelligence generated content. The method constructs a three-dimensional action database based on standardized motion capture data, obtains candidate three-dimensional action data according to target action requirements, and determines target three-dimensional action data based on human motion constraints and action semantic matching relationships. According to target observation angles, key action areas and action information retention requirements, virtual camera parameters are adjusted, target three-dimensional action data is converted into a two-dimensional posture control sequence, consistency checking and abnormal correction are performed according to three-dimensional space motion relationships, human structure adaptation is performed in combination with target character appearance characteristics, video generation control information is obtained through fusion, and a video generation model is input to generate a target video. The application can maintain consistent transmission of three-dimensional action information in an AIGC video generation link, improve the spatial structure retention capability, motion continuity and multi-view expression consistency of character actions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-generated content technology, specifically to an AIGC video generation method and system based on motion capture data, wherein AIGC refers to artificial intelligence-generated content.

[0002] The motion capture data in this invention is used to characterize the three-dimensional spatial position of human skeletal joints, joint rotation state, and their relationship with time changes; after coordinate normalization, human skeletal model unification, and motion time scale standardization, the motion capture data forms three-dimensional motion data used to drive video generation. Background Technology

[0003] With the development of AI-generated content technology, video generation technology based on deep learning models is gradually being applied to scenarios such as film and television production, virtual digital humans, game animation generation, short video creation, and sports action demonstrations. In recent years, video generation models have continued to improve in terms of image quality, temporal consistency, multimodal fusion, audio-visual synchronization, and the ability to follow prompts. The industry's development direction is gradually shifting from "generating video content" to "high-quality, controllable, reusable, and finely interactive video generation."

[0004] In video generation schemes where human movements are controllable, motion control information is typically required to enable the characters in the generated video to perform specified actions. Common existing methods include motion transfer based on 2D reference videos and motion control based on 2D human pose sequences. These methods generally first acquire a human video containing the target action, then use a human pose estimation algorithm to extract keypoint information from the video frames, forming a 2D skeletal pose sequence. This 2D skeletal pose sequence is then input into the video generation model to control the characters in the generated video to perform corresponding actions.

[0005] However, two-dimensional pose data mainly represents the positional relationship of human body key points in the image plane, lacking depth information, spatial rotation information, and true motion relationships of human body joints in three-dimensional space. When the human body performs forward and backward movement, body rotation, limb crossing, dance, gymnastics, martial arts, or other high-precision continuous movements, two-dimensional key points cannot fully express the true motion state of the human body, which can easily lead to problems such as limb deformation, deviation in movement amplitude, limb crossing, motion stuttering, and discontinuous movement sequence in the generated video.

[0006] Meanwhile, two-dimensional motion reference data is usually bound to the original shooting perspective, creating a coupling relationship between motion information and the viewing angle. When users need to generate videos from different shooting angles based on the same action, they typically need to reacquire motion reference materials from the corresponding perspective or repeatedly generate videos for different perspectives, making it difficult to flexibly adjust the viewing angle while maintaining consistency in the motion content. For scenarios such as film and television storyboarding, sports instruction, virtual choreography, and game animation that require multiple cameras to display the same action, the above method reduces the efficiency of motion data reuse and increases the cost of post-production adjustments and repeated generation.

[0007] Furthermore, specialized movements in professional dance, martial arts, and competitive sports typically involve complex physical coordination and clear spatial motion patterns, making it difficult to obtain suitable 2D video footage for reference. Even if approximate movement footage is available, its shooting angle, human body shape, aspect ratio, and movement rhythm may not match the target generation requirements. This makes it difficult to directly adapt the extracted 2D pose sequences to the target person and shot, affecting the accuracy and naturalness of the movements in the generated video.

[0008] With the development of 3D motion capture technology, optical motion capture devices, inertial motion capture devices, or human 3D pose estimation devices can be used to acquire the positional changes and rotational states of human skeletal joints in 3D space, providing a data foundation for accurately describing human movement. Compared to 2D pose sequences, 3D motion data can simultaneously express the spatial position, depth changes, joint rotational relationships, and continuous motion patterns of human movements, theoretically making it more suitable for high-precision human motion control.

[0009] However, existing video generation solutions based on motion capture data or 3D motion data typically only use 3D motion data as a rendering object or model input condition, lacking a mechanism for preserving motion information throughout the video generation process. Specifically, in the motion data selection stage, existing solutions struggle to determine whether the selected 3D motion data simultaneously satisfies the target motion semantics and human motion laws; in the perspective transition stage, existing solutions usually only perform projection processing based on the camera parameters input by the user, failing to adequately determine whether the target perspective can fully express the key motion areas and motion trajectories; in the 3D motion to 2D pose control information conversion stage, existing solutions mostly use simple coordinate projection, lacking the detection and correction of consistency between 3D motion laws and 2D pose changes; and in the character appearance fusion stage, existing solutions are also prone to problems such as facial misalignment, body deformation, loss of identity features, or motion inconsistencies due to mismatches between the target character's body proportions and standard motion skeletons.

[0010] Therefore, although existing motion-driven video generation technologies can utilize text, human images, 2D pose sequences, or partial 3D motion data as generation conditions, the selection of motion data, perspective switching, generation of pose control information, and fusion of human appearance are usually independent of each other. The lack of continuous motion information to maintain constraints leads to problems such as weakened spatial structure, decreased motion continuity, incomplete perspective expression, and shift of pose control information when 3D motion information is transmitted in the AIGC video generation chain.

[0011] Therefore, it is necessary to propose an AIGC video generation method and system based on motion capture data. By establishing continuous motion information preservation constraints in motion retrieval, motion consistency judgment, virtual camera parameter adjustment, 3D motion to 2D pose control sequence conversion and human body structure adaptation, the 3D motion information can be consistently transmitted in the AIGC video generation chain, thereby improving the spatial structure preservation ability, motion continuity and multi-view expression consistency of human motion in the generated video. Summary of the Invention

[0012] In view of this, the purpose of this invention is to provide an AIGC video generation method and system based on motion capture data to solve the problem that existing motion-driven video generation processes lack continuous constraints in motion data selection, viewpoint transformation, posture control information transformation, and character fusion, resulting in the difficulty of consistently transmitting 3D motion information throughout the AIGC video generation chain. This invention establishes a motion information preservation mechanism that runs through the motion retrieval, motion consistency judgment, viewpoint adaptation optimization, 3D / 2D posture consistency verification, and human structure adaptation processes. This ensures that the 3D motion information formed by motion capture data can be stably transmitted to the final generated video, thereby improving the accuracy, continuity, and consistency of multi-view expression of character movements in the generated video.

[0013] To achieve the above objectives, the present invention provides the following technical solution: In one possible implementation, the present invention provides an AIGC video generation method based on motion capture data, comprising: The system acquires the target motion requirement information input by the user, and obtains candidate 3D motion data from a 3D motion database pre-constructed based on motion capture data based on the target motion requirement information. The 3D motion database stores 3D motion data formed by standardizing motion capture data, and the 3D motion data includes a 3D spatial position sequence corresponding to multiple joints of the human body, a joint rotation parameter sequence, and motion semantic information. Based on the relationship between human motion constraints and action semantic matching, the candidate 3D action data is judged for action consistency to determine the target 3D action data that meets the action retention conditions. Acquire target viewing angle information, and based on the key action area and action information retention requirements corresponding to the target's three-dimensional action data, adjust the initial parameters of the virtual camera determined by the target viewing angle information to obtain the target virtual camera parameters; Based on the target virtual camera parameters, the target 3D motion data is converted into a 2D attitude control sequence, and based on the spatial motion relationship in the target 3D motion data, the 2D attitude control sequence is subjected to consistency verification and anomaly correction. The appearance features of the target person are obtained, the human body structure parameters of the target person are determined based on the appearance features, and the two-dimensional posture control sequence is adapted to the human body structure based on the human body structure parameters. The two-dimensional posture control sequence adapted to human structure is fused with the appearance features of the target person to obtain video generation control information, and the video generation control information is input into the video generation model to generate the target video. Among them, the motion consistency judgment, virtual camera parameter adjustment, two-dimensional posture control sequence consistency verification, and human body structure adaptation together constitute the motion information maintenance process for ensuring the consistent transmission of three-dimensional motion information in the AIGC video generation link.

[0014] Furthermore, the construction of the pre-built 3D motion database based on motion capture data includes: Collect human motion capture data; The collected human motion capture data is processed by coordinate normalization, human skeleton model unification, and motion time scale standardization to obtain standardized three-dimensional motion data. Based on the standardized 3D motion data, action semantic labels, static posture features, and dynamic motion features are established, and the action semantic labels, static posture features, and dynamic motion features are associated with the corresponding 3D motion data and stored in the 3D motion database.

[0015] Furthermore, obtaining candidate 3D motion data from a pre-built 3D motion database based on motion capture data includes: Generate action retrieval features based on the target action requirement information; A first-stage retrieval is performed based on the action retrieval features and the action semantic information in the three-dimensional action database to obtain a candidate action set. Based on the static spatial distribution characteristics and dynamic motion trajectory characteristics of human joints, a second-stage matching is performed on the candidate action set. Based on the matching results of the second stage, candidate 3D action data is output from the candidate action set.

[0016] Furthermore, the step of determining the target 3D motion data that meets the motion retention conditions by performing motion consistency judgment on the candidate 3D motion data based on human motion constraints and action semantic matching relationships includes: Obtain the spatial motion trajectory of key human joints in the candidate 3D motion data; The motion continuity corresponding to the candidate 3D motion data is determined based on the changes in the position, direction of motion, and rotation of the key joints between adjacent motion frames. Based on the motion continuity and the matching relationship between the action semantic information and the target action requirement information, it is determined whether the candidate 3D action data meets the action retention condition; When the candidate 3D motion data meets the motion retention condition, the candidate 3D motion data is determined as the target 3D motion data.

[0017] Furthermore, the action consistency judgment also includes: Based on the spatial positional relationship of human joints in the candidate 3D motion data, the trajectory of the change of the human body's center of gravity is obtained; The body's balance during the execution of the action is determined based on the trajectory of the change in the human body's center of gravity. When the trajectory of the change of the human body's center of gravity does not match the preset motion state corresponding to the target motion type, the matching weight of the corresponding candidate three-dimensional motion data is reduced.

[0018] Furthermore, the key motion region is a human body region used to express the characteristics of the target motion during the execution of the target motion. The key motion region includes at least one of the hand region, foot region, trunk rotation region, and target joint local region. The target joint local region is a local motion region formed by at least one target joint determined by the target motion requirements and the human skeletal segment connected to the target joint. The target joint includes at least one of the shoulder joint, elbow joint, wrist joint, hip joint, knee joint, ankle joint, and neck joint. The adjustment of the initial parameters of the virtual camera determined by the target observation perspective information to obtain the target virtual camera parameters includes: Based on the initial parameters of the virtual camera, the three-dimensional motion data of the target is projected and analyzed to obtain the projection state of the key motion area under the target's viewing angle. Based on the degree of projection overlap, spatial separation, observability, and motion trajectory continuity of the key action areas, determine whether the initial parameters of the virtual camera meet the requirements for maintaining motion information. When the initial parameters of the virtual camera do not meet the requirements for maintaining motion information, at least one of the virtual camera's position, orientation, and field of view parameters is adjusted to obtain the target virtual camera parameters.

[0019] Furthermore, the step of performing consistency verification and anomaly correction on the two-dimensional attitude control sequence based on the spatial motion relationships in the target three-dimensional motion data includes: Calculate the three-dimensional motion change trend of human joints based on the target three-dimensional motion data; Based on the projection relationship corresponding to the target virtual camera parameters, calculate the two-dimensional motion change trend of the corresponding two-dimensional key points in the two-dimensional attitude control sequence; Compare the three-dimensional motion change trend with the two-dimensional motion change trend; When the difference between the three-dimensional motion change trend and the two-dimensional motion change trend exceeds a preset range, the corresponding two-dimensional posture data is determined as abnormal posture data, and the abnormal posture data is corrected according to the spatial motion relationship in the target three-dimensional motion data.

[0020] Further, the step of determining the human body structure parameters of the target person based on the appearance features of the target person, and performing human body structure adaptation on the two-dimensional posture control sequence based on the human body structure parameters, includes: Extract human proportion parameters, body contour parameters, and size parameters of key human body parts from images or video frames of the target person; Based on the human body proportion parameters, body contour parameters, and key human body part size parameters, a mapping relationship is established between the target human body structure and the standard motion skeletal model. Based on the mapping relationship, the human skeleton scale and key point position relationship in the two-dimensional posture control sequence are adjusted so that the appearance structure of the target person matches the target action posture.

[0021] Furthermore, the video generation method further includes a generation result feedback optimization step, which includes: Human pose detection is performed on the generated target video to obtain the corresponding action performance information of the generated video; The motion retention error is determined based on the difference between the motion performance information and the motion reference information corresponding to the target three-dimensional motion data; Adjust at least one of the following parameters based on the motion retention error: 3D motion data retrieval weight, motion consistency judgment parameter, virtual camera parameter, posture consistency verification parameter, and human body structure adaptation parameter.

[0022] In one possible implementation, the present invention also provides an AIGC video generation system driven by motion capture data, comprising: The 3D motion database module is used to store 3D motion data formed by standardized processing of motion capture data. The 3D motion data includes 3D spatial position sequences corresponding to multiple joints of the human body, joint rotation parameter sequences, and motion semantic information. The action retrieval module is connected to the three-dimensional action database module and is used to retrieve candidate three-dimensional action data from the three-dimensional action database module based on the target action requirement information input by the user. The action consistency judgment module, connected to the action retrieval module, is used to judge the action consistency of the candidate 3D action data based on human motion constraints and action semantic matching relationship, and output the target 3D action data that meets the action retention conditions. The viewpoint adaptation optimization module, connected to the action consistency judgment module, is used to adjust the virtual camera parameters based on the target observation viewpoint information, the key action area corresponding to the target's three-dimensional action data, and the action information preservation requirements, to obtain the target virtual camera parameters. The attitude conversion and consistency verification module is connected to the viewpoint adaptation and optimization module. It is used to convert the target three-dimensional motion data into a two-dimensional attitude control sequence according to the target virtual camera parameters, and to perform consistency verification and anomaly correction on the two-dimensional attitude control sequence based on the spatial motion relationship in the target three-dimensional motion data. The character structure adaptation module is connected to the posture conversion and consistency verification module. It is used to determine the human body structure parameters of the target character based on the appearance features of the target character, and to perform human body structure adaptation on the two-dimensional posture control sequence based on the human body structure parameters. The video generation module, connected to the human body structure adaptation module, is used to fuse the two-dimensional posture control sequence after human body structure adaptation with the appearance features of the target human body to obtain video generation control information, and generate the target video based on the video generation control information.

[0023] In one possible implementation, the present invention also provides an electronic device including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method described in any of the preceding embodiments.

[0024] In one possible implementation, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding embodiments.

[0025] Based on the above technical solutions, the AIGC video generation method and system driven by motion capture data of the present invention obtains candidate 3D motion data from a 3D motion database and determines target 3D motion data based on human motion constraints and motion semantic matching relationships, so that the motion data entering the video generation link simultaneously meets the requirements of motion semantics and human motion laws; by adjusting the virtual camera parameters based on the requirements for key motion regions and motion information preservation, the target 3D motion data can still maintain the observability of key motion regions, the continuity of motion trajectories, and human spatial relationships from the target's viewing perspective; by performing consistency verification and anomaly correction on the 2D posture control sequence based on 3D spatial motion relationships, the information loss in the process of converting 3D motion data into 2D posture control information is reduced; and by adapting the human body structure, the target motion can be matched with the appearance structure of the target person.

[0026] Therefore, this invention does not simply use motion capture data or 3D motion data as input conditions for the video generation model. Instead, through continuous processing such as motion consistency judgment, viewpoint adaptation optimization, posture consistency verification, and human body structure adaptation, it enables 3D motion information to be consistently transmitted in the AIGC video generation chain, thereby improving the ability to maintain the spatial structure of human movements, the continuity of motion, and the consistency of multi-view expression in the generated video. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall architecture of an AIGC video generation system based on motion capture data driven by the present invention. Figure 2 This is a flowchart illustrating an AIGC video generation method based on motion capture data driven by the present invention. Figure 3 This is a schematic diagram of the motion transformation and video generation process based on three-dimensional motion information and constraints in this invention.

[0028] Explanation of reference numerals in the attached figures: In the diagram: 100, 3D motion database module; 200, motion retrieval module; 300, motion consistency judgment module; 400, viewpoint adaptation optimization module; 500, posture transformation and consistency verification module; 600, character structure adaptation module; 700, video generation module; 800, feedback optimization module. Detailed Implementation

[0029] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the following specific embodiments are used to further illustrate the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0030] like Figure 1As shown, this invention provides an AIGC video generation system driven by motion capture data. The system includes a 3D motion database module, a motion retrieval module, a motion consistency judgment module, a viewpoint adaptation optimization module, a pose transformation and consistency verification module, a character structure adaptation module, and a video generation module. These modules work collaboratively to ensure the consistent transmission of 3D motion information throughout the video generation process, thereby achieving a video generation process that maintains constraints based on 3D motion information.

[0031] like Figure 2 As shown, this invention provides an AIGC video generation method driven by motion capture data. The method starts with target motion requirement information, obtains candidate 3D motion data from a pre-built 3D motion database, and filters target 3D motion data through motion consistency judgment. Subsequently, based on motion information preservation requirements, the virtual camera parameters corresponding to the target's viewing angle are optimized, converting the target 3D motion data into a 2D posture control sequence. The converted 2D posture control sequence undergoes 3D / 2D posture consistency verification and anomaly correction. Furthermore, human body structure adaptation is performed by combining the target person's appearance features, ultimately generating the target video.

[0032] like Figure 3 As shown, this invention further illustrates the motion transformation and video generation process based on 3D motion information preservation constraints. In this process, 3D motion data is not directly used as input to the video generation model. Instead, before being converted into video generation control information, it undergoes sequential processing such as motion consistency judgment, viewpoint preservation evaluation, posture consistency verification, and character structure adaptation. This ensures that the 3D motion information can be consistently transmitted between different processing stages, including motion selection, viewpoint transformation, posture control information generation, and character fusion.

[0033] In this embodiment, the motion information preservation constraint is not a local optimization process set for a single processing stage, but a continuous control mechanism that runs through the entire AIGC video generation process. Specifically, this invention establishes a three-dimensional motion data foundation in the motion data acquisition stage, introduces human motion constraints in the motion screening stage, introduces motion information preservation requirements in the viewpoint transition stage, introduces three-dimensional and two-dimensional consistency verification in the pose generation stage, introduces human structure adaptation in the character fusion stage, and introduces motion preservation error optimization in the generation feedback stage. This enables the three-dimensional motion information formed by motion capture data to be stably transmitted along the video generation link, thereby improving the spatial structure preservation ability, motion continuity, and multi-view expression consistency of character movements in the generated video.

[0034] In this embodiment, the motion information preservation requirement refers to the judgment conditions used to constrain the consistent transmission of three-dimensional motion information during motion data filtering, viewpoint switching, posture control information generation, and character structure adaptation. The motion information preservation requirement includes at least: the observability of key motion areas in the target motion from the target's viewing perspective; the continuity of the motion trajectory of key motion areas; the degree of preservation of the spatial relationship between human joints; the consistency between the three-dimensional motion change trend and the two-dimensional posture control sequence motion change trend; and the degree of matching between the target character's human structure and motion posture.

[0035] Specifically, in the perspective transition stage, the motion information preservation requirement is used to determine whether the target's viewing perspective can fully express the key action areas, action trajectories, and human spatial relationships; in the posture control information generation stage, the motion information preservation requirement is used to determine whether the two-dimensional posture control sequence maintains the spatial motion relationships in the target's three-dimensional motion data; in the character structure adaptation stage, the motion information preservation requirement is used to determine whether the target's action posture matches the target character's body proportions, body contours, and key component dimensions. Through these motion information preservation requirements, the three-dimensional motion information can be stably transmitted during the conversion from motion capture data to video generation control information.

[0036] Example 1: AIGC Video Generation Method Based on Motion Capture Data

[0037] like Figure 2 As shown, this embodiment provides an AIGC video generation method based on motion capture data. This embodiment acquires, filters, optimizes viewpoints, and performs pose transformation processing on 3D motion data. Before the motion data enters the video generation model, a multi-stage motion information preservation constraint mechanism is established, ensuring that human motion information maintains consistency in spatial structure and motion patterns during the conversion from 3D motion data to the target video.

[0038] Specifically, this embodiment first obtains candidate 3D motion data from a pre-constructed 3D motion database based on motion capture data according to the target motion requirement information input by the user, and performs a consistency judgment on the candidate 3D motion data based on human motion constraints to determine the target 3D motion data that meets the motion retention conditions; then, according to the target observation perspective information and motion information retention requirements, the virtual camera parameters are adaptively adjusted so that the target motion can maintain key motion information under different observation perspectives; further, the 3D motion data after perspective optimization is converted into a 2D posture control sequence, and the 2D posture control sequence is subjected to consistency verification and anomaly correction based on 3D motion laws; finally, human body structure adaptation is performed in combination with the target person's appearance features, and the fused motion control information is input into the video generation model to generate the target video.

[0039] Specifically, the video generation method provided in this embodiment includes the following steps: S1. Construct a 3D motion database based on motion capture data. In this embodiment, in order to achieve a complete description of human motion information, a three-dimensional motion database is first constructed for storing and managing three-dimensional human motion data formed by motion capture data.

[0040] Specifically, the three-dimensional motion database stores multiple three-dimensional motion samples formed by standardized processing of motion capture data. Each three-dimensional motion sample is used to describe the spatial motion state of the human body when performing the corresponding action, and includes at least the three-dimensional spatial position sequence of human skeletal joints, the human joint rotation parameter sequence, and semantic description information corresponding to the action.

[0041] The human skeletal joints can be defined based on a preset human skeletal model, including joints of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. By recording the spatial position changes of these joints in consecutive time frames, the spatial relationships and motion changes between different body parts during human movement can be obtained.

[0042] Unlike motion data that only contains two-dimensional image planar position information, the three-dimensional motion data in this embodiment simultaneously contains information on the positional changes of human joints in the spatial coordinate system and information on joint rotation changes. Therefore, it can more completely reflect the spatial motion characteristics during human movement, providing a data foundation for subsequent motion consistency judgment and viewpoint adaptation processing.

[0043] S1.1 Motion capture data acquisition and standardization processing

[0044] In one alternative implementation, the motion samples in the three-dimensional motion database can be formed from motion capture data through standardized processing.

[0045] Specifically, optical motion capture devices, inertial motion capture devices, human three-dimensional pose estimation devices based on multi-view images, or other human motion acquisition devices can be used to collect the motion state of the human body during different actions to obtain human motion capture data; the motion capture data includes the three-dimensional spatial position of human skeletal joints, joint rotation state, and their relationship with time.

[0046] Since different acquisition methods may have different coordinate references, different human body model structures, and different motion execution times, the raw motion capture data needs to be standardized before it is stored in the 3D motion database.

[0047] In this embodiment, the standardization process includes at least coordinate system normalization, skeletal model unification, and motion time series standardization.

[0048] In the coordinate system normalization process, the human joint point position data obtained under different acquisition environments are spatially transformed according to the preset spatial reference coordinate system, so that the motion data from different sources are all under a unified spatial coordinate system, thereby eliminating the spatial offset caused by changes in the acquisition position.

[0049] In the unified processing of the skeletal model, the skeletal joints corresponding to different human objects are mapped according to the preset human skeletal topology, so that the motion data corresponding to different human body sizes and different skeletal proportions can be converted into a data expression form under the unified skeletal model.

[0050] In motion time series standardization, the time scale of motion data at different execution speeds is adjusted according to the key motion stages in the motion execution process, so that different motion samples can be compared under a unified time dimension.

[0051] Through the above standardization process, different motion samples in the 3D motion database have a unified data structure, which facilitates subsequent matching analysis based on motion spatial characteristics and motion change patterns.

[0052] S1.2 Construction of Action Feature Information

[0053] Furthermore, to improve the accuracy of subsequent action retrieval and consistency judgment, this embodiment also constructs multi-dimensional action feature information based on the standardized three-dimensional action data.

[0054] Specifically, the action feature information includes spatial posture features, dynamic motion features, and action semantic features.

[0055] The spatial posture features are used to describe the spatial posture state of the human body at various time points during the execution of an action, including the three-dimensional coordinates of human joints, the spatial distance relationship between joints, and the angular relationship between human bones.

[0056] The dynamic motion features are used to describe the motion patterns of human body movements over time, including changes in displacement of key joints, changes in motion speed, changes in acceleration, and information on continuous motion trajectories.

[0057] The action semantic features are used to describe the semantic meaning of the action. They can be generated by manual annotation or based on a semantic coding model, such as action description information like "waving", "turning around", and "jumping".

[0058] By constructing the aforementioned multi-dimensional motion feature information, motion samples in the three-dimensional motion database can not only be retrieved based on textual semantics, but also further filtered by combining the actual movement patterns of the human body, thereby improving the accuracy of motion data matching.

[0059] S2. Action consistency judgment based on human motion constraints

[0060] In this embodiment, after acquiring candidate 3D motion data, the candidate 3D motion data is not directly used as motion control data in the video generation process. Instead, the consistency of the candidate 3D motion data is further judged based on human motion constraints in order to filter out 3D motion data that can maintain the target motion features.

[0061] Specifically, different action samples may have the same or similar semantic descriptions, but their actual movement processes may differ significantly. For example, even within the same "waving" action, different action samples may correspond to small arm swings, large arm rotations, or complex movements accompanied by body rotation. If matching is based solely on semantic information, it can easily lead to the selected action data being inconsistent with the user's actual needs, thus affecting the subsequent video generation results.

[0062] Therefore, this embodiment further introduces human motion constraints on the basis of motion retrieval. By analyzing the motion change patterns of human joints in three-dimensional space, the consistency of candidate three-dimensional motion data is judged.

[0063] Specifically, the steps include the following: S2.1 Obtain key motion information from candidate 3D motion data In one implementation, the corresponding critical action area is first determined based on the target action requirement information.

[0064] The key motion region is used to characterize the human body areas that undergo significant motion changes during the execution of the target action, including but not limited to: The hand area; the arm area; the leg area; the trunk rotation area; and the joint areas associated with the target movement.

[0065] For example, when the target action requirement is "the character waves his hand", the system can use the shoulder joint, elbow joint and wrist joint as the main analysis objects; when the target action requirement is "the character completes a jumping action", the hip joint, knee joint and ankle joint can be used as the main analysis objects.

[0066] Subsequently, the three-dimensional motion information corresponding to the aforementioned key motion regions was extracted from the candidate three-dimensional motion data, including: Spatial position change information of each key node in consecutive time frames; Information on changes in spatial distance between adjacent joints; Information on changes in joint rotation angle.

[0067] The above methods can be used to obtain motion characteristic data that reflects the human body's movement execution process.

[0068] S2.2 Consistency Analysis Based on Spatial Motion Trajectory

[0069] In this embodiment, in order to determine whether the candidate three-dimensional motion data conforms to the laws of human movement, the spatial motion trajectory of key joints of the human body is first analyzed.

[0070] Specifically, the spatial movement trajectory of the same human joint is determined based on the three-dimensional spatial coordinates of the same human joint in multiple consecutive time frames.

[0071] Furthermore, based on the positional change relationship between adjacent time frames, the motion direction and motion amplitude of the corresponding joint points are determined.

[0072] For example, for hand joints, the direction of hand movement can be determined by the changes in spatial position in consecutive time frames; for leg joints, the range of leg swing can be determined by analyzing the spatial changes in leg joints.

[0073] Furthermore, based on the coordinated movement relationship between key joints, the coordination of movement between different parts of the human body is assessed.

[0074] For example, in normal walking, the movement of the left and right legs should follow an alternating pattern, and the body's center of gravity should change accordingly with the gait cycle. If there are abnormalities in the relationship between the left and right leg movements or if the change in the center of gravity does not conform to the gait pattern in the candidate movement data, it indicates that the movement data deviates from the target movement requirements.

[0075] Through the above analysis, we can obtain the spatial motion consistency evaluation results corresponding to the candidate 3D motion data.

[0076] S2.3 Consistency judgment based on joint motion continuity

[0077] Furthermore, this embodiment also judges the continuity of joint motion in the candidate three-dimensional motion data.

[0078] Specifically, the changes in the position of human joints and the changes in joint rotation between adjacent action frames are obtained.

[0079] When the change in position or rotation between adjacent action frames exceeds the preset range of motion change, it is determined that there is a sudden change in motion in the corresponding action data.

[0080] For example, during the continuous raising of a human arm, the movements of the shoulder joint, elbow joint, and wrist joint should have a continuous relationship; if the position of the wrist joint suddenly changes significantly in a certain time frame, it indicates that there may be a sampling error or abnormal movement in the action sequence.

[0081] In this embodiment, when a sudden change in motion is detected in the motion data, the matching weight of the candidate motion data can be reduced, or motion smoothing processing can be performed on the corresponding motion data.

[0082] This method avoids issues such as motion stuttering and limb jumps in the generated video due to discontinuous motion data.

[0083] S2.4 Judging motion stability based on changes in the human body's center of gravity

[0084] Furthermore, to improve the accuracy of motion consistency judgment, this embodiment also introduces a human body center of gravity change analysis mechanism.

[0085] Specifically, the trajectory of the center of gravity during human movement is calculated based on the spatial relationship of the joints of the human skeleton.

[0086] Among them, the trajectory of the human body's center of gravity is used to reflect the overall motion state of the human body during the execution of actions.

[0087] For example, in a jumping action, the human body's center of gravity usually includes the take-off phase, the flight phase, and the landing phase, with different phases having corresponding spatial change trends. For rotational movements, the body's center of gravity should change within a certain spatial range, and should not deviate significantly from the laws of human movement.

[0088] Therefore, this embodiment analyzes the trajectory of the center of gravity change in the candidate motion data to determine whether the overall motion state of the human body conforms to the motion law corresponding to the target motion type.

[0089] When a mismatch is detected between the center of gravity change trend in the candidate action data and the target action type, the action matching degree of the candidate action data is reduced.

[0090] S2.5, Action Consistency Evaluation Based on Multidimensional Motion Indicators

[0091] In this embodiment, to avoid the problem of inconsistent action spatial structure caused by action matching based solely on action semantic information, an action consistency evaluation mechanism based on human motion characteristics is further established.

[0092] Specifically, based on the human spatial motion characteristics, joint motion continuity characteristics, and overall human motion stability characteristics corresponding to the candidate three-dimensional motion data, motion consistency evaluation indicators are constructed respectively, and the comprehensive results of multiple evaluation indicators are used to determine whether the candidate three-dimensional motion data meets the motion retention conditions.

[0093] In one embodiment, the motion consistency evaluation results include spatial trajectory consistency index, joint continuity index, and motion stability index.

[0094] (1) Determination of spatial trajectory consistency index

[0095] In this embodiment, the spatial trajectory consistency index is used to evaluate the degree of consistency between the motion trajectory of key human joints in candidate 3D motion data and the motion characteristics corresponding to the target motion requirement.

[0096] Specifically, for the i-th key joint in the target action, its three-dimensional spatial position sequence is obtained within consecutive time frames: The three-dimensional spatial position sequence is used to describe the spatial movement trajectory of the key joint during the execution of the action.

[0097] Furthermore, based on the positional change relationship between adjacent time frames, the spatial motion trajectory characteristics of the corresponding key joints are determined.

[0098] The spatial motion trajectory features include: Direction of movement of key joints; distance of movement of key joints; range of motion of key joints.

[0099] Then, the spatial motion trajectory features corresponding to the candidate 3D motion data are compared with the expected motion features corresponding to the target motion requirements to determine the degree of consistency of the spatial trajectories.

[0100] When the motion direction, motion amplitude, and spatial change trend of key joints in the candidate 3D motion data match the target motion requirements, the spatial trajectory is considered to have high consistency.

[0101] This method avoids judging similarity of actions solely based on action names or text semantics, and allows for further assessment of whether the actual human movement process is consistent.

[0102] (2) Determination of joint continuity index

[0103] In this embodiment, the joint continuity index is used to evaluate whether the changes in the movement of each joint during the execution of human movements conform to the law of continuous movement.

[0104] Specifically, the changes in human joints between adjacent action frames in the candidate 3D motion data are obtained.

[0105] The changes in the joint points include: Changes in joint position; changes in joint velocity; changes in joint rotation.

[0106] Furthermore, based on the motion change relationship between multiple consecutive action frames, it can be determined whether there are any abnormal abrupt changes in human joint movement.

[0107] For example, during normal human movement, the shoulder, elbow, and wrist should maintain a continuous movement relationship; if the wrist position changes abnormally in adjacent action frames, but the shoulder and elbow movement states do not change accordingly, it indicates that there is an abnormality in the continuity of movement in the action data.

[0108] In one implementation, when the amount of motion change corresponding to a certain key joint exceeds a preset continuity threshold, the motion matching weight of the corresponding candidate 3D motion data is reduced.

[0109] The above methods can be used to filter out discontinuous motion data caused by motion acquisition errors, abnormal motion execution, or missing data.

[0110] (3) Determination of motion stability index

[0111] In this embodiment, the motion stability index is used to evaluate whether the overall human motion process conforms to the laws of human motion.

[0112] Specifically, based on the spatial relationship of the joints of the human skeleton, the trajectory of the overall center of gravity change during human movement is calculated.

[0113] Among them, the trajectory of the human body's center of gravity change is used to reflect the overall balance state of the human body during the execution of actions.

[0114] For example, for jumping movements, the body's center of gravity should change continuously during takeoff, flight, and descent; for rotational movements, the body's center of gravity should change within a reasonable spatial range; and for walking movements, the body's center of gravity should correspond to changes in gait.

[0115] Furthermore, the trajectory of the center of gravity change in the candidate 3D motion data is compared with the preset motion law corresponding to the target motion.

[0116] When the trend of the change in the center of gravity of the human body in the candidate action data conforms to the motion law corresponding to the target action, the consistency evaluation result of the candidate action data is improved. When the change of the human body's center of gravity is abnormally offset or does not conform to the change pattern of the target movement stage, the consistency evaluation result of the candidate movement data will be reduced.

[0117] By incorporating the assessment of overall human motion stability, the reliability of motion matching results can be further improved.

[0118] (4) Comprehensive evaluation of motion consistency

[0119] In this embodiment, the candidate three-dimensional motion data are comprehensively evaluated based on the aforementioned spatial trajectory consistency index, joint continuity index, and motion stability index.

[0120] Specifically, corresponding evaluation weights can be set according to the importance of different evaluation indicators, and the consistency evaluation results of actions can be calculated based on each evaluation indicator.

[0121] In one implementation, the action consistency evaluation result is determined by the following factors: Spatial trajectory consistency index; joint continuity index; motion stability index.

[0122] Among them, the spatial trajectory consistency index is used to reflect whether the local movements of the human body meet the target requirements; the joint continuity index is used to reflect whether the movement process is continuous and natural; and the motion stability index is used to reflect whether the overall movement of the human body conforms to the laws of human movement.

[0123] When the motion consistency evaluation result meets the preset motion retention conditions, the corresponding candidate 3D motion data is determined as the target 3D motion data.

[0124] If the action consistency evaluation result does not meet the preset action retention conditions, then the candidate 3D action data will be reacquired, or the action retrieval parameters will be adjusted according to the evaluation result and the action matching process will be executed again.

[0125] Through the above-mentioned action consistency evaluation mechanism, this embodiment realizes the transformation from "action semantic matching" to "action spatial structure matching", so that the target three-dimensional action data not only meets the action description requirements input by the user, but also meets the constraints of human motion law.

[0126] Compared to existing methods that select actions based solely on text semantics or two-dimensional pose similarity, this embodiment improves the accuracy of the action data screening process by combining the spatial trajectory of human joints, the continuity of joint movement, and the overall stability of human movement. This provides a reliable data foundation for maintaining consistency of action information during subsequent video generation.

[0127] In this embodiment, in order to quantitatively evaluate the consistency between candidate 3D motion data and target motion requirements, a motion consistency evaluation model based on human motion constraints is established.

[0128] Specifically, the motion consistency evaluation value is jointly determined by the spatial trajectory consistency index, the joint continuity index, and the motion stability index, and its expression is as follows:

[0129] in: The value is the consistency evaluation value for actions, which is a dimensionless parameter and its value ranges from 0 to 1; As an index of spatial trajectory consistency; It serves as an indicator of joint continuity. As an indicator of motion stability; , , These are the weight coefficients for the corresponding indicators, satisfying:

[0130] The weighting coefficients are preset and adjusted according to the target action type.

[0131] (1) Spatial trajectory consistency index

[0132] The spatial trajectory consistency index is used to evaluate the degree of consistency between the motion trajectory of key joints of the human body in the candidate action and the corresponding motion trajectory of the target action.

[0133] Its expression is:

[0134] in: The number of key joints in the human body participating in the evaluation; For the candidate 3D motion data, the first i Normalized spatial motion trajectory feature vectors corresponding to key joints; The reference motion trajectory feature vector corresponding to the target action requirement; Indicates vector distance calculation; The normalized scale parameters for human motion space.

[0135] because:

[0136] Since it is a normalized distance, it has no dimension of length, making Keep it within the range of 0 to 1.

[0137] When the candidate action trajectory is closer to the target action trajectory The closer it is to 1.

[0138] (2) Joint continuity index

[0139] The joint continuity index is used to evaluate whether the movement of human joints is continuous during candidate movements.

[0140] Its expression is:

[0141] in: The number of frames in the action time; Let be the normalized motion velocity of key human joints in time frame t; The preset maximum normalized motion speed.

[0142] When the motion changes between adjacent time frames are smoother The smaller, The closer it is to 1.

[0143] (3) Motion stability index

[0144] The motion stability index is used to evaluate whether the overall human motion process conforms to the motion law corresponding to the target action.

[0145] Its expression is:

[0146] in: The normalized position of the human body's center of gravity in the t-th time frame of the candidate action; The trend of reference center of gravity change corresponding to the target action; It is a normalized scale for human body space.

[0147] The above model can be used to further extend action semantic matching to human movement pattern matching, thereby improving the accuracy of target 3D action data selection.

[0148] S2.6 Comprehensive Evaluation of Motion Consistency

[0149] In this embodiment, the results of the spatial motion trajectory analysis, the results of the joint motion continuity analysis, and the results of the human body center of gravity change analysis are comprehensively evaluated.

[0150] Specifically, calculations can be performed separately based on preset weights: Spatial trajectory consistency index; joint continuity index; overall motion stability index.

[0151] The consistency evaluation results of the candidate three-dimensional motion data are determined based on various indicators.

[0152] Furthermore, the results of the motion consistency evaluation are compared with the preset motion maintenance conditions.

[0153] When the candidate 3D motion data meets the motion retention condition, the candidate 3D motion data is determined as the target 3D motion data; If the candidate 3D action data does not meet the action retention condition, the system returns to the 3D action database to retrieve the candidate action data again, or the candidate action data is optimized and then a consistency judgment is performed again.

[0154] In this embodiment, instead of simply selecting action data based on action semantics, the action data is further constrained and filtered by combining human movement patterns on the basis of semantic matching, so that the action data used for video generation can simultaneously meet the requirements of action semantics and the requirements of human movement continuity.

[0155] Compared to methods that rely solely on two-dimensional motion reference information or text descriptions to control motion generation, this embodiment can reduce video generation errors caused by incomplete motion information or motion matching deviations, thereby improving the accuracy and naturalness of human movements in the generated video.

[0156] S3. Adaptive optimization of virtual camera based on motion information preservation requirements

[0157] In this embodiment, after determining the target's three-dimensional motion data, in order to enable the generated video to display the target's motion from the user's desired viewing angle, it is necessary to further optimize the virtual viewing angle corresponding to the three-dimensional motion data.

[0158] Existing video generation methods based on 3D motion data typically render 3D models directly according to user-defined camera parameters. They primarily focus on achieving changes in viewpoint without further considering whether changes in viewpoint will lead to the loss of motion information.

[0159] However, because human movements have distinct spatial structural features, different viewing angles may obscure some key movement areas or prevent the complete representation of movement trajectories. For example, when observing arm swinging movements from a side viewpoint, the hand's movement trajectory may not be fully represented due to the torso obscuring the movement; when observing leg movements from a top-down viewpoint, the spatial relationship between the legs may be difficult to accurately represent.

[0160] Therefore, in setting up the virtual camera, this embodiment introduces the requirement of motion information preservation. By analyzing the completeness of the target action under different viewing angles, the virtual camera parameters are adaptively adjusted so that the viewing angle in the generated video not only meets the user's needs, but also preserves the key motion information of the target action.

[0161] Specifically, the steps include the following: S3.1 Obtain target observation angle parameters In this embodiment, the target viewing angle information input by the user is first obtained.

[0162] The target viewing angle information, used to describe the user's desired video viewing direction, may include: Virtual camera spatial position parameters; virtual camera orientation parameters; virtual camera focal length parameters; virtual camera field of view parameters.

[0163] Specifically, users can input a specified viewing angle according to their video creation needs, for example: Frontal viewpoint; side viewpoint; oblique viewpoint; top-down viewpoint.

[0164] The system builds a corresponding virtual camera model in a virtual 3D space based on the target observation perspective information input by the user.

[0165] S3.2 Determine the critical motion area corresponding to the target motion.

[0166] In this embodiment, to determine whether the target's viewing perspective can fully express the target's action, a motion structure analysis is first performed on the target's three-dimensional motion data to identify the key motion regions used to express the characteristics of the target's action during its execution. These key motion regions refer to local areas of the human body that directly influence the semantic expression of the action, the maintenance of the action trajectory, or the maintenance of the human body's spatial relationships during the execution of the target's action.

[0167] Specifically, key motion regions are determined based on target motion requirement information, spatial displacement changes and joint rotation changes of each human joint in the target three-dimensional motion data, and motion semantic information. The key motion regions include at least one of the following: hand region, foot region, torso rotation region, and local regions of the target joint.

[0168] The term "target joint local area" does not refer to all joints in the human body, but rather to at least one target joint determined by the target movement requirement, and the local motion area formed by the human skeletal segment connected to that target joint. The target joint includes at least one of the shoulder, elbow, wrist, hip, knee, ankle, and neck joints.

[0169] For example, when the target movement requirement is a waving motion, the key movement area may include the hand area, as well as the target joint local area formed by the shoulder joint, elbow joint, and wrist joint and the connected upper arm and forearm bone segments; when the target movement requirement is a jumping motion, the key movement area may include the foot area, as well as the target joint local area formed by the hip joint, knee joint, and ankle joint and the connected thigh, calf, and foot bone segments; when the target movement requirement is a turning motion, the key movement area may include the trunk rotation area, as well as the target joint local areas corresponding to the neck joint, shoulder joint, and hip joint.

[0170] Furthermore, based on the three-dimensional motion trajectory, spatial displacement variation, joint rotation variation, and semantic correlation of each key motion region, the importance of each key motion region in the motion execution process is determined. For example, for dance movements, if the motion trajectory variation of the hand region is greater than that of other human body regions, and the hand region has a direct correspondence with the target motion semantics, then the weight of the hand region in the perspective evaluation process is increased; for martial arts movements, if the torso rotation region and the local area of ​​the target joint corresponding to the arm jointly determine the direction of force exertion, then the weight of the above regions in the perspective evaluation process is increased.

[0171] In this way, the subsequent perspective judgment process can be centered on the expression needs of the target action, and the virtual camera parameters can be adjusted according to the observability of the key action area under the target's observation perspective, the degree of projection overlap, the degree of spatial separation, and the degree of continuity of the motion trajectory, rather than judging the perspective solely based on the image display effect.

[0172] S3.3 Establish evaluation indicators for motion information retention

[0173] In this embodiment, in order to determine the degree of retention of target motion information under different viewing angles, it is necessary to establish a motion information retention evaluation index for evaluating the rationality of virtual camera parameters.

[0174] Specifically, because human movements in three-dimensional space have directional, regional correlation, and continuous motion characteristics, human movement information may be lost due to changes in projection relationships when the viewing direction of the virtual camera changes. Therefore, this embodiment does not directly determine the virtual camera parameters based on the user-input viewing angle, but rather establishes a motion information preservation evaluation index based on the motion characteristics of the target action itself to evaluate the ability to express motion information from the current viewing perspective.

[0175] In one embodiment, the motion information retention evaluation index includes a key motion area visibility index, a motion trajectory integrity index, and a human spatial relationship retention index.

[0176] (1) Visibility index of key action area

[0177] The key motion area visibility index is used to evaluate whether the human body area with the main motion information in the target motion can be effectively observed from the current observation perspective.

[0178] Specifically, based on the motion amplitude, spatial displacement changes, and semantic information of each human joint in the target's three-dimensional motion data, the key motion region corresponding to the target's motion is determined.

[0179] For example, for a waving motion, the areas corresponding to the shoulder joint, elbow joint, and wrist joint can be identified as key motion areas; for a jumping motion, the hip area, knee area, and ankle area can be identified as key motion areas; for martial arts movements, the fist area, arm extension area, and torso rotation area can be identified as key motion areas.

[0180] Furthermore, based on the current virtual camera parameters, projection calculations are performed on the human joints in the key motion area to obtain the projection state of the key motion area in the two-dimensional imaging plane.

[0181] The projection state includes at least the following: The projection range of the key action area; The degree of overlap of projections between key action areas; The degree to which the key movement area is obscured by other human body areas.

[0182] In this embodiment, when the key action area has a large visible range in the two-dimensional projection plane and different key action areas can be effectively distinguished, the current viewing angle is considered to have high visibility of the key action area.

[0183] Conversely, when the key action area is obscured by other areas of the human body for a long time, or when multiple key action areas overlap significantly in the projection plane, it is believed that the current observation perspective will lead to a loss of action information.

[0184] By establishing visibility metrics for key action areas, the virtual camera adjustment process can be made to revolve around the target action expression requirements, rather than simply switching perspectives based on user-specified angles.

[0185] (2) Motion trajectory integrity index

[0186] The motion trajectory integrity index is used to evaluate whether the motion trajectory of key parts of the human body can be continuously expressed from the current observation perspective.

[0187] Specifically, the motion trajectory of key joints in three-dimensional space is first obtained based on the target's three-dimensional motion data.

[0188] The motion trajectory includes: Direction of spatial movement at key points; Changes in the distance the joint point moves; Trends in joint velocity changes.

[0189] Furthermore, based on the current virtual camera parameters, the three-dimensional motion trajectory is converted into a corresponding two-dimensional projection trajectory.

[0190] Then, the correspondence between the two-dimensional projection trajectory and the original three-dimensional motion trajectory is analyzed.

[0191] For example, when the arm extends forward, the wrist joints of the human body exhibit significant depth movement in three-dimensional space. If the current viewing angle causes the depth movement to be compressed in a two-dimensional projection, the arm extension process may not be reflected in the two-dimensional image.

[0192] Therefore, this embodiment determines the completeness of the motion trajectory corresponding to the current observation perspective by judging whether the two-dimensional projection trajectory can reflect the three-dimensional motion trend.

[0193] When the two-dimensional projection trajectory can continuously reflect the movement direction and amplitude of key joints, the integrity of the motion trajectory is considered to be high. When the two-dimensional projection trajectory shows motion stagnation, abnormal direction changes, or missing trajectory, it is determined that the integrity of the motion trajectory is insufficient.

[0194] (3) Human spatial relationship maintenance index

[0195] The human body spatial relationship preservation index is used to evaluate whether the spatial structure between different joints of the human body can be maintained from the current observation perspective.

[0196] Specifically, based on the topological relationship of the human skeleton in the target 3D motion data, the spatial relationship between key joints of the human body is obtained.

[0197] The spatial associations include: The distance relationship between adjacent joints; Joint connection direction relationship; The relative positional relationships between different regions of the human body.

[0198] Furthermore, based on the virtual camera projection results, the degree to which the above spatial relationships are preserved in the two-dimensional imaging plane is analyzed.

[0199] For example, in the action of extending both arms, it is necessary to maintain the spatial separation between the left and right hand areas; in the action of alternating leg movements, it is necessary to maintain the positional change relationship between the left and right legs; in the action of rotating the body, it is necessary to maintain the relative movement relationship between the shoulders, hips and torso.

[0200] When the two-dimensional projection result can maintain the main spatial structural relationships of the human body, it is considered that the current observation perspective meets the requirements for maintaining the spatial relationships of the human body.

[0201] When two-dimensional projection causes severe overlap of key areas of the human body, or makes it impossible to distinguish different movement areas of the human body, it is considered that the current observation perspective may cause the loss of movement structure information.

[0202] S3.4 Comprehensive Evaluation of Motion Information Retention

[0203] In this embodiment, after establishing the motion information preservation evaluation index, the motion expression effect corresponding to the current virtual camera parameters is further comprehensively evaluated based on multiple evaluation indexes.

[0204] Specifically, the visibility index of key action areas, the integrity index of action trajectory, and the human spatial relationship preservation index are integrated to obtain the action information preservation evaluation result corresponding to the current observation perspective.

[0205] In one implementation, the evaluation weights corresponding to different evaluation indicators are determined based on the type of target action.

[0206] For example, for dance movements, since the hand trajectories and leg movements change more significantly, the weight of the movement trajectory integrity index can be increased. For martial arts movements, since there are a lot of limb extensions and body rotations during the movements, the weights of the visibility index of key movement areas and the human body spatial relationship maintenance index can be increased. For sports-related movements, since the human body's support relationship and movement posture are quite important, the weight of the human body's spatial relationship maintenance index can be increased.

[0207] Furthermore, based on the evaluation results and corresponding weights of each evaluation indicator, the action information is used to maintain the comprehensive evaluation result.

[0208] Specifically, when the key action area can be effectively observed, the action trajectory can be continuously expressed, and the human spatial relationship can be maintained, the current virtual camera parameters are determined to meet the requirements for maintaining action information; when any evaluation index is lower than the preset threshold, the current virtual camera parameters are determined to have a risk of loss of action information.

[0209] Furthermore, based on the comprehensive evaluation results of the motion information, optimization decisions are made for the virtual camera parameters.

[0210] When the motion information and the overall evaluation result meet the preset conditions, the current virtual camera parameters are maintained, and the attitude control sequence corresponding to the target viewpoint is generated using the virtual camera parameters.

[0211] When the overall evaluation result of motion information does not meet the preset conditions, the type of motion information loss is determined based on the evaluation result, and the corresponding virtual camera parameters are adjusted.

[0212] For example, when the visibility index of key action areas is low, the spatial position of the virtual camera is adjusted first to create spatial separation of the key action areas; when the integrity index of action trajectory is low, the observation direction of the virtual camera is adjusted so that the main direction of motion can be fully reflected in the two-dimensional projection plane; when the human body spatial relationship retention index is low, the distance, field of view, or height parameters of the virtual camera are adjusted to improve the observability of the overall human body structure.

[0213] Through the aforementioned comprehensive evaluation mechanism for maintaining motion information, this embodiment establishes a closed-loop control relationship of "virtual camera parameters - motion expression effect - camera parameter optimization," so that the virtual camera adjustment process is no longer based on experience-based manual settings, but rather on adaptive optimization according to the requirements for maintaining target motion information.

[0214] Therefore, while ensuring the user-specified viewing direction, this embodiment can reduce problems such as loss of key action areas, incomplete action trajectories, and distortion of human spatial structure caused by changes in viewing angle, thereby improving the consistency of action expression during multi-view video generation.

[0215] In this embodiment, an evaluation model for motion information preservation is established to assess the degree of preservation of target motion information under different virtual camera parameters.

[0216] The motion information retention evaluation value is used to determine whether the current observation perspective will lead to the loss of motion information.

[0217] Its expression is:

[0218] in: The evaluation value is used to preserve motion information. It is a dimensionless parameter with a value range of 0 to 1. This is a key indicator of the visibility of the action area. As an indicator of the integrity of the motion trajectory; Indicators for maintaining spatial relationships within the human body; , , For the corresponding weight coefficients, and satisfying:

[0219] (1) Visibility index of key action area

[0220] in: This represents the effective visible area of ​​the key action region from the current virtual camera's perspective. This represents the total projected area of ​​the critical action zone.

[0221] in:

[0222] When the critical action area is unobstructed Approaching 1.

[0223] (2) Motion trajectory integrity index

[0224] in: The normalized trajectory length of the key joint in three-dimensional space; The length of the two-dimensional motion trajectory after being projected by the virtual camera.

[0225] To eliminate the influence of spatial scale, the trajectory length is normalized.

[0226] When a two-dimensional projection can fully reflect the trend of three-dimensional motion changes, T approaches 1.

[0227] (3) Human spatial relationship maintenance index

[0228] in: The number of key joints in the human body; Let be the normalized distance of the k-th group of human joints in three-dimensional space; This corresponds to the two-dimensional projection distance; It is a normalized scale for human body space.

[0229] When a two-dimensional projection can preserve the structural relationships of the human body, R approaches 1.

[0230] when If the current virtual camera parameters are determined to be insufficient to maintain motion information, the camera parameters will be adjusted accordingly.

[0231] in, Maintain the evaluation threshold for the preset action.

[0232] S3.5. Judging the expressibility of viewpoint based on the requirement of maintaining motion information.

[0233] In this embodiment, after obtaining the target viewing angle and key action area, a virtual camera is used to perform pre-projection analysis on the target's three-dimensional action data.

[0234] Specifically, based on the current virtual camera parameters, the human joints in the target 3D motion data are projected onto the virtual imaging plane, and the projection state corresponding to the key motion areas is analyzed.

[0235] The projection state includes at least the following: The degree of spatial overlap between key action areas; Visibility of key action areas; The degree of continuity of the movement trajectory in the key action area.

[0236] For example, when analyzing hand movements, it can be determined whether the projection position of the hand remains continuously changing across multiple time frames; if there are many instances where the projection trajectory of the hand is obscured by the torso, it indicates that the current observation perspective cannot fully express the hand movements.

[0237] Furthermore, this embodiment determines whether the current observation perspective meets the motion information preservation condition based on the above analysis results.

[0238] Specifically, a risk of loss of action information can be determined when any of the following conditions are met: (1) The critical action area has a high occlusion rate in multiple consecutive time frames; (2) The spatial relationship between key action areas cannot be effectively distinguished in the projection plane; (3) The motion trajectory corresponding to the key action area is discontinuous.

[0239] When the above situation is detected, it is considered that the current virtual camera parameters do not meet the requirements for motion information preservation and further parameter adjustments are required.

[0240] S3.6, Adaptive adjustment of virtual camera parameters

[0241] In this embodiment, when it is detected that the current viewing angle cannot fully express the target action, the virtual camera parameters are adjusted.

[0242] Specifically, based on the evaluation results maintained by the motion information, at least one of the following parameters is adjusted: Virtual camera spatial position; virtual camera viewing direction; virtual camera height; virtual camera focal length; virtual camera field of view.

[0243] For example, when the human hand movement area is detected to be obscured by the torso, the horizontal position of the virtual camera can be adjusted to create spatial separation between the hand area and the torso area in the projection plane; when the leg movement area is detected to have a high degree of overlap in the top-down view, the height and viewing direction of the virtual camera can be adjusted to ensure that the movement trajectory of both legs can be fully expressed.

[0244] Furthermore, after each adjustment of the virtual camera parameters, the key motion region projection analysis is re-executed until the target virtual camera parameters that meet the motion information preservation conditions are obtained.

[0245] Through the above iterative optimization method, the virtual camera parameters are no longer determined solely by user input, but are adaptively adjusted based on the characteristics of the target action itself.

[0246] S3.7 Motion control information conversion process based on motion information holding constraints

[0247] In this embodiment, after determining the target virtual camera parameters that meet the motion information preservation requirements, the target three-dimensional motion data is further transformed based on the target virtual camera parameters to generate motion control information for driving the video generation model.

[0248] like Figure 3 As shown, the motion control information conversion process in this embodiment is not a simple conversion of three-dimensional motion data to two-dimensional posture data. Instead, a motion information maintenance constraint mechanism is introduced during the transmission of three-dimensional motion information to ensure that the spatial structure, motion trajectory, and key motion features of human motion remain consistent across different data representation formats.

[0249] Specifically, the motion control information conversion process includes motion data filtering, viewpoint maintenance optimization, three-dimensional motion projection, and posture consistency verification.

[0250] First, based on the aforementioned action consistency judgment results, target three-dimensional action data that meets the action retention conditions are determined from the candidate three-dimensional action data.

[0251] The target three-dimensional motion data includes a three-dimensional spatial position sequence corresponding to key joints of the human body, a joint rotation parameter sequence, and motion time sequence information.

[0252] Through the motion consistency judgment process, the 3D motion data input into the subsequent conversion process not only meets the semantic requirements of the user's input motion, but also meets the requirements of the continuity of human joint movement, the law of spatial trajectory change, and the stability of the overall human motion.

[0253] Furthermore, a corresponding virtual camera model is established based on the target observation perspective information, and the current virtual camera parameters are analyzed using a motion information retention evaluation mechanism.

[0254] Specifically, based on the key motion regions in the target's 3D motion data, the motion information that needs to be emphasized during the motion expression process is determined, and each is evaluated from the current observation perspective: The visibility of key movement areas; the completeness of movement trajectories; and the degree to which the spatial structure of the human body is preserved.

[0255] When the motion information evaluation result meets the preset conditions, the current virtual camera parameters are determined as the target virtual camera parameters.

[0256] If the motion information preservation evaluation result does not meet the preset conditions, the position parameters, orientation parameters and / or field of view parameters of the virtual camera are adjusted according to the type of motion information loss, and the motion information preservation evaluation is re-executed until the target virtual camera parameters that meet the motion expression requirements are obtained.

[0257] Furthermore, based on the target virtual camera parameters, the target's three-dimensional motion data is subjected to spatial projection transformation.

[0258] Specifically, the three-dimensional spatial coordinates corresponding to each key joint of the human body in the target three-dimensional motion data are obtained, and a mapping relationship between the three-dimensional spatial coordinates and the two-dimensional image plane coordinates is established based on the positional relationship of the virtual camera, the viewing direction, and the projection parameters.

[0259] Through the mapping relationship, the human body joints in three-dimensional space are converted into two-dimensional key point positions under the target observation view, thereby generating a two-dimensional posture control sequence corresponding to the observation view.

[0260] However, since human motion in three-dimensional space contains information about changes in depth direction, and the two-dimensional projection process may weaken or lose some motion information, this embodiment further uses the spatial motion laws in the original three-dimensional motion data to perform consistency verification on the two-dimensional posture control sequence after generating the two-dimensional posture control sequence.

[0261] Specifically, the direction of motion, amplitude of motion, and spatial relationship between human joints in the three-dimensional motion data are compared with the corresponding motion features in the two-dimensional posture control sequence.

[0262] When it is detected that: the direction of movement of two-dimensional key points is inconsistent with the trend of three-dimensional motion changes; the amplitude of movement of two-dimensional key points cannot reflect the amplitude of three-dimensional motion; or there are abnormal changes in the connection relationship of human skeleton, it is determined that there is a loss of motion information in the current two-dimensional posture control sequence.

[0263] In response to the above-mentioned anomalies, the abnormal two-dimensional posture data is corrected based on the motion patterns in the three-dimensional motion data and the information of adjacent normal posture frames, so that the corrected two-dimensional posture control sequence can maintain the original three-dimensional motion characteristics.

[0264] Through the above processing, motion control information optimized by motion information preservation constraints is obtained.

[0265] The motion control information includes: The target's three-dimensional motion features after motion consistency judgment; the two-dimensional attitude control sequence after viewpoint optimization; and the attitude correction information after three-dimensional-two-dimensional consistency verification.

[0266] Furthermore, the motion control information is input into the video generation model, so that the video generation model generates a video of the target person performing the target action based on the motion control information.

[0267] Through the above-described motion control information conversion process based on motion information maintaining constraints, this embodiment achieves stable transmission between three-dimensional human motion data and video generation control information.

[0268] Compared to existing methods that directly use two-dimensional pose data or unprocessed three-dimensional motion data to drive video generation, this embodiment establishes corresponding constraint mechanisms in the processes of motion data selection, observation perspective determination, and pose transformation. This ensures that the spatial structure and motion laws of human movements are maintained during the transformation process, thereby reducing problems such as motion deformation, perspective distortion, and pose abnormalities, and improving the accuracy and naturalness of the human movement performance in the generated video.

[0269] S4. Three-dimensional-two-dimensional attitude consistency verification and anomaly correction based on three-dimensional motion laws

[0270] In this embodiment, after determining the target virtual camera parameters, it is necessary to further convert the target's three-dimensional motion data into attitude control information suitable for the video generation model.

[0271] Specifically, since existing video generation models typically receive two-dimensional pose sequences or two-dimensional pose images as motion control conditions, it is necessary to convert human motion data in three-dimensional space into two-dimensional pose control sequences from the target's observation perspective.

[0272] However, during the conversion from 3D motion data to 2D pose data, factors such as changes in viewing angle, spatial projection relationships, and occlusion of human joints may cause the 2D pose control sequence to fail to accurately reflect the motion changes in the original 3D motion. For example, when a human arm moves along the depth direction in 3D space, the change in its 2D projection position may be small, or even the motion trend may be unclear. If this 2D pose sequence is directly used to control video generation, it may lead to problems such as motion pauses, insufficient motion amplitude, or abnormal limb movements in the generated video.

[0273] Therefore, in this embodiment, after generating a two-dimensional attitude control sequence from a three-dimensional motion projection, the consistency of the two-dimensional attitude control sequence is further verified based on the three-dimensional motion law to determine whether the two-dimensional attitude information retains the motion characteristics in the original three-dimensional motion, and attitude correction is performed when an anomaly is detected.

[0274] Specifically, the steps include the following: S4.1, 3D motion data projection conversion In this embodiment, the target's three-dimensional motion data is first converted into a two-dimensional attitude control sequence under the corresponding viewing angle based on the target virtual camera parameters.

[0275] Specifically, the three-dimensional spatial coordinate information corresponding to each joint point of the human body in the target three-dimensional motion data is obtained, and the mapping relationship between the three-dimensional space and the two-dimensional imaging plane is established based on the position parameters, orientation parameters and projection parameters of the virtual camera.

[0276] Furthermore, based on the mapping relationship, the three-dimensional coordinates of human joints in each time frame are converted into key point coordinates in a two-dimensional plane.

[0277] For example, for a human joint point in time frame t, it corresponds to a spatial position in three-dimensional space. After projection through a virtual camera, the corresponding position of the joint point in the two-dimensional image plane can be obtained.

[0278] By performing the above projection processing on multiple consecutive time frames, a two-dimensional attitude control sequence containing the motion changes of human key points is obtained.

[0279] S4.2 Establishing three-dimensional motion change reference information

[0280] In this embodiment, in order to determine whether the two-dimensional attitude control sequence accurately maintains the original motion information, three-dimensional motion change reference information is first established based on the target three-dimensional motion data.

[0281] Specifically, the three-dimensional motion state of the corresponding joint is calculated based on the changes in the three-dimensional spatial position of the human joint in continuous time frames.

[0282] The three-dimensional motion states include: Direction of spatial movement of human joints; distance of spatial movement of human joints; changes in the speed of movement of human joints; trends in the rotation of human joints.

[0283] For example, when analyzing arm movements, the overall direction and range of motion of the arm can be determined by the positional changes of the shoulder, elbow, and wrist joints in three-dimensional space.

[0284] Furthermore, based on the spatial relationships between multiple joints in the human body, topological constraints on human motion are established.

[0285] Among them, the topological constraints of human motion are used to describe the stability of the human skeletal structure during motion, for example: The connection between the shoulder and elbow; the connection between the elbow and wrist; the connection between the hip and knee.

[0286] The above methods can be used to obtain three-dimensional motion reference information that reflects the laws of real human movement.

[0287] S4.3, Consistency Detection of Two-Dimensional Attitude Control Sequence

[0288] After obtaining the two-dimensional attitude control sequence and the three-dimensional motion reference information, this embodiment further performs consistency detection.

[0289] Specifically, the motion change trends of human key points in the two-dimensional posture control sequence are compared with the motion change trends in the corresponding three-dimensional motion data.

[0290] The comparison process includes: Compare whether the movement directions of key joint points are consistent; Compare whether the changes in the amplitude of motion at key joints satisfy the mapping relationship; Compare whether the spatial relationship between adjacent human joints is maintained.

[0291] For example, in three-dimensional motion data, the wrist joint moves along the front of the body while the elbow joint maintains a stable connection. If, after two-dimensional projection, the wrist key point in the two-dimensional attitude control sequence shows abnormal stagnation, or the distance relationship between the elbow and wrist changes significantly, it indicates that the two-dimensional attitude control sequence has failed to accurately maintain the three-dimensional motion information.

[0292] Furthermore, the attitude consistency evaluation result is determined based on the above comparison results.

[0293] When the two-dimensional attitude control sequence meets the preset consistency condition, it is considered that the current attitude control information can effectively express the original three-dimensional motion.

[0294] When the two-dimensional attitude control sequence does not meet the preset consistency conditions, it is determined that there is a loss of motion information and attitude correction processing needs to be performed.

[0295] In this embodiment, a three-dimensional two-dimensional attitude consistency error model is established to determine whether the two-dimensional attitude control sequence accurately maintains the original three-dimensional motion features.

[0296] Its expression is:

[0297] in, This refers to attitude consistency error; This refers to the positional error of key points; This represents the error in motion trend. Errors related to human anatomy; , , Let the weighting coefficients satisfy:

[0298] (1) Key point position error

[0299] in, This represents the position of the i-th keypoint in the two-dimensional attitude control sequence. This is the reference position obtained from the 3D motion projection.

[0300] (2) Motion trend error

[0301] in, This represents the change in motion direction in a two-dimensional attitude sequence; This refers to the change in the direction of motion corresponding to a three-dimensional action. This is the normalized scale of motion change.

[0302] (3) Error in skeletal structure relationship

[0303] in, Let be the length of the k-th bone connection group in the two-dimensional pose; The connection length corresponding to the 3D motion projection; It is a normalized scale for the human body.

[0304] when When the current two-dimensional attitude control sequence is found to have lost motion information, attitude correction is performed.

[0305] in, This is a preset attitude error threshold.

[0306] S4.4, Abnormal Attitude Data Correction

[0307] In this embodiment, when an anomaly is detected in the two-dimensional attitude control sequence, the abnormal attitude data is corrected according to the three-dimensional motion law.

[0308] Specifically, the first step is to determine the human body key points corresponding to the abnormal pose frames.

[0309] Then, based on the motion trends corresponding to normal pose frames before and after the abnormal pose frames, and combined with the spatial motion constraints in the 3D motion data, the positions of the abnormal key points are adjusted.

[0310] For example, when the hand key points in a certain time frame are abnormally offset due to projection, the position of the two-dimensional key points in that time frame can be corrected based on the hand movement direction, movement speed, and three-dimensional spatial movement trajectory in the preceding and following consecutive frames.

[0311] Furthermore, the connection relationships of the human skeleton remain unchanged during the correction process.

[0312] For example, when correcting the position of the wrist key points, the distance relationship between the wrist and the elbow is constrained at the same time so that the corrected two-dimensional posture still conforms to the human body structure.

[0313] The above methods can reduce pose distortion caused by the 3D projection process.

[0314] S4.5 Generate the optimized attitude control sequence

[0315] After consistency verification and anomaly correction, the optimized two-dimensional attitude control sequence is obtained.

[0316] The optimized two-dimensional attitude control sequence can simultaneously satisfy: Requirements for the target observation perspective; constraints of the human skeletal structure; laws governing three-dimensional motion.

[0317] Furthermore, the optimized two-dimensional attitude control sequence is used as motion control information in the video generation model.

[0318] In this embodiment, instead of directly obtaining two-dimensional posture data by three-dimensional motion projection, the original three-dimensional motion information is used to perform reverse verification and constraint correction on the two-dimensional posture results after the projection conversion.

[0319] Compared to existing methods that rely solely on two-dimensional posture sequences to control video generation, this embodiment utilizes three-dimensional spatial motion information as a constraint benchmark, enabling the two-dimensional posture control information to more accurately maintain the spatial structure and motion change patterns during human movement, thereby reducing problems such as motion deviation, limb abnormalities, and discontinuous motion during video generation.

[0320] S5. Human motion adaptation based on the target person's appearance structure

[0321] In this embodiment, after obtaining the two-dimensional posture control sequence after consistency verification, in order to enable the target person in the generated video to perform the target action while maintaining their own appearance features, it is necessary to further adapt the posture control sequence to the human body structure of the target person.

[0322] Specifically, due to differences in height ratio, limb length, body outline, and body structure among different individuals, directly applying the posture control sequence corresponding to the standard human skeleton to different individuals may result in problems such as abnormal limb proportions, uncoordinated movements, or distorted human body structure in the generated video.

[0323] Therefore, in this embodiment, human body structure parameters are established based on the appearance information of the target person, and the posture control sequence is adapted using the human body structure parameters so that the target action can conform to the human body structure characteristics of the target person.

[0324] S5.1 Obtain the appearance feature information of the target person

[0325] In this embodiment, the appearance information of the target person corresponding to the video to be generated is first obtained.

[0326] The appearance information of the target person can be obtained through user-input images of the person, video frames of the person, or pre-stored digital models of the person.

[0327] Furthermore, feature extraction is performed on the appearance information of the target person to obtain appearance feature parameters related to human body structure.

[0328] The appearance feature parameters include: Human body contour information; human body proportion information; dimensions of key human body parts; personal identification information.

[0329] Among them, human body contour information is used to describe the overall external structure of the target person; Human proportion information is used to describe the proportional relationships between different body parts; Personal identification information is used to maintain consistency in the appearance of people in the generated video.

[0330] For example, for the same action, different people may have different arm lengths and leg proportions, so it is necessary to adjust the motion control information according to the target person's own structure.

[0331] S5.2 Establishing a mapping relationship for human body structure

[0332] In this embodiment, a mapping relationship between the target person's human body structure and the standard motion skeletal model is established based on the target person's appearance feature information.

[0333] Specifically, the first step is to identify the key body parts corresponding to the target person.

[0334] For example, the shoulder area; the elbow area; the wrist area; the hip area; the knee area; and the ankle area.

[0335] Then, based on the human proportion parameters of the target person, the proportional difference between the human body structure parameters of the target person and the standard skeletal model is calculated.

[0336] For example, if the target character's arm length is greater than the corresponding length of the standard skeletal model, then the corresponding arm area needs to be scaled; if the target character's leg proportions differ from the standard skeletal model, then the leg movement trajectory needs to be adapted accordingly.

[0337] The above methods enable standardized motion data to be adapted to different human body structures.

[0338] S5.3, Attitude Control Sequence Structure Adaptation

[0339] In this embodiment, the two-dimensional posture control sequence is adjusted according to the established human body structure mapping relationship.

[0340] Specifically, while keeping the movement patterns unchanged, the positional relationships between key points of the human body are scaled.

[0341] For example, for arm swinging movements, while keeping the movement relationship between the shoulder, elbow, and wrist unchanged, the range of motion of the wrist is adjusted according to the arm length of the target person; for walking movements, while keeping the gait cycle unchanged, the foot movement trajectory is adjusted according to the leg proportion of the target person.

[0342] Furthermore, during the adjustment process, the connection between the human skeleton and the constraint relationship of joint movement should not be disrupted.

[0343] In other words, this embodiment does not simply scale the two-dimensional posture control sequence as a whole, but rather adapts it locally according to the structural features of different regions of the human body, so that the motion control information can conform to the real human body structure of the target person.

[0344] S5.4 Fusion of Appearance Features and Attitude Control Information

[0345] Furthermore, after completing the human body structure adaptation, the appearance features of the target person are fused with the optimized posture control sequence.

[0346] Specifically, feature encoding can be used to separately address: The target person's physical appearance, movement and posture characteristics, and human body structure adaptation information are encoded.

[0347] Then, the above features are input into the feature fusion module to establish a correspondence between the appearance information and the motion control information.

[0348] The feature fusion process can employ attention mechanisms, cross-attention mechanisms, or other fusion methods that can achieve multi-source feature association.

[0349] In this way, the video generation model can simultaneously refer to the following when generating a video of the target person's actions: Characteristic features; physical appearance; human anatomy; movement and posture.

[0350] This reduces issues such as facial changes, abnormal body proportions, and awkward movements caused by mismatches between character appearance and motion control information.

[0351] S6. Video generation based on constraint optimization of motion control information

[0352] In this embodiment, after completing the motion information preservation constraint processing and character structure adaptation, the finally obtained video generation control information is input into the video generation model to generate the target video.

[0353] Specifically, the video generation control information includes at least: The target's 3D motion information after motion consistency judgment; the 2D posture control sequence after viewpoint optimization; the posture correction information after 3D and 2D consistency verification; and the target's appearance feature information.

[0354] In one implementation, the video generation model may be a diffusion-based video generation model, a temporal Transformer-based video generation model, or other generation models capable of generating continuous video frames based on multi-source control information.

[0355] After receiving the control information, the video generation model processes the character's appearance information, action posture information, and viewing angle information together to generate a video of the target character performing the target action.

[0356] S6.1 Video Timing Generation

[0357] During the video generation process, the video generation model determines the character's motion state in each video frame based on the input pose control sequence.

[0358] Since the input attitude control sequence has already been processed: Action consistency assessment; viewpoint information preservation; 3D / 2D consistency verification; human body structure adaptation; Therefore, the video generation model does not need to determine changes in human actions through a large amount of random inference, but can generate continuous video frames based on explicit motion control information.

[0359] S6.2, Video Quality Optimization

[0360] Furthermore, after generating the initial video, video quality optimization processing can be performed according to application requirements.

[0361] For example, it can enhance video resolution, optimize inter-frame stability, and adjust video colors. It can also detect motion in the generated video.

[0362] Specifically, it can detect the continuity of movement at key joints of the human body; the consistency of movement trajectories; and the degree to which a person's identity is maintained.

[0363] When a motion deviation is detected in the generated result, the detection result can be fed back to the motion data matching stage or the posture optimization stage to adjust the corresponding parameters.

[0364] Through the above steps, the AIGC video generation method based on motion capture data provided in this embodiment performs motion consistency judgment, viewpoint information preservation optimization, three-dimensional and two-dimensional pose consistency verification, and human body structure adaptation processing in sequence before the three-dimensional motion data enters the video generation model, so that motion information can be continuously transmitted in different processing stages.

[0365] Specifically, a 3D motion database provides complete human spatial motion information, avoiding the problem of missing depth information in traditional 2D motion reference data; a motion consistency judgment mechanism improves the matching degree between motion data and user needs; a virtual camera optimization mechanism based on motion information preservation requirements enables the same motion to adapt to different viewing angles; a posture consistency verification mechanism under the constraints of 3D motion laws reduces motion information loss during the conversion of 3D motion into 2D control information; and a human body structure adaptation mechanism improves the naturalness of different people performing target actions.

[0366] Therefore, this embodiment can improve the accuracy, continuity, and perspective adaptability of character movements in the generated video while ensuring the consistency of the character's identity characteristics.

[0367] Example 2: A Multi-View Video Generation Method Based on Motion Information Preservation Constraints

[0368] In this embodiment, a multi-view video generation method based on motion information preservation constraints is provided.

[0369] This embodiment further optimizes the AIGC video generation method based on motion capture data as described in Embodiment 1, addressing the issue of generating videos from multiple viewing angles for the same action.

[0370] In traditional video generation solutions, when users need to generate videos of the same person's actions from different shooting angles, they typically need to input motion reference materials from different perspectives, or re-execute the video generation process for each viewing angle. Because the two-dimensional motion reference data is bound to the shooting perspective, it is difficult to directly reuse the same action from different perspectives.

[0371] In this embodiment, since the motion data comes from a three-dimensional motion database and human motion information is stored in three-dimensional spatial motion relationships, videos corresponding to different viewing angles can be generated by adjusting the virtual camera parameters while keeping the human motion unchanged.

[0372] Furthermore, this embodiment does not simply set multiple camera positions for the same 3D motion data, but rather judges the retention of motion information under different viewing angles during the multi-view generation process, and optimizes the corresponding virtual camera parameters based on the judgment results, thereby ensuring that the videos generated from different perspectives can fully express the target motion.

[0373] Specifically, the steps include the following: S201, Requirements for acquiring target motion data and multiple observation perspectives In this embodiment, the target motion data corresponding to the video to be generated is first obtained.

[0374] The target motion data can be the target three-dimensional motion data after motion consistency judgment in Embodiment 1.

[0375] Specifically, the target three-dimensional motion data includes information on the three-dimensional spatial position changes of each joint point of the human body, information on joint rotation changes, and information on motion time sequence.

[0376] Furthermore, multiple target observation perspectives are obtained based on the user's video production needs.

[0377] For example, in film and television production scenarios, users may need to generate separate sets for the same martial arts move: Frontal view video; left-side view video; right-side view video; top-down view video.

[0378] In this embodiment, the observation viewpoint information of each target corresponds to a set of virtual camera parameters.

[0379] The set of virtual camera parameters includes: Virtual camera spatial position; virtual camera orientation; virtual camera focal length; virtual camera field of view.

[0380] S202, Establish initial viewpoints for multiple virtual cameras

[0381] After acquiring multiple target observation perspectives, corresponding virtual cameras are created based on each target observation perspective.

[0382] Specifically, multiple virtual cameras are set up in three-dimensional space based on the observation direction input by the user.

[0383] For example: To meet the requirements of frontal observation, a first virtual camera is set up in front of the target's three-dimensional motion data; To meet the requirements for side observation, a second virtual camera is set up to the side of the target's 3D motion data; To meet the needs of top-down observation, a third virtual camera is set up above the target's three-dimensional motion data.

[0384] By setting up multiple virtual cameras, the 3D motion data of the same target can be analyzed and processed simultaneously from multiple viewing angles.

[0385] S203. Multi-view expressibility analysis based on motion information preservation requirements.

[0386] In this embodiment, before generating corresponding viewpoint videos using different virtual cameras, it is further determined whether each observation viewpoint can fully express the target action.

[0387] Specifically, for each virtual camera, the target's 3D motion data is pre-projected and analyzed using the corresponding camera parameters.

[0388] The pre-projection analysis includes: Visibility analysis of key action areas; Spatial relationship analysis of human joints; Analysis of motion trajectory continuity.

[0389] Specifically, first, the key action area is determined based on the type of target action.

[0390] For example: For dance movements, the focus is on analyzing the areas of hand movement, leg movement, and body rotation. For martial arts movements, the focus is on analyzing the movement areas of the fist, the extension areas of the arms, and the rotation areas of the torso; For sports movements, the focus is on analyzing the body's support areas and the areas where force is generated.

[0391] Subsequently, based on the observation angles of each virtual camera, the distribution of key action areas in the two-dimensional projection plane was analyzed.

[0392] S204. Determine the retention status of motion information under different observation perspectives.

[0393] In this embodiment, the state of motion information retention corresponding to different viewing angles is determined based on the pre-projection analysis results.

[0394] Specifically, it can be evaluated based on at least one of the following indicators: (1) Evaluation of occlusion in key areas It is used to determine whether the key movement area is obscured by other parts of the body.

[0395] For example, when observing arm swinging movements from a side viewpoint, if the hand area overlaps with the human torso area for a long period of time, it is considered that there is a risk of loss of motion information in the current viewpoint.

[0396] (2) Evaluation of the integrity of the motion trajectory

[0397] Used to determine whether the motion trajectory corresponding to the key action area is continuous.

[0398] For example, if the projected positions of the hand in different time frames cannot form a continuous motion trajectory for a continuous waving motion, it means that the current perspective cannot accurately reflect the motion process.

[0399] (3) Evaluation of the maintenance of human spatial relationships

[0400] It is used to determine whether the positional relationship between different joints of the human body can be correctly expressed from the perspective of observation.

[0401] For example, in alternating leg movements, if the spatial relationship between the left and right legs cannot be distinguished from the current perspective, it may affect the performance of the movement.

[0402] Furthermore, based on the above evaluation results, a motion retention evaluation value corresponding to the observation perspective is generated.

[0403] S205, Virtual Camera Parameter Adaptive Optimization

[0404] In this embodiment, when the action maintenance evaluation value corresponding to a certain observation view is lower than the preset condition, the virtual camera parameters corresponding to that observation view are optimized and adjusted.

[0405] Specifically, the corresponding adjustment method is selected based on the type of action information loss.

[0406] For example, when a critical action area is detected to be occluded, the horizontal position of the virtual camera is adjusted to create spatial separation of the critical area in the projection plane; When insufficient expression of motion trajectory is detected, the virtual camera's observation direction is adjusted to make the direction of motion change more obvious; When insufficient representation of human spatial relationships is detected, the height or field of view of the virtual camera is adjusted to improve the observability of the overall human structure.

[0407] Furthermore, after adjusting the virtual camera parameters, the motion information is re-evaluated.

[0408] When the adjusted viewing angle meets the preset motion retention conditions, the set of virtual camera parameters is determined as the target viewing angle parameters.

[0409] By using the above methods, the video generation process from different viewing angles can meet the requirements of action expression.

[0410] S206. Generate attitude control sequences corresponding to multiple viewpoints.

[0411] After determining the parameters of multiple target virtual cameras, the target 3D motion data are projected using each target virtual camera.

[0412] Specifically, for the first virtual camera, a two-dimensional attitude control sequence corresponding to the first viewing angle is generated; for the second virtual camera, a two-dimensional attitude control sequence corresponding to the second viewing angle is generated; and for the third virtual camera, a two-dimensional attitude control sequence corresponding to the third viewing angle is generated.

[0413] Furthermore, each two-dimensional attitude control sequence executes the three-dimensional-two-dimensional consistency verification process in Example 1.

[0414] When there is a motion information deviation in the two-dimensional attitude control sequence corresponding to any viewpoint, the attitude sequence corresponding to that viewpoint is independently corrected.

[0415] S207, Multi-view Video Generation

[0416] In this embodiment, the posture control sequences corresponding to different viewing angles are fused with the appearance features of the target person and then input into the video generation model.

[0417] The video generation model generates video results from multiple observation perspectives based on the control information corresponding to different viewpoints.

[0418] For example, when the same character performs the same set of dance moves, they can generate simultaneously: Frontal dance video; 45-degree angle dance video; side view dance video; top-down view dance video.

[0419] Since all videos are generated based on the same 3D motion data, the characters' movements remain consistent across different videos, with only the viewing angle differing.

[0420] Through the above method, this embodiment realizes the generation of multi-view videos based on the same three-dimensional motion data.

[0421] Compared with the prior art, this embodiment has the following beneficial effects: First, by using 3D motion data as the basis for motion control, the same motion can be freed from the limitations of the original acquisition perspective, eliminating the need to acquire motion reference materials again for different observation angles.

[0422] Secondly, by introducing a motion information retention and judgment mechanism during the multi-view generation process, the virtual camera parameters not only meet the user's perspective requirements, but also ensure that key motion areas, human spatial relationships, and motion trajectories can be fully expressed.

[0423] Furthermore, by performing consistency checks on the attitude control sequences corresponding to different viewing angles, distortion of motion control information due to changes in viewing angle can be avoided.

[0424] Therefore, this embodiment achieves effective decoupling between action content and viewing perspective, enabling the same 3D action data to quickly generate video results from multiple perspectives, improving video content production efficiency, and ensuring consistency of action performance between videos from different perspectives.

[0425] Example 3: Motion-Preserving Constraint Video Generation Method Based on Feedback Optimization of Generation Results

[0426] In this embodiment, a motion-keeping constraint video generation method based on feedback optimization of generation results is provided.

[0427] Based on Embodiments 1 and 2, this embodiment further addresses the issue of unstable generation results caused by differences in the generation model's inference errors, the complexity of the character's appearance, and the degree of adaptation of motion control information during the video generation process. It introduces a closed-loop optimization mechanism based on the feedback of the generation results.

[0428] Existing video generation methods typically output the target video directly after completing motion data input and model inference, lacking mechanisms for motion consistency detection and feedback adjustment of the generated results. Therefore, even if the input video control information has been optimized, errors in feature mapping within the generation model may still lead to problems such as deviations in motion amplitude, changes in motion rhythm, abnormal character postures, or insufficient retention of identity features in the generated video.

[0429] Therefore, this embodiment further performs multi-dimensional quality evaluation on the generated results after video generation, determines the generation error type based on the evaluation results, and feeds the error information back to the motion data selection, motion holding constraint parameters, and posture control parameter adjustment processes, thereby forming a closed-loop processing process of "generation-detection-optimization-regeneration".

[0430] Specifically, the steps include the following: S301. Generate initial video based on motion preservation constraints. In this embodiment, motion control information optimized by motion retention constraints is first obtained according to the method described in Embodiment 1.

[0431] The motion control information includes: Target 3D motion data after motion consistency assessment; Virtual camera parameters optimized to maintain the viewing angle; The attitude control sequence after 3D-2D consistency verification; Control information of the target person after human body structure adaptation.

[0432] Furthermore, the aforementioned motion control information is input into the video generation model.

[0433] The video generation model infers and generates an initial target video by reasoning from consecutive video frames based on the input information on the person's appearance, posture, and viewing angle.

[0434] The video generation model can be a video generation network based on a diffusion model, a video generation network based on a temporal Transformer, or other generation models that can generate video content based on motion control information.

[0435] S302. Extract motion performance information from the generated video.

[0436] In this embodiment, after obtaining the initial target video, motion performance information is extracted from the generated video.

[0437] Specifically, human pose detection is performed on each video frame in the generated video to obtain a two-dimensional pose sequence corresponding to the generated character.

[0438] The two-dimensional attitude sequence includes: Location of key joints in the human body; skeletal connections in the human body; movement trajectories of joints over time.

[0439] Furthermore, motion performance features in the generated video are extracted based on the two-dimensional pose sequence.

[0440] The characteristics of the action performance include: The movement trajectory of key joints in the human body; changes in joint movement speed; the relationship between the execution time of the movement; and the trend of changes in human posture.

[0441] The above method is used to obtain the actual human motion state presented in the generated video.

[0442] S303. Establish a consistency evaluation model for generated results and actions.

[0443] In this embodiment, to determine whether the generated video accurately retains the target motion information, the motion performance information corresponding to the generated video is compared and analyzed with the target motion control information.

[0444] Specifically, the human motion state in the generated video is matched with the motion reference information corresponding to the target 3D motion data.

[0445] The comparisons include: (1) Evaluation of motion space consistency The motion trajectories of key human joints in the generated video are compared with the reference trajectory of the target 3D motion data after projection.

[0446] For example, for arm swinging motions, compare whether the range of motion of the wrist joint in the generated video is consistent with the range of motion of the target motion. For rotational movements, compare whether the changes in the rotation angle of the human torso conform to the target movement.

[0447] When the range of motion of the key motion area in the generated video deviates significantly from the target motion, it is determined that there is a motion space deviation.

[0448] (2) Evaluation of the continuity of action time

[0449] Furthermore, the changes in human posture between consecutive frames in the generated video are analyzed.

[0450] Specifically, this includes the magnitude of changes in key points between adjacent video frames; the trend of changes in motion speed; and the consistency of motion rhythm.

[0451] For example, if the arm extends in a continuous acceleration manner during the target action, but the generated video shows a sudden stop or rapid jump, then the generated result is judged to have a time continuity error.

[0452] (3) Evaluation of human body structure consistency

[0453] Furthermore, the human anatomy information in the generated video is compared with the human anatomy information of the target person.

[0454] For example, detecting human body proportions; limb length relationships; joint connection relationships.

[0455] When abnormal human proportions, changes in limb length, or incorrect joint connections appear in the generated video, it is determined that there is an error in the consistency of human body structure.

[0456] S304. Determine the type of generation error.

[0457] In this embodiment, the problems existing in the generated video are classified according to the above-mentioned action consistency evaluation results.

[0458] Specifically, the generation error can include: Action matching error; perspective expression error; posture transformation error; character structure adaptation error.

[0459] Among them, when the generated action differs significantly from the target action, it is identified as an action matching error; when the action content is correct but some key areas cannot be clearly expressed, it is identified as a perspective expression error; when there are abnormal jumps in the action trajectory, it is identified as a posture conversion error; and when the body proportions of the character are abnormal when performing the action, it is identified as a human body structure adaptation error.

[0460] By distinguishing error types, the subsequent optimization process can take corresponding adjustment measures for different problems.

[0461] S305. Determine the closed-loop optimization object based on the error type.

[0462] In this embodiment, after determining the error type corresponding to the generated video, the video generation model is not directly adjusted as a whole. Instead, the corresponding closed-loop optimization object is determined based on the location and type of the error.

[0463] Specifically, the video generation process in this embodiment includes: Motion data selection process; motion consistency constraint process; viewpoint preservation optimization process; 3D / 2D pose transformation process; Human body structure adaptation process; video generation process.

[0464] Therefore, motion deviations in the generated video may be caused by different processing stages.

[0465] For example, when the human body movements in the generated video are inconsistent with the overall target movements, it may be because the 3D motion data selected in the motion data matching stage does not meet the target requirements; when the motion content is correct but some motion areas cannot be clearly expressed, it may be because the virtual camera parameters fail to meet the requirements for motion information preservation; when there are jumps in the motion trajectory or abnormal postures, it may be because there is information loss during the conversion from 3D motion to 2D posture; when the proportions of the limbs are abnormal when a person performs an action, it may be because the human body structure adaptation parameters are unreasonable.

[0466] Therefore, this embodiment determines the closed-loop objects that need to be optimized based on the generated result evaluation information.

[0467] S305.1, Action Data Matching Closed-Loop Optimization Object

[0468] When motion space deviation is detected in the generated video, the 3D motion database retrieval process is used as the first closed-loop optimization object.

[0469] Specifically, based on the difference between the motion trajectory of key joints of the human body in the generated video and the target motion reference information, it is determined whether the current target three-dimensional motion data meets the motion retention requirements.

[0470] For example, if the generated video shows a hand movement range that is less than the target action requirement; or the trend of the body's center of gravity changes is inconsistent with the target action; or there is a deviation in the sequence of human joint movements, then the current action data matching result is determined to be insufficient.

[0471] Furthermore, the matching parameters in the action retrieval process are adjusted based on the action error evaluation results.

[0472] Specifically, this includes: increasing the matching weight corresponding to the spatial trajectory consistency index; and increasing the matching weight corresponding to the joint continuity index. Increase the matching weight corresponding to the consistency index of human body center of gravity changes.

[0473] The target motion data is then re-filtered from the 3D motion database based on the adjusted matching parameters.

[0474] This method allows the motion data matching process to be optimized based on the actual generated results, thereby improving the accuracy of subsequent motion selection.

[0475] S305.2, Viewpoint Preservation Optimization Closed-Loop Object

[0476] When a problem is detected that the key motion regions are not adequately represented in the generated video, the virtual camera parameter optimization process is used as the second closed-loop optimization object.

[0477] Specifically, based on the action performance results in the generated video, we analyze whether there is occlusion in the key action areas; missing motion trajectories; and indistinguishable spatial relationships.

[0478] For example, if hand movements cannot be fully displayed in the generated video, while other areas of the human body are normal, it indicates that the current virtual camera's viewing angle may cause a loss of hand movement information.

[0479] Furthermore, the virtual camera parameters are adjusted based on the evaluation results and motion information.

[0480] Specific adjustments include adjusting the virtual camera's spatial position; adjusting the virtual camera's viewing direction; adjusting the virtual camera's height; and adjusting the virtual camera's field of view. After adjustments, the action information is re-executed and the evaluation is maintained.

[0481] When the visibility index of key action areas, the integrity index of action trajectory, and the index of maintaining human spatial relationships meet the preset requirements, new virtual camera parameters are determined.

[0482] This method enables the perspective optimization process to be formed: The process includes: result detection; motion expression evaluation; camera parameter adjustment; and closed-loop optimization of the regenerated result.

[0483] S305.3, Three-dimensional and two-dimensional attitude transformation closed-loop optimization object

[0484] When abnormal motion trajectories, posture jumps, or joint connection abnormalities are detected in the generated video, the three-dimensional and two-dimensional posture transformation process is used as the third closed-loop optimization object.

[0485] Specifically, the human posture sequence obtained from the generated video detection is compared with the motion patterns corresponding to the target's three-dimensional motion data.

[0486] When it is discovered: Two-dimensional posture changes cannot reflect three-dimensional motion trends; the movement direction of key joints is abnormal; the connection relationship of the human skeleton is offset; then it is determined that there is an error in the posture transformation process.

[0487] Furthermore, based on the detection results, the following adjustments are made: the 2D posture consistency judgment threshold; abnormal posture detection parameters; and posture correction weights. The 3D motion projection and posture verification process is then re-executed.

[0488] This method can reduce the loss of motion information caused by the projection process, making the posture control information of the input video generated model more consistent with the original three-dimensional motion laws.

[0489] S305.4, Human Body Structure Adaptation Closed-Loop Optimization Object

[0490] When an abnormal human proportion or incoordination between movement and human structure is detected in the generated video, the human body structure adaptation process is used as the fourth closed-loop optimization object.

[0491] Specifically, the human body contours, limb proportions, and joint movement relationships in the generated video are detected.

[0492] For example, if the target person's arm length is abnormal in the generated video; or the leg proportions do not match the person's appearance; or the range of motion exceeds the reasonable range of the person's body structure, then it is determined that there is a deviation in the human body structure mapping parameters.

[0493] Furthermore, based on the test results, the following parameters were readjusted: human proportion parameters; skeletal scale mapping parameters; and local limb adjustment parameters.

[0494] Then the character structure adaptation process is re-executed.

[0495] The above methods enable the same action to be adapted more naturally to different characters.

[0496] S305.5 Establish a multi-object collaborative feedback closed loop

[0497] In a preferred embodiment, this embodiment can also determine multiple closed-loop optimization objects simultaneously based on multiple error evaluation results.

[0498] Specifically, error correlations are established based on the quality evaluation results of the generated videos.

[0499] For example, when both abnormal motion trajectories and occlusion issues in critical areas are detected simultaneously, adjustments can be made at the same time: Action consistency judgment parameters; virtual camera parameters.

[0500] When both abnormal character proportions and deviations in movement are detected simultaneously, adjustments can be made at the same time: Human body structure mapping parameters; motion scale adaptation parameters.

[0501] Through multi-object collaborative optimization, the system is able to make comprehensive adjustments to address complex generation errors.

[0502] S306. Adjust the motion holding constraint parameters based on the feedback results.

[0503] In this embodiment, the corresponding processing steps are optimized based on the determined generation error type.

[0504] Specifically, it includes: (1) Optimization of motion matching feedback When an action matching error is detected, the action retrieval parameters in the 3D action database are adjusted.

[0505] For example, increase the matching weight related to the motion trajectory of key joints; increase the evaluation weight related to the motion space structure; Reduce the weight of action data that are semantically similar but do not match the motion patterns.

[0506] By employing the methods described above, the subsequent motion retrieval process can better focus on the actual movement patterns of the human body.

[0507] (2) Optimization of viewpoint parameter feedback

[0508] When a perspective error is detected, the virtual camera parameters are adjusted.

[0509] For example, when the generated video does not adequately represent the hand movement area, the visibility evaluation weight of the corresponding hand area is increased, and the virtual camera position is readjusted. When insufficient display of the overall human body structure is detected, the virtual camera distance or field of view is adjusted.

[0510] Through feedback optimization, the subsequently generated videos can more completely express the target action.

[0511] (3) Attitude control feedback optimization

[0512] When an attitude transition error is detected, adjust the 3D-2D attitude consistency verification parameters.

[0513] For example, improving the sensitivity of abnormal posture detection; adjusting the motion constraint range of key joints; and increasing posture correction weights can reduce information loss during posture transition.

[0514] (4) Character structure feedback optimization

[0515] When a human body structure adaptation error is detected, the human body structure mapping parameters are adjusted.

[0516] For example, recalculate the target person's body proportions; optimize the mapping relationship of motion scales for different body regions. This enables the target person to perform the target actions more naturally.

[0517] S307, Regenerate video based on optimized parameters

[0518] After adjusting the feedback parameters, the video generation process is re-executed using the updated action to maintain the constraint parameters.

[0519] Specifically, the target's 3D motion data is redefined based on the optimized motion retrieval parameters; the posture control sequence is regenerated based on the optimized viewpoint parameters; and the character's motion mapping relationship is adjusted based on the optimized structure adaptation parameters. Then, the updated control information is input into the video generation model to generate the optimized video result.

[0520] Furthermore, the process of video generation, motion consistency detection, parameter feedback adjustment, and video regeneration can be repeated until the generated result meets the preset quality requirements.

[0521] S308, Update motion hold constraint parameters

[0522] In a preferred embodiment, this embodiment can also update the motion parameters in the three-dimensional motion database over a long period of time based on the feedback results from multiple generation processes.

[0523] Specifically, based on statistics of historical generation results: Success rate of generating different motion data; motion retention effect under different viewpoints; adaptation effect corresponding to different human body structures.

[0524] Then, update based on the statistical results: Action retrieval weight; action consistency evaluation parameters; perspective optimization parameters.

[0525] In this way, the system can continuously optimize its motion retention capabilities as more tasks are generated.

[0526] Through the above methods, this embodiment establishes a closed-loop optimization mechanism for video generation based on feedback of generation results.

[0527] Compared with existing video generation methods, this embodiment does not directly output the result after completing a model inference. Instead, it further detects the motion performance in the video after generation and adjusts the motion selection, viewpoint control, posture transformation and human body structure adaptation parameters in reverse based on the detection results.

[0528] Specifically, the consistency evaluation mechanism for generated actions can detect action deviations that occur during the video generation process. Through the error classification mechanism, corresponding optimizations can be performed for different types of problems; through the feedback adjustment mechanism, the effect of maintaining constraints on actions can be continuously improved.

[0529] Therefore, this embodiment can further improve the ability to preserve motion information during video generation, thereby enhancing the accuracy of character movements, the completeness of perspective expression, and the consistency of character structure in the generated video.

[0530] In this embodiment, the motion information retention constraint is not an independent constraint process set for a single processing stage, but a continuous control mechanism that runs through the entire video generation process. By establishing corresponding constraint relationships in multiple stages such as motion data acquisition, perspective transformation, posture generation, character fusion, and generation feedback, human motion information can be consistently transmitted between different processing stages.

[0531] Specifically, in the motion data acquisition stage, the present invention performs consistency judgment on candidate three-dimensional motion data based on human motion constraints. By analyzing the spatial motion trajectory of key human joints, the continuity of joint motion, and the overall motion state of the human body, the candidate motion data is screened so that the three-dimensional motion data input into the video generation process not only meets the semantic requirements of motion, but also conforms to the real motion law of the human body.

[0532] During the perspective transition phase, this invention does not directly project the 3D motion data based on the user's input observation angle. Instead, it evaluates the visibility of key motion areas, the integrity of motion trajectories, and the preservation of human spatial relationships under different observation angles based on the motion information preservation evaluation mechanism. The virtual camera parameters are then adjusted according to the evaluation results to reduce the loss of motion information caused by changes in observation angle.

[0533] During the attitude transition phase, this invention uses the spatial motion relationships contained in the three-dimensional motion data as a constraint basis to perform consistency verification on the two-dimensional attitude control sequence generated by the three-dimensional motion projection. By comparing the consistency between the three-dimensional motion change trend and the two-dimensional attitude change trend, abnormal attitude data is corrected so that the two-dimensional attitude control information can maintain the motion characteristics of the original three-dimensional motion.

[0534] In the character fusion stage, this invention adapts the posture control sequence to human structure based on the human proportion parameters, body contour parameters and human structural features of the target character, so that the standardized motion data can match the body structure of different characters and avoid abnormal motion performance due to differences in human structure.

[0535] During the feedback generation stage, the present invention further detects the motion performance results in the generated video, determines the corresponding optimization object based on the detected motion error type, and makes feedback adjustments to the motion data matching parameters, virtual camera parameters, posture transformation parameters, and human body structure adaptation parameters, so that the video generation process can continuously optimize the motion retention capability based on the actual generation effect.

[0536] Through the aforementioned multi-stage motion information retention constraint mechanism, this invention establishes a complete control chain from 3D motion data acquisition, motion selection, perspective control, posture transformation to result feedback optimization, enabling motion information to be stably transmitted from the original 3D motion data to the final generated video, thereby improving the spatial structure retention, motion continuity, and multi-view expression consistency of the character's movements in the generated video.

[0537] Example 4: AIGC Video Generation System Based on Motion Capture Data

[0538] In this embodiment, an AIGC video generation system based on motion capture data is provided to implement the video generation methods described in Embodiments 1 to 3 above.

[0539] like Figure 1 As shown, the video generation system includes a 3D motion database module 100, a motion retrieval module 200, a motion consistency judgment module 300, a viewpoint adaptation optimization module 400, a posture conversion and consistency verification module 500, a character structure adaptation module 600, and a video generation module 700.

[0540] Furthermore, in a preferred embodiment, the video generation system further includes a feedback optimization module 800, which is used to detect motion performance information in the generated video, determine motion retention error based on the detection results, and further adjust motion matching parameters, viewpoint control parameters, and posture control parameters.

[0541] The three-dimensional motion database module 100 is used to store three-dimensional motion data formed by standardizing motion capture data.

[0542] Specifically, the three-dimensional motion data includes a three-dimensional spatial position sequence corresponding to the joints of the human skeleton, a joint rotation parameter sequence, motion semantic information, and motion feature information related to the motion execution process.

[0543] In this embodiment, the motion data in the 3D motion database module 100 is formed by standardizing motion capture data. The motion capture data can be obtained through optical motion capture devices, inertial motion capture devices, human 3D pose estimation devices, or other 3D motion acquisition methods, and undergoes coordinate normalization, human skeleton model unification, and time series standardization processing, so that motion data from different sources can be stored and retrieved in a unified data format.

[0544] By setting up the 3D motion database module 100, the system can use 3D motion data formed by motion capture data and containing spatial motion information as the basis for video generation control.

[0545] The action retrieval module 200 is connected to the three-dimensional action database module 100 and is used to obtain candidate three-dimensional action data from the three-dimensional action database module 100 based on the target action requirement information input by the user.

[0546] Specifically, the action retrieval module 200 can match action data in the database based on the action semantic features, static posture features, and dynamic motion features corresponding to the target action requirement, thereby obtaining a set of candidate actions related to the user's needs.

[0547] Furthermore, the action retrieval module 200 sends the acquired candidate 3D action data to the action consistency judgment module 300 to further determine whether the action data meets the action retention requirements.

[0548] The motion consistency judgment module 300 is used to perform human motion constraint analysis on candidate three-dimensional motion data and determine the target three-dimensional motion data that meets the motion retention conditions.

[0549] Specifically, the motion consistency judgment module 300 evaluates the consistency of candidate motions based on the spatial motion trajectory of key joints of the human body, the continuity of joint motion, and the overall motion state of the human body in the candidate three-dimensional motion data.

[0550] When the candidate 3D motion data meets the preset motion retention conditions, the motion consistency judgment module 300 outputs the corresponding target 3D motion data; when the candidate motion data does not meet the motion retention conditions, it returns to the motion retrieval module 200 to reacquire the candidate motion data.

[0551] By setting up the action consistency judgment module 300, the system can avoid selecting action data solely based on action semantics, thereby improving the matching degree between action control information and user target actions.

[0552] The viewpoint adaptation optimization module 400 is used to optimize and adjust the virtual camera parameters according to the target observation viewpoint information and the requirements for maintaining motion information.

[0553] Specifically, the viewpoint adaptation and optimization module 400 acquires the target observation viewpoint input by the user and evaluates the current virtual camera parameters based on the distribution of key action areas in the target's 3D motion data.

[0554] The evaluation content includes: Visibility of key movement areas; completeness of movement trajectories; degree of preservation of human spatial relationships.

[0555] When the current viewing angle is detected to potentially obscure key action areas, cause missing action trajectories, or result in insufficient representation of human spatial structure, the viewing angle adaptation and optimization module 400 adjusts the position parameters, orientation parameters, and / or field of view parameters of the virtual camera to obtain target virtual camera parameters that meet the requirements for maintaining action information.

[0556] By setting the perspective adaptation optimization module 400, the system can generate videos from different viewing angles while keeping the action content unchanged.

[0557] The attitude conversion and consistency verification module 500 is used to convert the target three-dimensional motion data into a two-dimensional attitude control sequence and to perform consistency verification on the converted attitude information.

[0558] Specifically, the posture conversion and consistency verification module 500 converts the three-dimensional spatial position of the human joints into two-dimensional key point information under the corresponding observation viewpoint based on the target virtual camera parameters.

[0559] Furthermore, this module utilizes the spatial motion patterns in the original three-dimensional motion data to verify the two-dimensional attitude control sequence.

[0560] When the motion change trend in the two-dimensional attitude control sequence is found to be inconsistent with the corresponding motion trend in the three-dimensional motion data, the abnormal attitude data is corrected.

[0561] By setting up the posture transformation and consistency verification module 500, the information loss generated during the 3D motion transformation process can be reduced, and the accuracy of the posture control information of the input video generation model can be improved.

[0562] The character structure adaptation module 600 is used to adapt the posture control sequence to human body structure based on the appearance characteristics of the target character.

[0563] Specifically, the character structure adaptation module 600 acquires the target character's human proportion parameters, body contour parameters, and human structure features, and establishes a mapping relationship between the target character's structure and the standard motion skeleton model.

[0564] Furthermore, the scale of the human skeleton in the posture control sequence is adjusted according to the mapping relationship of human body structure, so that the target action can be adapted to different human objects.

[0565] By setting up the character structure adaptation module 600, the problem of motion distortion caused by differences in human body shape can be reduced, thereby improving the naturalness of character movements in the generated video.

[0566] The video generation module 700 is used to generate a target video based on the control information optimized by motion constraints.

[0567] Specifically, the video generation module 700 receives character posture control information from the posture transformation and consistency verification module 500 and character appearance control information from the character structure adaptation module 600, and inputs the above information into the video generation model.

[0568] The video generation model generates and processes consecutive video frames based on action posture information, character appearance information, and viewing angle information, and outputs a video of the target character performing the target action.

[0569] The video generation model can be a diffusion-based video generation model, a temporal Transformer-based video generation model, or other model structures that can generate video content based on control information.

[0570] In a preferred embodiment, the video generation system further includes a feedback optimization module 800.

[0571] The feedback optimization module 800 is used to detect the generated video results and optimize the relevant control parameters based on the detection results.

[0572] Specifically, the feedback optimization module 800 can extract human posture information from the generated video and analyze the degree of consistency between the generated video and the target action.

[0573] When motion deviations are detected in the generated video, the corresponding optimization target is determined based on the error type.

[0574] For example, when motion space deviation is detected, the motion retrieval parameters are adjusted; when insufficient viewpoint expression is detected, the virtual camera parameters are adjusted; when pose conversion error is detected, the pose verification parameters are adjusted; and when abnormal character structure adaptation is detected, the human body structure mapping parameters are adjusted.

[0575] Through the above feedback optimization process, the video generation system forms a closed-loop optimization mechanism that includes result detection, error location, parameter adjustment, and regeneration.

[0576] With the above structure, the video generation system provided in this embodiment can use three-dimensional motion data as the basis for motion control, and through the collaborative work of multiple modules such as motion consistency judgment, viewpoint adaptation optimization, posture consistency verification, and character structure adaptation, the motion information is stably transmitted during the video generation process.

[0577] Compared to existing methods that only utilize two-dimensional motion references or directly use three-dimensional motion data to drive video generation, this embodiment improves the accuracy, continuity, and multi-view expression capabilities of character movements in the generated video by establishing motion information preservation constraint links.

[0578] It should be noted that the functional modules in this embodiment can be implemented by software, hardware or a combination of software and hardware. The data transmission relationship and functional division between the modules are only used to illustrate the technical solution of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0579] In summary, this invention provides an AIGC video generation method and system based on motion capture data. Its core lies in establishing a motion information persistence link that runs through motion data selection, viewpoint adaptation, posture control information conversion, character structure adaptation, and generation result feedback, so that the three-dimensional motion information formed by motion capture data can be consistently transmitted in the AIGC video generation link.

[0580] Compared to existing technologies that rely on 2D motion references or directly utilize 3D motion data to drive video generation, this invention focuses not only on the motion data input itself but also on the stability of 3D motion information transmission throughout the entire video generation process. Specifically, in the motion data acquisition stage, this invention determines target 3D motion data that meets motion retention conditions through human motion constraints. In the viewpoint transition stage, it optimizes virtual camera parameters based on the visibility of key motion areas, the integrity of motion trajectories, and the requirements for maintaining human spatial relationships. In the pose transition stage, it uses 3D motion laws to perform consistency checks and anomaly corrections on the 2D pose control sequence. Finally, in the character fusion and generation feedback stage, it further adjusts the human structure adaptation relationship and motion retention errors.

[0581] Therefore, this invention can reduce the information loss of three-dimensional motion information during AIGC video generation caused by motion matching, perspective changes, posture projection transformation and differences in human structure, so that the human motion in the generated video remains consistent in terms of spatial structure, motion continuity and multi-view expression. It can be widely used in film and television content production, virtual digital human generation, game animation production, sports action display and other video generation scenarios that require high-precision human motion control.

[0582] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, various equivalent substitutions, simple modifications, or improvements can be made without departing from the technical concept of the present invention, and all of these should be included within the scope of protection of the present invention.

[0583] The scope of protection of this invention shall be determined by the claims. The specific embodiments, examples and drawings in the specification are only used to explain the technical solutions of this invention and should not be construed as limiting the scope of protection of this invention.

Claims

1. A method for generating AIGC videos based on motion capture data, characterized in that, include: The system acquires the target motion requirement information input by the user, and obtains candidate 3D motion data from a 3D motion database pre-constructed based on motion capture data based on the target motion requirement information. The 3D motion database stores 3D motion data formed by standardizing motion capture data, and the 3D motion data includes a 3D spatial position sequence corresponding to multiple joints of the human body, a joint rotation parameter sequence, and motion semantic information. Based on the relationship between human motion constraints and action semantic matching, the candidate 3D action data is judged for action consistency to determine the target 3D action data that meets the action retention conditions. Acquire target viewing angle information, and based on the key action area and action information retention requirements corresponding to the target's three-dimensional action data, adjust the initial parameters of the virtual camera determined by the target viewing angle information to obtain the target virtual camera parameters; Based on the target virtual camera parameters, the target 3D motion data is converted into a 2D attitude control sequence, and based on the spatial motion relationship in the target 3D motion data, the 2D attitude control sequence is subjected to consistency verification and anomaly correction. The appearance features of the target person are obtained, the human body structure parameters of the target person are determined based on the appearance features, and the two-dimensional posture control sequence is adapted to the human body structure based on the human body structure parameters. The two-dimensional posture control sequence adapted to human structure is fused with the appearance features of the target person to obtain video generation control information, and the video generation control information is input into the video generation model to generate the target video. Among them, the motion consistency judgment, virtual camera parameter adjustment, two-dimensional posture control sequence consistency verification, and human body structure adaptation together constitute the motion information maintenance process for ensuring the consistent transmission of three-dimensional motion information in the AIGC video generation link.

2. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The construction of the pre-built 3D motion database based on motion capture data includes: Collect human motion capture data; The collected human motion capture data is processed by coordinate normalization, human skeleton model unification, and motion time scale standardization to obtain standardized three-dimensional motion data. Based on the standardized 3D motion data, action semantic labels, static posture features, and dynamic motion features are established, and the action semantic labels, static posture features, and dynamic motion features are associated with the corresponding 3D motion data and stored in the 3D motion database.

3. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The step of obtaining candidate 3D motion data from a pre-built 3D motion database based on motion capture data includes: Generate action retrieval features based on the target action requirement information; A first-stage retrieval is performed based on the action retrieval features and the action semantic information in the three-dimensional action database to obtain a candidate action set. Based on the static spatial distribution characteristics and dynamic motion trajectory characteristics of human joints, a second-stage matching is performed on the candidate action set. Based on the matching results of the second stage, candidate 3D action data is output from the candidate action set.

4. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The process of determining target 3D motion data that meets the motion retention conditions by performing motion consistency judgment on the candidate 3D motion data based on human motion constraints and action semantic matching relationships includes: Obtain the spatial motion trajectory of key human joints in the candidate 3D motion data; The motion continuity corresponding to the candidate 3D motion data is determined based on the changes in the position, direction of motion, and rotation of the key joints between adjacent motion frames. Based on the motion continuity and the matching relationship between the action semantic information and the target action requirement information, it is determined whether the candidate 3D action data meets the action retention condition; When the candidate 3D motion data meets the motion retention condition, the candidate 3D motion data is determined as the target 3D motion data.

5. The AIGC video generation method based on motion capture data as described in claim 4, characterized in that, The action consistency judgment also includes: Based on the spatial positional relationship of human joints in the candidate 3D motion data, the trajectory of the change of the human body's center of gravity is obtained; The body's balance during the execution of the action is determined based on the trajectory of the change in the human body's center of gravity. When the trajectory of the change of the human body's center of gravity does not match the preset motion state corresponding to the target motion type, the matching weight of the corresponding candidate three-dimensional motion data is reduced.

6. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The key motion area is a human body area used to express the characteristics of the target motion during the execution of the target motion. The key motion area includes at least one of the hand area, foot area, trunk rotation area and target joint local area. The target joint local area is a local motion area formed by at least one target joint determined by the target movement requirements and the human skeletal segment connected to the target joint. The target joint includes at least one of the shoulder joint, elbow joint, wrist joint, hip joint, knee joint, ankle joint and neck joint. The adjustment of the initial parameters of the virtual camera determined by the target observation perspective information to obtain the target virtual camera parameters includes: Based on the initial parameters of the virtual camera, the three-dimensional motion data of the target is projected and analyzed to obtain the projection state of the key motion area under the target's viewing angle. Based on the degree of projection overlap, spatial separation, observability, and motion trajectory continuity of the key action areas, determine whether the initial parameters of the virtual camera meet the requirements for maintaining motion information. When the initial parameters of the virtual camera do not meet the requirements for maintaining motion information, at least one of the virtual camera's position, orientation, and field of view parameters is adjusted to obtain the target virtual camera parameters.

7. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The process of performing consistency verification and anomaly correction on the two-dimensional attitude control sequence based on the spatial motion relationships in the target's three-dimensional motion data includes: Calculate the three-dimensional motion change trend of human joints based on the target three-dimensional motion data; Based on the projection relationship corresponding to the target virtual camera parameters, calculate the two-dimensional motion change trend of the corresponding two-dimensional key points in the two-dimensional attitude control sequence; Compare the three-dimensional motion change trend with the two-dimensional motion change trend; When the difference between the three-dimensional motion change trend and the two-dimensional motion change trend exceeds a preset range, the corresponding two-dimensional posture data is determined as abnormal posture data, and the abnormal posture data is corrected according to the spatial motion relationship in the target three-dimensional motion data.

8. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, The step of determining the human body structure parameters of the target person based on their appearance features, and then adapting the two-dimensional posture control sequence to the human body structure based on these parameters, includes: Extract human proportion parameters, body contour parameters, and size parameters of key human body parts from images or video frames of the target person; Based on the human body proportion parameters, body contour parameters, and key human body part size parameters, a mapping relationship is established between the target human body structure and the standard motion skeletal model. Based on the mapping relationship, the human skeleton scale and key point position relationship in the two-dimensional posture control sequence are adjusted so that the appearance structure of the target person matches the target action posture.

9. The AIGC video generation method based on motion capture data as described in claim 1, characterized in that, It also includes a result feedback optimization step, which includes: Human pose detection is performed on the generated target video to obtain the corresponding action performance information of the generated video; The motion retention error is determined based on the difference between the motion performance information and the motion reference information corresponding to the target three-dimensional motion data; Adjust at least one of the following parameters based on the motion retention error: 3D motion data retrieval weight, motion consistency judgment parameter, virtual camera parameter, posture consistency verification parameter, and human body structure adaptation parameter.

10. An AIGC video generation system based on motion capture data, characterized in that, include: The 3D motion database module is used to store 3D motion data formed by standardized processing of motion capture data. The 3D motion data includes 3D spatial position sequences corresponding to multiple joints of the human body, joint rotation parameter sequences, and motion semantic information. The action retrieval module is connected to the three-dimensional action database module and is used to retrieve candidate three-dimensional action data from the three-dimensional action database module based on the target action requirement information input by the user. The action consistency judgment module, connected to the action retrieval module, is used to judge the action consistency of the candidate 3D action data based on human motion constraints and action semantic matching relationship, and output the target 3D action data that meets the action retention conditions. The viewpoint adaptation optimization module, connected to the action consistency judgment module, is used to adjust the virtual camera parameters based on the target observation viewpoint information, the key action area corresponding to the target's three-dimensional action data, and the action information preservation requirements, to obtain the target virtual camera parameters. The attitude conversion and consistency verification module is connected to the viewpoint adaptation and optimization module. It is used to convert the target three-dimensional motion data into a two-dimensional attitude control sequence according to the target virtual camera parameters, and to perform consistency verification and anomaly correction on the two-dimensional attitude control sequence based on the spatial motion relationship in the target three-dimensional motion data. The character structure adaptation module is connected to the posture conversion and consistency verification module. It is used to determine the human body structure parameters of the target character based on the appearance features of the target character, and to perform human body structure adaptation on the two-dimensional posture control sequence based on the human body structure parameters. The video generation module, connected to the human body structure adaptation module, is used to fuse the two-dimensional posture control sequence after human body structure adaptation with the appearance features of the target human body to obtain video generation control information, and generate the target video based on the video generation control information.

Images