Digital human hand-object collaborative video generation method based on manifold projection

Abstract

The invention discloses a digital human hand-object collaborative video generation method based on manifold projection, which belongs to the field of digital human generation, and comprises the following steps: S1, multi-modal feature extraction; s2, interactive gesture modeling based on semantic manifold embedding; s3, generating an interaction track based on geometric manifold embedding; and S4, interactive video dynamic generation based on time sequence manifold projection: based on a time sequence manifold space, taking semantic-driven gesture representation and track representation as input, extracting time sequence condition features, performing condition generation based on a diffusion model, and generating a target interactive video frame by frame. By adopting the manifold projection-based digital human hand-object collaborative video generation method, modeling is performed based on manifold projection in three dimensions of semantics, geometry and time sequence, so that the naturalness, accuracy and reality sense of digital human hand-object collaborative video generation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human generation, and in particular to a method for generating digital human hand-object collaborative videos based on manifold projection. Background Art

[0002] In the current era of booming digital human technology, digital humans have been widely used in many fields, including virtual social interaction, education and training, and intelligent customer service. However, the technology for generating digital human hand-object interaction videos still has the following drawbacks:

[0003] 1. In terms of semantic consistency, traditional methods lack effective modeling of the complex relationships between speech semantics, object functional characteristics, and object geometric features, making it impossible to accurately map semantics to actions. This makes it difficult for existing interactive video generation technologies to accurately translate speech semantics into corresponding gestures and interactive behaviors. For example, in a virtual teaching scenario, when a teacher's digital human explains how to use an object, their hand movements may not match the semantic content being described, making it difficult for students to understand.

[0004] 2. From a physical rationality perspective, existing technologies fail to fully consider factors such as the surface characteristics and contact point distribution when modeling the geometric relationship between hand movements and objects. This results in a lack of physical constraints on the generated interaction trajectories, and the resulting interaction movements often fail to conform to the actual geometric characteristics of the objects. For example, in some virtual display scenes, the digital human's hand may directly pass through the object to grasp it, or the grasping movement may not match the object's shape or size. This not only undermines the authenticity of the interaction but also reduces the user's trust in the virtual scene.

[0005] 3. Regarding dynamic coherence, traditional video generation technology cannot effectively capture the dynamic dependencies between frames when processing time series features, making it difficult to ensure the natural and smooth flow of the video over time. As a result, the generated interactive videos often exhibit discontinuous and uneven hand movements and trajectories over time. For example, in a virtual livestream selling product, the digital human's hand movements may become jerky and jumpy while displaying the product, distracting the viewer and affecting the effectiveness of the message.

[0006] Furthermore, current technology struggles to generate personalized interactive actions. Different users have varying expectations and preferences for digital humans' interactions, but existing methods typically employ fixed models and parameters, failing to generate unique gestures and interactive behaviors tailored to individual user needs. This limits the application of digital humans in personalized services. Summary of the Invention

[0007] The purpose of the present invention is to provide a digital human hand-object collaborative video generation method based on manifold projection to solve the above technical problems.

[0008] To achieve the above objectives, the present invention provides a method for generating digital human-hand-object collaborative videos based on manifold projection, comprising the following steps:

[0009] S1. Multimodal feature extraction: Obtain person videos and object videos, and use encoders to extract person semantic features, object semantic features, and object geometric depth features from the person videos and object videos;

[0010] S2. Interactive gesture modeling based on semantic manifold embedding: Based on the semantic manifold mapping function, the semantic features of people, objects, and geometric depth features of objects are projected into the semantic manifold space to generate semantically driven gesture representations;

[0011] S3. Interaction trajectory generation based on geometric manifold embedding: Based on the geometric manifold mapping function, semantically driven gesture representation, person semantic features, and object geometric depth features are projected into the geometric manifold space to generate trajectory representations that conform to geometric property constraints;

[0012] S4. Dynamic generation of interactive videos based on temporal manifold projection: Based on the temporal manifold space, semantically driven gesture representation and trajectory representation are used as input, temporal conditional features are extracted, and conditional generation is performed based on the diffusion model to generate the target interactive video frame by frame.

[0013] Preferably, in step S1, the character semantic feature E p By Character Video V p The speech-driven semantic clues and personalized style features extracted from the

[0014] E p =Encoder p (V p ) (1);

[0015] Where, Encoder p (V p ) indicates the character video V p Perform feature extraction operations;

[0016] Object semantic features By object video V o The object categories, functional information components, and object geometric depth features extracted from By object video V o The object size, shape, and material information extracted from the object are composed of:

[0017]

[0018] Where, Encoder o (V o ) represents the object video V o Perform feature extraction operations.

[0019] Preferably, the semantically driven gesture representation H generated in step S2 is expressed as follows:

[0020]

[0021] Where, Φ sem (·) represents the semantic manifold mapping function, MLP s (·) represents a multi-layer perceptron, and [·||·] represents a feature-level concatenation operation.

[0022] Preferably, in step S2, the generated semantically driven gesture representation H is optimized by the following loss function:

[0023]

[0024] Where, represents the loss function used to optimize the generated semantic-driven gesture representation H; H gt Semantically driven gesture representation of the target; λ sem Represents the weight factor; W ij represents the weight matrix element, N(i) represents the domain set of the i-th element, E i 、E j and E k Represent the semantic features extracted from the i-th element, j-th element, and k-th element respectively; H i and H j denote the i-th and j-th elements in the semantically driven gesture representation H, respectively.

[0025] Preferably, the trajectory expression T generated in step S3 that meets the geometric characteristic constraints is as follows:

[0026]

[0027] Where, Φ geo (·) represents the geometric manifold mapping function, MLP g (·) represents the multi-layer perceptron used for fusion, and [·||·] represents the feature-level concatenation operation.

[0028] Preferably, in step S3, the trajectory representation T that meets the geometric property constraints is optimized by the following loss function:

[0029]

[0030] Where, represents the loss function of optimizing the trajectory representation T that meets the geometric property constraints; T gt represents the trajectory representation of the target; geo and λ smooth Both represent weight factors; T i The trajectory that complies with the geometric property constraints represents the i-th element of T; M geo represents the geometric prototype structure or reference trajectory template, which is used to measure whether the current trajectory frame is consistent with the geometric prior; T (t+1) and T (t) The trajectories that meet the geometric property constraints represent the states of T at time t+1 and time t respectively.

[0031] Preferably, step S4 specifically includes the following steps:

[0032] S41. Take the semantically driven gesture representation H and the trajectory representation T that conforms to the geometric feature constraints as input, and use the temporal manifold projection encoder to extract the temporal conditional feature C time :

[0033] C time =Encoder time (H,T) (7);

[0034] Where, Encoder time (H, T) represents the feature extraction operation of the semantic-driven gesture representation H and the trajectory representation T that complies with geometric feature constraints;

[0035] S42, based on the time sequence condition feature C time As the input of the diffusion model, the target interaction video is generated frame by frame through step-by-step denoising:

[0036]

[0037] Where p θ (V interact |C time ) represents the characteristic C under given timing conditions time Under this condition, the diffusion model generates the target interactive video V interact probability; Indicates the video status at a given moment and the temporal conditional feature C time Next, the diffusion model generates the current moment video The probability of ; D represents the number of video frames.

[0038] Preferably, in step S42, the target interactive video is generated by optimizing the following loss function:

[0039]

[0040] Where, represents the loss function used to optimize the generated target interaction video; V interact represents the generated target interaction video; V gt Interaction video representing the target; time represents the weight factor; and Represent the states of the temporal condition features at time t and time t-1 respectively.

[0041] Therefore, the present invention adopts the above-mentioned digital human hand-object collaborative video generation method based on manifold projection, which has the following beneficial effects:

[0042] 1. A dedicated encoder extracts multimodal features and projects them into a semantic manifold space to generate semantically driven gesture representations. By optimizing the loss function, gestures are accurately matched with semantics, resolving semantic consistency issues and ensuring that digital human hand movements conform to speech semantics during interaction, thereby improving the accuracy and rationality of interaction.

[0043] 2. Projecting cross-modal correlation features onto the geometric manifold space generates trajectories that conform to the object's geometric constraints. Using a loss function, the trajectories are ensured to meet the target requirements, maintain a reasonable distribution on the geometric manifold, and exhibit temporal smoothness. This addresses the issues of physical rationality and dynamic coherence, ensuring that digital hand-object interaction actions conform to the object's actual geometric characteristics and are natural and smooth.

[0044] 3. The temporal manifold projection encoder is used to extract temporal conditional features, and the video is generated frame by frame based on the diffusion model. The generation result is optimized using a comprehensive loss function, so that the generated video is consistent with the target video in content, while maintaining smooth dynamic characteristics in the temporal dimension, further ensuring dynamic coherence.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of the method for generating digital hand-object collaborative video based on manifold projection according to the present invention;

[0047] Figure 2 This is a simulation example diagram of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar numbers throughout represent the same or similar elements or elements with the same or similar functions.

[0049] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0050] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] like Figure 1 and Figure 2 As shown, the digital human-hand-object collaborative video generation method based on manifold projection includes the following steps:

[0052] S1. Multimodal feature extraction: Obtain (with speech) person videos and object videos, and use an encoder (based on conditional variational autoencoder) to extract person semantic features, object semantic features, and object geometric depth features from the person videos and object videos;

[0053] In step S1, the character semantic feature E p By Character Video V p The speech-driven semantic clues and personalized style features extracted from the

[0054] E p =Encoder p (V p ) (1);

[0055] Where, Encoder p (V p ) indicates the character video V p Perform feature extraction operations;

[0056] Object semantic features By object video V o The object categories, functional information components, and object geometric depth features extracted from By object video V oThe object size, shape, and material information extracted from the object are composed of:

[0057]

[0058] Where, Encoder o (V o ) represents the object video V o Perform feature extraction operations.

[0059] The character semantic feature E p and object semantic features It can reflect the relationship between speech intention and object function, while the geometric depth features of objects It can describe the physical properties of objects, and thus generate semantically driven gestures after fusion.

[0060] S2. Interactive Gesture Modeling Based on Semantic Manifold Embedding: Based on the semantic manifold mapping function, the semantic features of people, objects, and geometric depth are projected into the semantic manifold space to generate semantically driven gesture representations. The goal is to generate grasping gesture representations that are consistent with semantic intent and can reflect personalized features. Specifically, through multimodal feature fusion, high-dimensional mapping modeling from speech semantics and object descriptions to gestures is achieved. Semantic manifold embedding is used to capture the complex nonlinear mapping relationship between semantic intent and gestures, ensuring that the generated gestures are highly natural, semantically consistent, and personalized.

[0061] The semantic-driven gesture representation H generated in step S2 is expressed as follows:

[0062]

[0063] Where, Φ sem (·) represents the semantic manifold mapping function, MLP s (·) represents a multi-layer perceptron, and [·||·] represents a feature-level concatenation operation.

[0064] The semantic manifold mapping function is designed to capture the nonlinear relationship between speech semantics, object functions, and object geometric properties, and to reflect the semantic association of multimodal data through manifold embedding. Action representations with similar semantic intentions will be close in the manifold space, and the geometric properties of the object will be used to influence the expression of gestures. This design can effectively solve the problem of semantic consistency and improve the naturalness of actions through personalized feature embedding. At the same time, the semantic manifold mapping combines the physical constraints of hand movements to ensure that the grasping action fits the function and shape of the object, and that actions such as waving reflect the character's style, making the generated results both semantically matched and natural and smooth.

[0065] In step S2, to ensure that the generated gesture representation can be clustered on the semantic manifold and has semantic consistency and naturalness, the generated semantically driven gesture representation H is optimized by the following loss function:

[0066]

[0067]

[0068] Where, represents the loss function used to optimize the generated semantic-driven gesture representation H; H gt Semantically driven gesture representation of the target; λ sem Represents the weight factor; W ij represents the weight matrix element, N(i) represents the domain set of the i-th element, E i 、E j and E k Represent the semantic features extracted from the i-th element, j-th element, and k-th element respectively; H i and H j denote the i-th and j-th elements in the semantically driven gesture representation H, respectively.

[0069] It should be noted that the first term in formula (4) The second item is used to constrain the generated gesture to be consistent with the target semantics. Manifold regularization is used to ensure that the distribution of semantic gesture representations remains smooth on the manifold.

[0070] S3. Interaction trajectory generation based on geometric manifold embedding: Interaction trajectory generation needs to solve the following two core scientific challenges: (1) Multimodal fusion of semantics and object geometry: Combine semantically driven gesture action representation with object geometric characteristics (such as three-dimensional shape, surface normal vector, contact point distribution, etc.) to generate dynamic trajectories that meet the interaction intention. This process requires joint modeling of object surface characteristics and hand movements to ensure that the trajectory generation results can fully reflect the geometric constraints of the object and the logical consistency of semantic intentions. (2) Dynamic rationality and smoothness: In the time dimension, the generated trajectory must meet physical rationality and dynamic smoothness to avoid abrupt and incoherent motion changes. For example, when grasping a cylindrical object, the hand trajectory needs to adjust around the object surface to achieve precise fit, while when pointing at the object, the trajectory needs to dynamically adjust the contact path and accurately locate the contact point.

[0071] Therefore, based on the geometric manifold mapping function, semantically driven gesture representations, person semantic features, and object geometric depth features are projected into the geometric manifold space to generate trajectory representations that conform to geometric property constraints. Interaction trajectory generation is a key step in modeling natural interactive actions. Its essence is the deep coupling between hand movements and object geometric properties. It not only directly affects the physical rationality of the generated actions, but also significantly determines the visual naturalness and interaction logic consistency of the generated results.

[0072] The trajectory representation T generated in step S3 that complies with geometric feature constraints is expressed as follows (based on semantically driven gesture representation, person semantic features, and object geometric features, cross-modal correlation features are extracted and projected into the geometric manifold space to describe the nonlinear mapping relationship between the hand motion path and the object geometric surface):

[0073]

[0074] Where, Φ geo (·) represents the geometric manifold mapping function, MLP g (·) represents the multi-layer perceptron used for fusion, and [·||·] represents the feature-level concatenation operation.

[0075] Another key goal of geometric manifold embedding is to ensure the dynamic consistency of the trajectory. Since hand movements are a continuous time series, trajectory generation must not only satisfy the geometric constraints of a single frame, but also exhibit smooth dynamic changes in the temporal dimension to avoid mutations or physical conflicts. For example, when the hand moves around the surface of an object, the trajectory should maintain a smooth transition in space while avoiding jumps or stagnation in time. Therefore, in step S3, the trajectory representation T that meets the geometric property constraints is optimized using the following loss function:

[0076]

[0077] Where, represents the loss function of optimizing the trajectory representation T that meets the geometric property constraints; T gt represents the trajectory representation of the target; geo and λ smooth Both represent weight factors; T i The trajectory that complies with the geometric property constraints represents the i-th element of T; M geo represents the geometric prototype structure or reference trajectory template, which is used to measure whether the current trajectory frame is consistent with the geometric prior; T (t+1) and T (t) The trajectories that meet the geometric property constraints represent the states of T at time t+1 and time t respectively.

[0078] It should be noted that in formula (6), the first term Constrain the generated trajectory to be consistent with the target trajectory; the second term (λ geo ∑ i dist(T i ,M geo )) Trajectories are aggregated onto the geometric manifold through manifold distance constraints; the third term Through temporal smoothing regularization, the dynamic consistency and continuity of the trajectory in the time dimension are ensured.

[0079] The generated interactive video needs to be optimized and consistent in the following aspects: (1) Time series smoothness: The dynamic changes of hand movements and trajectories in the time dimension must be continuous and smooth, avoiding sudden changes in movement, trajectory breaks, or movement jumps that violate physical laws, so as to ensure the naturalness and physical feasibility of the generated results. (2) Semantic content consistency: The overall dynamic changes of the interactive video must be highly consistent with the semantic intention of the input, ensuring that each frame of the generated action conforms to the semantic logic and clearly expresses the voice-driven interaction goals, avoiding semantic deviations or action logic errors. (3) Inter-frame dependency modeling: Gesture action generation must fully capture the dynamic dependency between frames, ensure the spatial continuity of the motion trajectory and the temporal coherence of the gesture shape, and avoid action incoordination or trajectory offset due to the lack of contextual information.

[0080] Therefore, S4, dynamic generation of interactive videos based on temporal manifold projection: Based on the temporal manifold space (the temporal manifold space can effectively capture the dependencies and dynamic characteristics between frames in the time series by modeling the nonlinear mapping relationship between time series features), the semantically driven gesture representation and trajectory representation are used as input, the temporal conditional features are extracted, and conditional generation is performed based on the diffusion model to generate the target interactive video frame by frame.

[0081] Step S4 specifically includes the following steps:

[0082] S41. Take the semantically driven gesture representation H and the trajectory representation T that conforms to the geometric feature constraints as input, and use the temporal manifold projection encoder to extract the temporal conditional feature C time (It represents the time series characteristics of gesture movements and trajectories in manifold space):

[0083] C time =Encoder time (H,T) (8);

[0084] Where, Encoder time (H, T) represents the feature extraction operation of the semantic-driven gesture representation H and the trajectory representation T that complies with geometric feature constraints;

[0085] S42, based on the time sequence condition feature Ctime As the input of the diffusion model, the target interaction video is generated frame by frame through step-by-step denoising:

[0086]

[0087] Where p θ (V interact |C time ) represents the characteristic C under given timing conditions time Under this condition, the diffusion model generates the target interactive video V interact probability; Indicates the video status at a given moment and the temporal conditional feature C time Next, the diffusion model generates the current moment video The probability of ; D represents the number of video frames.

[0088] Preferably, in step S42, the target interactive video is generated by optimizing the following loss function:

[0089]

[0090] Where, represents the loss function used to optimize the generated target interaction video; V interact represents the generated target interaction video; V gt Interaction video representing the target; time represents the weight factor; and Represent the states of the temporal condition features at time t and time t-1 respectively.

[0091] It should be noted that in formula (10), the first term Constrain the generated video to be consistent with the target video in content, the second By constraining the distribution of temporal conditional features through temporal regularization, the features are aggregated onto the temporal manifold, so that the generated interactive video maintains smooth dynamic characteristics in the temporal dimension.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating digital human-hand-object collaborative videos based on manifold projection, characterized by: The following steps are involved: S1. Multimodal feature extraction: Obtain person videos and object videos, and use encoders to extract person semantic features, object semantic features, and object geometric depth features from the person videos and object videos; S2. Interactive gesture modeling based on semantic manifold embedding: Based on the semantic manifold mapping function, the semantic features of people, objects, and geometric depth features of objects are projected into the semantic manifold space to generate semantically driven gesture representations; S3. Interaction trajectory generation based on geometric manifold embedding: Based on the geometric manifold mapping function, semantically driven gesture representation, person semantic features, and object geometric depth features are projected into the geometric manifold space to generate trajectory representations that conform to geometric property constraints; S4. Dynamic generation of interactive videos based on temporal manifold projection: Based on the temporal manifold space, semantically driven gesture representation and trajectory representation are used as input, temporal conditional features are extracted, and conditional generation is performed based on the diffusion model to generate the target interactive video frame by frame.

2. The method for generating digital hand-object collaborative videos based on manifold projection according to claim 1, characterized in that: In step S1, the character semantic feature E p By Character Video V p The speech-driven semantic clues and personalized style features extracted from the E p =Encoder p (V p ) (1); Where, Encoder p (V p ) indicates the character video V p Perform feature extraction operations; Object semantic features By object video V o The object categories, functional information components, and object geometric depth features extracted from By object video V o The object size, shape, and material information extracted from the object are composed of: Where, Encoder o (V o ) represents the object video V o Perform feature extraction operations.

3. The method for generating digital hand-object collaborative videos based on manifold projection according to claim 1, characterized in that: The semantic-driven gesture representation H generated in step S2 is expressed as follows: Where, Φ sem (·) represents the semantic manifold mapping function, MLP s (·) represents a multi-layer perceptron, and [·||·] represents a feature-level concatenation operation.

4. The method for generating digital hand-object collaborative videos based on manifold projection according to claim 3, characterized in that: In step S2, the generated semantically driven gesture representation H is optimized by the following loss function: Where, represents the loss function used to optimize the generated semantic-driven gesture representation H; H gt semantically driven gesture representations that represent goals; λ sem represents the weight factor; W ij represents the weight matrix element, N(i) represents the domain set of the i-th element, E i 、E j and E k Represent the semantic features extracted from the i-th element, j-th element, and k-th element respectively; H i and H j denote the i-th and j-th elements in the semantically driven gesture representation H, respectively.

5. The method for generating digital hand-object collaborative videos based on manifold projection according to claim 3, characterized in that: The trajectory expression T generated in step S3 that meets the geometric characteristic constraints is as follows: Where, Φ geo (·) represents the geometric manifold mapping function, MLP g (·) represents the multi-layer perceptron used for fusion, and [·||·] represents the feature-level concatenation operation.

6. The method for generating digital human-hand-object collaborative videos based on manifold projection according to claim 5, characterized in that: In step S3, the trajectory representation T that complies with the geometric property constraints is optimized by the following loss function: Where, represents the loss function of optimizing the trajectory representation T that meets the geometric property constraints; T gt Trajectory representation of the target; λ geo and λ smooth Both represent weight factors; T i The trajectory that complies with the geometric property constraints represents the i-th element of T; M geo Represents the geometric prototype structure or reference trajectory template, which is used to measure whether the current trajectory frame is consistent with the geometric prior; T (t+1) and T (t) The trajectories that meet the geometric property constraints represent the states of T at time t+1 and time t respectively.

7. The method for generating digital human-hand-object collaborative videos based on manifold projection according to claim 5, characterized in that: Step S4 specifically includes the following steps: S41. Take the semantically driven gesture representation H and the trajectory representation T that conforms to the geometric feature constraints as input, and use the temporal manifold projection encoder to extract the temporal conditional feature C time : C time =Encoder time (H,T) (8); Where, Encoder time (H, T) represents the feature extraction operation of the semantic-driven gesture representation H and the trajectory representation T that complies with geometric feature constraints; S42, based on the time sequence condition feature C time As the input of the diffusion model, the target interaction video is generated frame by frame through step-by-step denoising: Where p θ (V interact |C time ) represents the characteristic C under given timing conditions time Under this condition, the diffusion model generates the target interactive video V interact probability; Indicates the video status at a given moment and the temporal conditional feature C time Next, the diffusion model generates the current moment video The probability of ; D represents the number of video frames.

8. The method for generating digital human-hand-object collaborative videos based on manifold projection according to claim 7, characterized in that: In step S42, the generated target interactive video is optimized using the following loss function: Where, V represents the loss function used to optimize the generated target interaction video; interact represents the generated target interaction video; V gt Interaction video representing the target; time represents the weight factor; and Represent the states of the temporal condition features at time t and time t-1 respectively.

Images