Three-dimensional hand object interaction action sequence generation method and system based on hybrid expert and visual feedback

Abstract

The invention discloses a three-dimensional hand object interaction action sequence generation method and system based on mixed expert and visual feedback, belongs to the field of hand object interaction action generation, and provides an end-to-end generation architecture, so that end-to-end joint optimization can be realized in a model training stage; by introducing a hybrid expert mechanism and a visual feedback mechanism, the expert groups guided by layered semantics and a double-channel visual feedback mechanism are constructed, so that the different expert groups are specific to different hand-object interaction modes, and meanwhile, the expert network of each expert group is intelligently gated through the visual features of interaction images; accurate capture of fine-grained motion features is achieved, the problem that a general model is difficult to give consideration to global postures and local details is effectively solved, and the quality of generated actions is remarkably improved. Besides, real-time rendered interactive image feedback is adopted, so that the generative network is endowed with physical perception and self-correction capability, and the physical consistency and robustness of interactive actions are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of hand object interactive action generation technology, and more specifically, relates to a method and system for generating three-dimensional hand object interactive action sequences based on hybrid expert and visual feedback. Background Technology

[0002] 3D hand object interaction motion generation aims to generate natural, realistic, and physically consistent 3D hand movements based on text descriptions and object trajectories. Currently, several methods exist for generating hand object interaction motions, which can be broadly categorized into two types.

[0003] The first type is a static hand grasping action generation method based on multimodal information. This type of method is mainly aimed at robotic dexterous hand grasping operations, aiming to generate robust and diverse dexterous hand grasping actions based on multimodal inputs such as object point clouds and text. Current work in this area mainly relies on diffusion models, extracting features from object point clouds, text, and other multimodal information through appropriate feature encoders, and using these features as conditional inputs in the diffusion model to guide hand grasping action generation. In addition, some works also generate contact probability values ​​for the object point cloud, i.e., contact maps, to reflect the probability that a certain region of the object may interact with the hand. The generated contact maps provide rich hand-object interaction information to guide or optimize the final generated grasping actions.

[0004] The second approach is based on multimodal information to generate hand-object interaction action sequences. Compared to static hand grasping action generation tasks, hand-object interaction action sequence generation methods are geared towards dynamic scenes, requiring the generation of corresponding hand-operated object sequences based on multimodal information such as task text descriptions, object point clouds, and motion trajectories. Due to the complexity of hand-object interaction modeling and the dynamic nature of the sequence actions, generating high-quality hand-object interaction action sequences is very challenging. Current mainstream methods mainly employ a transformer-based diffusion model architecture; however, due to the complexity of hand-object interaction actions, generating high-quality hand-object interaction actions using only a diffusion model is extremely difficult. Therefore, current best methods mostly adopt a multi-stage paradigm, using an action refinement stage after the diffusion model generation stage to improve the quality of generated hand-object interaction actions. Although this approach has achieved good results, the two stages are separate, making end-to-end training and optimization impossible, thus hindering the model from finding the optimal solution. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for generating 3D hand object interactive action sequences based on hybrid expert and visual feedback, thereby solving the problems of existing single-stage models being unable to generate fine-grained interactive actions, lacking physical constraints, and multi-stage models being unable to be jointly optimized.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for generating a three-dimensional hand object interaction action sequence based on hybrid expert and visual feedback is provided, comprising: Step S1: Obtain task data for the hand-object interaction task, the task data including initial text description, object point cloud, object motion trajectory and action category label; refine the initial text description to generate refined text descriptions corresponding to the hand level, finger level and joint level respectively; Step S2: Based on the task data and the diffusion model time step, encode the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, and the time step embedding; Step S3: Input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into the diffusion-based generative network to obtain the coarse hand action sequence output by the generative network; the generative network contains expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate the hand action sequence; Step S4: Based on the rough hand movement sequence and the object point cloud, perform differentiable rendering to obtain an interactive image, extract the visual features of the interactive image, generate physical correction text corresponding to each level based on the interactive image, and extract the physical correction text features of the physical correction text. Step S5: Input the initial text features, the physical correction text features corresponding to each level of the physical correction text, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image back into the generator network to obtain the denoised hand action sequence output by the generator network; wherein, the physical correction text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

[0007] According to the above-described method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, the generation network includes multiple stacked generation sub-modules, each containing an expert group corresponding to each level. The expert group corresponding to any level includes a cross-attention layer for extracting hierarchical semantic features based on the hierarchical refined text features or physically corrected text features corresponding to the corresponding level, multiple expert networks for generating hand action sequences based on the hierarchical semantic features, the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory and the noisy hand action sequence, and a perceptual router for calculating the activation weights of each expert network based on the visual features, the action label features and the time step embedding.

[0008] According to the above-mentioned method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, the generation submodule also includes a self-attention layer; In step S3, the self-attention layer of the generation submodule performs a self-attention transformation based on the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the concatenation result of the noisy hand action sequence or the hand action sequence output by the previous generation submodule to obtain self-attention features; the cross-attention layer in each expert group of the generation submodule extracts hierarchical semantic features from the hierarchically refined text features corresponding to the corresponding level based on the cross-attention mechanism; the perceptual router in each expert group of the generation submodule calculates the activation weights of each expert network based on the action label features and the time step embedding; the expert networks in each expert group of the generation submodule generate and output hand action sequences based on the self-attention features and the hierarchical semantic features respectively; the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group in the generation submodule and the activation weights of the expert networks of each expert group; the hand action sequence output by the last generation submodule is the coarse hand action sequence. In step S5, the self-attention layer of the generation submodule performs a self-attention transformation based on the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the concatenation result of the noisy hand action sequence or the hand action sequence output by the previous generation submodule to obtain self-attention features; the cross-attention layer in each expert group of the generation submodule extracts hierarchical semantic features from the corresponding level of physically corrected text features based on the cross-attention mechanism; the perceptual router in each expert group of the generation submodule calculates the activation weights of each expert network based on the visual features, the action label features, and the time step embedding; the expert networks in each expert group of the generation submodule generate and output hand action sequences based on the self-attention features and the hierarchical semantic features respectively; the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group in the generation submodule and the activation weights of the expert networks of each expert group; and the hand action sequence output by the last generation submodule is the denoised hand action sequence.

[0009] According to the above-described method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, in steps S3 and S5, the hand action sequences output by the generation submodule are obtained by fusing them in the following manner: For each expert group, retain the K expert networks with the highest activation weights in that expert group, and reset the activation weights of the remaining expert networks to zero; The activation weights of the expert networks retained in all expert groups are renormalized to obtain the final activation weights of the corresponding expert networks. The hand movement sequences output by the corresponding expert networks are weighted and fused based on the final activation weights of the expert networks retained in all expert groups to obtain the hand movement sequences output by the generation submodule.

[0010] According to the above-described method for generating 3D hand-object interaction action sequences based on hybrid expert and visual feedback, the perception router is used to map the visual features, the action label features, and the time step embedding into visual spatial weight components, action weight components, and denoising step weight components, respectively. The visual spatial weight components, action weight components, and denoising step weight components are then added together and normalized to obtain the activation weights of each expert network in the corresponding expert group. Each of the visual spatial weight components, action weight components, and denoising step weight components contains multiple dimensions corresponding to each expert network. In step S3, the visual features are empty, and the visual spatial weight components are 0.

[0011] Based on the above-mentioned method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, the initial text description is refined to generate refined text descriptions corresponding to the hand level, finger level, and joint level, respectively, specifically including: Obtain a hierarchical prompt template containing hierarchical decomposition constraints, coarse-to-fine constraints, and interaction-specific constraints. Among them, the hierarchical decomposition constraints guide the large language model to refine the description into three levels: hand, finger, and joint; the coarse-to-fine constraints guide the large language model to transition from overall posture description to local detail description; and the interaction-specific constraints guide the large language model to describe the specific object components of the interaction and the coordination between the hands.

[0012] According to the above-mentioned method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, the loss functions used when training the generation network include diffusion reconstruction loss, geometric reconstruction loss, distance loss, contact loss, penetration loss, smoothing loss and expert balance loss. Wherein, the diffusion reconstruction loss L diff It is calculated based on the following formula:

[0013] in, This is a sequence of actual hand movements. It is a noisy hand motion sequence. The parameter is represented as The hand movement sequence output by the generative network. t For the diffusion model time step, c For task data, This is the expected value; The geometric reconstruction loss L rec It is calculated based on the following formula:

[0014] in, J and V These represent the actual hand joints and the hand mesh vertices, respectively. and The hand joints and hand mesh vertices are obtained by using the MANO model to generate the hand motion sequence output by the network. The distance loss L dist It is calculated based on the following formula:

[0015] in, Point cloud for objects, This represents the distance between two vertices; The contact loss L con It is calculated based on the following formula:

[0016] in, and These are the set of mutually contacting object points and the set of hand mesh vertices, determined based on a distance threshold, respectively. The penetration loss L pen It is calculated based on the following formula:

[0017] in, It is the set of object points that penetrate the grid of the hand; The smoothing loss L acc It is calculated based on the following formula:

[0018] Among them, v i v i-1 v i-2 These are three consecutive hand mesh vertices in the hand motion sequence output by the generating network; The expert balance loss L balance It is calculated based on the following formula:

[0019]

[0020] Among them, w a w t and w v These are the action weight components, the denoising step weight components, and the visual spatial weight components, respectively, where M is the number of expert networks in the generator network, and f i P represents the number of times the i-th expert network is retained during the current batch training process. i This represents the average activation weights of the i-th expert network calculated by the perceptual router during the current batch training process.

[0021] According to a second aspect of the present invention, a three-dimensional hand object interactive action sequence generation system based on hybrid expert and visual feedback is provided, comprising: The data acquisition unit is used to acquire task data for hand-object interaction tasks. The task data includes an initial text description, object point cloud, object motion trajectory, and action category label. The initial text description is refined to generate refined text descriptions corresponding to the hand level, finger level, and joint level, respectively. The encoding unit is used to encode initial text features, hierarchical refined text features corresponding to each level of refined text description, object geometric features, action label features, and time step embedding based on the task data and the diffusion model time step. An action sequence generation unit is used to input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into a diffusion-based generative network to obtain a coarse hand action sequence output by the generative network; the generative network includes expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate hand action sequences. Based on the rough hand motion sequence and the object point cloud, a differentiable rendering is performed to obtain an interactive image. The visual features of the interactive image are extracted. Based on the interactive image, physical correction text corresponding to each level is generated and the physical correction text features are extracted. The initial text features, the physically corrected text features corresponding to each level, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image are input again into the generator network to obtain the denoised hand action sequence output by the generator network. The physically corrected text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

[0022] According to a third aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0023] According to a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0024] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) A one-step end-to-end generation architecture is proposed, breaking through the limitations of the traditional multi-stage paradigm. This invention introduces a hybrid expert mechanism into the hand object interaction action generation task for the first time, constructing a one-step framework to replace the traditional "coarse generation-refinement" multi-stage method. This design realizes end-to-end joint optimization of the model, avoids the suboptimal solution problem caused by staged training, and significantly improves the model's ability to learn complex hand object interaction patterns through multi-expert collaboration.

[0025] (2) The integration of hierarchical experts and dynamic routing mechanisms significantly improves the precision and physical realism of motion. Based on the anatomical structure of the hand, this invention designs a hierarchical expert group of "hand-finger-joint" and combines it with a perceptual routing mechanism to achieve accurate capture of fine-grained motion features. This design effectively solves the problem that general models cannot take into account both global posture and local details, and significantly improves the quality of generated motion.

[0026] (3) Constructing a visual-semantic dual feedback loop to endow the model with physical perception and self-correction capabilities. Existing diffusion models are usually in a "blind generation" state, lacking perception of the geometric state during the generation process. This invention innovatively introduces differentiable rendering technology to construct an internal loop mechanism of "prediction-rendering-perception-correction". The geometric features extracted by the visual encoder enhance the router's perception and routing capabilities, while the semantic feedback of the visual language model is used to correct unreasonable physical interactions online. This mechanism transforms implicit physical constraints into explicit visual supervision signals, effectively solving physical distortion problems such as hand clipping and suspended grasping, and significantly improving the physical consistency and robustness of interactive actions. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, as provided in an embodiment of the present invention. Figure 2 An overview diagram of the hand object interaction action sequence generation method provided in the embodiments of the present invention; Figure 3 A schematic diagram of the architecture of the generative network provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the internal operation of the expert group as provided in this embodiment of the invention; Figure 5 The results of hand motion sequences under different text and object trajectory inputs are provided for embodiments of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] This invention provides a method for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, such as... Figure 1 As shown, it includes: Step S1: Obtain task data for the hand-object interaction task, the task data including initial text description, object point cloud, object motion trajectory and action category label; refine the initial text description to generate refined text descriptions corresponding to the hand level, finger level and joint level respectively; Step S2: Based on the task data and the diffusion model time step, encode the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, and the time step embedding; Step S3: Input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into the diffusion-based generative network to obtain the coarse hand action sequence output by the generative network; the generative network contains expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate the hand action sequence; Step S4: Based on the rough hand movement sequence and the object point cloud, perform differentiable rendering to obtain an interactive image, extract the visual features of the interactive image, generate physical correction text corresponding to each level based on the interactive image, and extract the physical correction text features of the physical correction text. Step S5: Input the initial text features, the physical correction text features corresponding to each level of the physical correction text, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image back into the generator network to obtain the denoised hand action sequence output by the generator network; wherein, the physical correction text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

[0030] Specifically, for the current hand-object interaction task, task data including initial text description, object point cloud, object motion trajectory, and action category labels are acquired. Here, the initial text description expresses the specific interaction method between the hand and the object in natural language, the object point cloud and object motion trajectory express the dynamic spatial data of the interactive object (i.e., the object) in three-dimensional space, and the action category labels mark the specific interaction type. Based on this, to improve the accuracy of hand-object interaction, a large language model can be used to hierarchically subdivide the initial text description, decomposing it into more detailed text descriptions for different levels of interaction. Specifically, three levels can be set: hand level, finger level, and joint level, thereby generating detailed text descriptions corresponding to these three levels.

[0031] In some embodiments, a hierarchical prompt template can be defined to guide the large language model in generating appropriate, refined text descriptions. This hierarchical prompt template includes hierarchical decomposition constraints, coarse-to-fine constraints, and interaction-specific constraints. The hierarchical decomposition constraints guide the large language model to refine the description into three levels: hand, fingers, and joints. The coarse-to-fine constraints guide the large language model to transition from overall posture description to local detail description, ensuring that the granularity of the description gradually transitions from the overall hand posture to the bending / extension of the fingers, and finally to the rotation and coordination of specific joints, in accordance with the natural logic of human kinematics. The interaction-specific constraints guide the large language model to describe the specific object components of the interaction and the coordination between the hands, thereby providing key information for generating physically accurate movements.

[0032] Subsequently, based on the aforementioned task data and the set diffusion model time step (since the subsequent generative model is built on a diffusion architecture, a diffusion model time step needs to be set), the corresponding encoder is used to encode the initial text features corresponding to the initial text description, the hierarchical refined text features corresponding to each level of the refined text description, the object geometric features corresponding to the object point cloud, the action label features corresponding to the action category label, and the time step embedding corresponding to the diffusion model time step, so that the generative network can generate hand action sequences.

[0033] In some embodiments, the initial text description, the refined text descriptions at each level, and the action category labels can be encoded using a pre-trained CLIP text encoder (Long-CLIP or standard CLIP, selected based on text length). The CLIP text encoder can also be used when subsequent encoding of the physically corrected text at each level is required. The object point cloud can be encoded using a pre-trained point cloud encoder (PointNet++). The diffusion model time steps can be encoded using a multilayer perceptron.

[0034] Figure 2 This is an overview diagram of the hand object interaction action sequence generation method provided in the embodiments of the present invention, as shown below. Figure 2 As shown, this embodiment of the invention constructs a denoising-perception-correction closed-loop architecture that includes sensing and feedback loops. In order to utilize intermediate prediction results to guide the generation process, the model feedforward computation within a single denoising time step is designed to include two consecutive forward propagation stages with shared network weights: a coarse estimation stage and a fine correction stage.

[0035] In the coarse estimation stage, the initial text features, the hierarchical refined text features corresponding to each level of refined text description, object geometric features, action label features, time step embeddings, object motion trajectories, and noisy hand action sequences are input into a diffusion-based generative network. Here, a random noise sequence with the same length as the target action sequence can be sampled from a standard Gaussian distribution as the noisy hand action sequence. It should be noted that during the training of the generative network, noise can be added to the real hand action sequence using a standard Gaussian distribution to obtain the noisy hand action sequence.

[0036] Among them, the initial text features and the hierarchical refined text features corresponding to each level describe the interaction between the hand and the object from the perspective of natural language and from different levels. The action label features represent the specific action type. The object geometric features and the object motion trajectory describe the dynamic data of the object in three-dimensional space during the interaction between the hand and the object from a spatial perspective. The time step embedding expresses the current noise intensity to guide the generator network to denoise. Under the guidance of the above information, the generator network can use the noisy hand action sequence as the initial sequence to generate a relatively coarse hand action sequence, that is, a rough hand action sequence.

[0037] To address the challenge of general models capturing the fine structure of the hand, this embodiment of the invention employs a multi-level expert group model. Specifically, the generative network includes expert groups corresponding to each level (i.e., hand expert group, finger expert group, and joint expert group). To accommodate the complexity of different action categories and noise intensities, each expert group contains multiple expert networks of varying sizes (hidden layer sizes) used to generate hand action sequences. Each expert network can utilize a multilayer perceptron structure.

[0038] After obtaining a rough hand motion sequence from the generator network output during the coarse estimation stage, an interactive image can be obtained by differentiable rendering based on this rough hand motion sequence and the object point cloud. Specifically, the rough hand motion sequence can be input into a differentiable renderer, and the corresponding interactive image can be obtained through real-time rendering of the object point cloud. The interactive image is then split into two feedback branches: In the visual feedback branch, the interactive image is input to a visual encoder (e.g., DINOv3) to extract visual features representing the spatial contact relationship between hand and object. These visual features are fed back to the perceptual router port of the generative network (described in detail later) to determine the activation weights of each expert network in each expert group. The outputs of the corresponding expert networks are then weighted and fused based on the activation weights of each expert network in each expert group to obtain the output of the entire generative network. In the semantic feedback loop, the interactive image is input to a visual language model (e.g., Qwen-VL). The visual language model recognizes the physical interactions in the interactive image and generates physical correction text in natural language form containing the corrected hand movements (e.g., "bend your thumb to touch the object surface"). This physical correction text is encoded to obtain physical correction text features and then fed back to the physical correction text feature port of the generative network to guide the generative network in correcting hand movements.

[0039] The process then proceeds to the fine-tuning stage. In this stage, the generator network again receives the same initial text features, object geometric features, action label features, time-step embeddings, object motion trajectories, and noisy hand motion sequences. In addition, the generator network's two feedback ports receive physically corrected text features corresponding to each level's physically corrected text and visual features from the interactive images. The visual features of the interactive images are used to reallocate the activation weights of the expert network, while the physically corrected text features are used to update the semantic guidance. The generator network performs a second inference based on the enhanced input fused with dual feedback signals, ultimately outputting a denoised hand motion sequence with higher geometric consistency and more realistic physical characteristics. It should be noted that steps S3 to S5 can be iteratively executed to achieve iterative denoising. The generated denoised hand motion sequence can be input into the MANO hand model to drive the generation of a visualized 3D hand mesh sequence, thus presenting an interactive animation that conforms to the text description and is physically realistic, such as... Figure 5 As shown.

[0040] As can be seen, the method proposed in this embodiment provides an end-to-end generation architecture, breaking through the limitations of the traditional multi-stage paradigm. This enables end-to-end joint optimization during model training, avoiding suboptimal solutions caused by staged training. By introducing a hybrid expert mechanism and a visual feedback mechanism, a hierarchical semantically guided expert group and a dual-channel visual feedback mechanism are constructed. This allows different expert groups to specialize in different hand-object interaction modes. Simultaneously, intelligent gating of the expert networks of each expert group is achieved through the visual features of the interactive images, enabling precise capture of fine-grained motion features. This effectively solves the problem that general models cannot simultaneously consider global pose and local details, significantly improving the quality of generated actions. Furthermore, real-time rendered interactive image feedback is used, and a visual encoder and visual language model are used to dynamically correct geometric penetration and physical inconsistencies during the generation process. This endows the generation network with physical perception and self-correction capabilities, overcoming the problem that existing diffusion models are usually in a "blind generation" state and lack perception of the geometric state during the generation process. Implicit physical constraints are transformed into explicit visual supervision signals, effectively solving physical distortion problems such as hand clipping and suspended grasping, significantly improving the physical consistency and robustness of interactive actions.

[0041] In some embodiments, such as Figure 3 As shown, the generative network can include multiple stacked generative submodules, each containing expert groups corresponding to different levels, such as... Figure 3 The expert groups shown are the hand expert group, the finger expert group, and the joint expert group. (As shown...) Figure 4 As shown, the expert group corresponding to any level includes a cross-attention layer for extracting hierarchical semantic features based on the hierarchical refined text features or physically corrected text features corresponding to the corresponding level, multiple expert networks for generating hand action sequences based on hierarchical semantic features, initial text features, object geometric features, action label features, time step embedding, object motion trajectory and noisy hand action sequences, and a perceptual router for calculating the activation weights of each expert network based on visual features, action label features and time step embedding.

[0042] In other embodiments, each generation submodule further includes a self-attention layer for performing a self-attention transformation based on the concatenation result of initial text features, object geometric features, action label features, time step embedding, object motion trajectory, and noisy hand action sequence or the hand action sequence output by the previous generation submodule (noisy hand action sequence for the first generation submodule, and the hand action sequence output by the previous generation submodule for subsequent generation submodules), to obtain self-attention features. Specifically, the object motion trajectory and the noisy hand action sequence or the hand action sequence output by the previous generation submodule can be concatenated along the feature dimension and mapped through a multilayer perceptron network to obtain a motion sequence. Then, the initial text features, object geometric features, action label features, and time step embedding are concatenated to the head of the motion sequence to obtain the final sequence. Finally, a self-attention mechanism is applied to this final sequence to obtain self-attention features.

[0043] The following section will use any one of the generation submodules as the object of description to describe the operation mechanism of the generation submodule in the coarse estimation stage and the fine correction stage. Those skilled in the art will understand that the operation mechanism of each generation submodule is the same and is executed in the order of stacking, so it will not be described in detail here.

[0044] In the coarse estimation stage (i.e., step S3), the self-attention layer of the generation submodule performs a self-attention transformation based on the initial text features, object geometric features, action label features, time step embedding, object motion trajectory, and the concatenation result of a noisy hand action sequence or the hand action sequence output by the previous generation submodule, to obtain self-attention features. These self-attention features are then input into the expert networks of each expert group within the generation submodule. Within each expert group of the generation submodule, the cross-attention layer extracts hierarchical semantic features from the corresponding level of refined text features based on a cross-attention mechanism. Subsequently, all expert networks within each expert group generate and output hand action sequences based on the aforementioned self-attention features and hierarchical semantic features. In this stage, the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group within the generation submodule and the activation weights of the expert networks of each expert group. The activation weights of the expert networks in any expert group are calculated by the perceptual router within that expert group based on action label features and time step embedding; that is, the input to the corresponding visual feature in the perceptual router is empty. The hand motion sequence output by the last generation submodule will be used as the output of the entire generation network, i.e., a coarse hand motion sequence.

[0045] In the fine-tuning stage (i.e., step S5), the self-attention layer of the generation submodule also performs self-attention transformation based on the initial text features, object geometric features, action label features, time step embedding, and the concatenation result of the object motion trajectory and noisy hand action sequence or the hand action sequence output by the previous generation submodule, to obtain self-attention features. Unlike step S3, in each expert group of this generation submodule, the cross-attention layer within each expert group extracts hierarchical semantic features from the corresponding level of physically corrected text features based on the cross-attention mechanism; simultaneously, the perceptual router in each expert group calculates the activation weights of each expert network based on visual features, action label features, and time step embedding. Subsequently, all expert networks within each expert group generate and output hand action sequences based on the aforementioned self-attention features and hierarchical semantic features. In this stage, the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group in this generation submodule and the activation weights of the expert networks of each expert group. Similarly, the hand motion sequence output by the last generation submodule will be used as the output of the entire generation network, i.e., the denoised hand motion sequence.

[0046] In other embodiments, in steps S3 and S5, the hand motion sequence output by the generation submodule is obtained by fusing the following: For each expert group, retain the K expert networks with the highest activation weights in that group, and reset the activation weights of the remaining expert networks to zero.

[0047] in, w i,k Let k be the activation weights of expert network k in expert group i. TopK ( w i () represents the set of activation weights of the K expert networks with the largest activation weights in the expert group; Then, the activation weights of the expert networks retained in all expert groups are renormalized to obtain the final activation weights of the corresponding expert networks. :

[0048] Based on the final activation weights of the expert networks retained in all expert groups, the hand movement sequences output by the corresponding expert networks are weighted and fused to obtain the hand movement sequence y output as the generation submodule:

[0049] in, This is the output of expert network k in expert group i.

[0050] In some embodiments, when allocating activation weights for each expert network in a corresponding expert group, the perception router can map visual features, action label features, and time step embeddings into visual spatial weight components, action weight components, and denoising step weight components, respectively. Each of these components contains multiple dimensions corresponding to each expert network. It should be noted that in step S3, the visual features are empty and the visual spatial weight component is 0. Here, the perception router can utilize independent linear layer branches to implement the mapping of these weight components:

[0051] in, w a , w t and w v These are the visual space weight components, action weight components, and denoising step weight components, all of which are three-dimensional vectors, with each dimension corresponding to an expert network. f a , f t and f v These are visual features, action label features, and time step embeddings, respectively.

[0052] Then, the above visual spatial weight components, action weight components, and denoising step weight components are added together and then normalized using Softmax to obtain the activation weights of each expert network in the corresponding expert group.

[0053] In other embodiments, the loss functions used when training the generative network include diffusion reconstruction loss, geometric reconstruction loss, distance loss, contact loss, penetration loss, smoothing loss, and expert balance loss. Among them, diffusion reconstruction loss L diff It is calculated based on the following formula:

[0054] in, This is a sequence of actual hand movements. It is a noisy hand motion sequence. The parameter is represented as The hand movement sequence output by the generative network. t For the diffusion model time step, c For task data, This is the expected value; Geometric Reconstruction Loss L rec It is calculated based on the following formula:

[0055] in, J and V These represent the actual hand joints and the hand mesh vertices, respectively. and The hand joints and hand mesh vertices are obtained by using the MANO model to generate the hand motion sequence output by the network. Distance loss L dist It is calculated based on the following formula:

[0056] in, Point cloud for objects, This represents the distance between two vertices; Contact loss L con It is calculated based on the following formula:

[0057] in, and These are the set of mutually contacting object points and the set of hand mesh vertices, determined based on a distance threshold, respectively. Penetration loss L pen It is calculated based on the following formula:

[0058] in, It is the set of object points that penetrate the grid of the hand; Contact loss and penetration loss optimize the distance for points in the contact area to encourage contact, and calculate the distance for points in the penetration area to penalize physical penetration. Smoothing loss L acc To penalize the acceleration of the hand vertices and ensure the temporal smoothness of the generated action sequence, a penalty can be applied, which can be calculated based on the following formula:

[0059] Among them, v i v i-1 v i-2 These are three consecutive hand mesh vertices in the hand motion sequence output by the generating network; To balance expert utilization and prevent the degradation of some experts, an expert balancing loss was applied to the expert activation weights in the perception router. L balance It is calculated based on the following formula:

[0060]

[0061] Among them, w a w t and w v These are the action weight components, the denoising step weight components, and the visual spatial weight components, respectively, where M is the number of expert networks in the generator network, and f i P represents the number of times the i-th expert network is retained during the current batch training process. i This represents the average activation weights of the i-th expert network calculated by the perceptual router during the current batch training process.

[0062] To enable the generative network to adapt to both the coarse generation stage ("without visual feedback") and the fine correction stage ("with visual feedback"), some embodiments employ a strategy combining online predictive feedback and probabilistic masking during training. Specifically, in each training iteration, the noisy input is first propagated forward using the current parameters of the generative network (corresponding to step S3) to obtain a coarse hand motion sequence, which is then used by a differentiable renderer and a visual encoder to generate visual features and physically corrected text features. Subsequently, the visual features and physically corrected text features are fed back to the generative network for a second forward propagation (corresponding to step S5). When calculating the model loss, a probabilistic masking mechanism is introduced: the visual features and physically corrected text features are randomly set to all-zero vectors or empty states with a preset probability. This strategy forces the generative network to alternately learn two capabilities during training: first, when visual feedback is masked, robust basic generation is performed relying solely on text and the diffusion time step; second, when visual feedback is present, the network can effectively utilize the visual features and physically corrected text features from the feedback to correct geometric errors. This effectively solves the state conflict problem caused by the difference in input distribution between the coarse and fine stages during the inference phase, ensuring the stability of closed-loop generation.

[0063] The following describes the 3D hand object interaction action sequence generation system based on hybrid expert and visual feedback provided by the present invention. The 3D hand object interaction action sequence generation system based on hybrid expert and visual feedback described below can be referred to in correspondence with the 3D hand object interaction action sequence generation method based on hybrid expert and visual feedback described above.

[0064] This invention provides a system for generating 3D hand object interaction action sequences based on hybrid expert and visual feedback, comprising: The data acquisition unit is used to acquire task data for hand-object interaction tasks. The task data includes an initial text description, object point cloud, object motion trajectory, and action category label. The initial text description is refined to generate refined text descriptions corresponding to the hand level, finger level, and joint level, respectively. The encoding unit is used to encode initial text features, hierarchical refined text features corresponding to each level of refined text description, object geometric features, action label features, and time step embedding based on the task data and the diffusion model time step. An action sequence generation unit is used to input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into a diffusion-based generative network to obtain a coarse hand action sequence output by the generative network; the generative network includes expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate hand action sequences. Based on the rough hand motion sequence and the object point cloud, a differentiable rendering is performed to obtain an interactive image. The visual features of the interactive image are extracted. Based on the interactive image, physical correction text corresponding to each level is generated and the physical correction text features are extracted. The initial text features, the physically corrected text features corresponding to each level, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image are input again into the generator network to obtain the denoised hand action sequence output by the generator network. The physically corrected text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

[0065] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0066] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0067] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0068] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a sequence of three-dimensional hand-object interaction actions based on hybrid expert and visual feedback, characterized in that, include: Step S1: Obtain task data for the hand-object interaction task, the task data including initial text description, object point cloud, object motion trajectory and action category label; refine the initial text description to generate refined text descriptions corresponding to the hand level, finger level and joint level respectively; Step S2: Based on the task data and the diffusion model time step, encode the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, and the time step embedding; Step S3: Input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into the diffusion-based generative network to obtain the coarse hand action sequence output by the generative network; the generative network contains expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate the hand action sequence; Step S4: Based on the rough hand movement sequence and the object point cloud, perform differentiable rendering to obtain an interactive image, extract the visual features of the interactive image, generate physical correction text corresponding to each level based on the interactive image, and extract the physical correction text features of the physical correction text. Step S5: Input the initial text features, the physical correction text features corresponding to each level of the physical correction text, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image back into the generator network to obtain the denoised hand action sequence output by the generator network; wherein, the physical correction text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

2. The hybrid-expert and visual feedback based three-dimensional hand-object interaction action sequence generation method of claim 1, wherein, The generative network includes multiple stacked generative sub-modules, each containing an expert group corresponding to each level. The expert group corresponding to any level includes a cross-attention layer for extracting hierarchical semantic features based on the hierarchical refined text features or physically corrected text features corresponding to the corresponding level, multiple expert networks for generating hand action sequences based on the hierarchical semantic features, the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory and the noisy hand action sequence, and a perceptual router for calculating the activation weights of each expert network based on the visual features, the action label features and the time step embedding.

3. The hybrid-expert and visual feedback based three-dimensional hand-object interaction action sequence generation method of claim 2, wherein, The generation submodule also includes a self-attention layer; In step S3, the self-attention layer of the generation submodule performs a self-attention transformation based on the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the concatenation result of the noisy hand action sequence or the hand action sequence output by the previous generation submodule to obtain self-attention features; the cross-attention layer in each expert group of the generation submodule extracts hierarchical semantic features from the hierarchically refined text features corresponding to the corresponding level based on the cross-attention mechanism; the perceptual router in each expert group of the generation submodule calculates the activation weights of each expert network based on the action label features and the time step embedding; the expert networks in each expert group of the generation submodule generate and output hand action sequences based on the self-attention features and the hierarchical semantic features respectively; the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group in the generation submodule and the activation weights of the expert networks of each expert group; the hand action sequence output by the last generation submodule is the coarse hand action sequence. In step S5, the self-attention layer of the generation submodule performs a self-attention transformation based on the initial text features, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the concatenation result of the noisy hand action sequence or the hand action sequence output by the previous generation submodule to obtain self-attention features; the cross-attention layer in each expert group of the generation submodule extracts hierarchical semantic features from the corresponding level of physically corrected text features based on the cross-attention mechanism; the perceptual router in each expert group of the generation submodule calculates the activation weights of each expert network based on the visual features, the action label features, and the time step embedding; the expert networks in each expert group of the generation submodule generate and output hand action sequences based on the self-attention features and the hierarchical semantic features respectively; the hand action sequence output by the generation submodule is obtained by weighted fusion of the hand action sequences output by the expert networks of each expert group in the generation submodule and the activation weights of the expert networks of each expert group; and the hand action sequence output by the last generation submodule is the denoised hand action sequence.

4. The hybrid-expert and visual feedback based three-dimensional hand-object interaction action sequence generation method of claim 3, wherein, In steps S3 and S5, the hand movement sequence output by the generation submodule is obtained by fusing the following methods: For each expert group, retain the K expert networks with the highest activation weights in that expert group, and reset the activation weights of the remaining expert networks to zero; The activation weights of the expert networks retained in all expert groups are renormalized to obtain the final activation weights of the corresponding expert networks. The hand movement sequences output by the corresponding expert networks are weighted and fused based on the final activation weights of the expert networks retained in all expert groups to obtain the hand movement sequences output by the generation submodule.

5. The hybrid expert and visual feedback based three-dimensional hand-object interaction action sequence generation method according to any one of claims 2 to 4, characterized in that, The perception router is used to map the visual features, the action label features, and the time step embedding into visual spatial weight components, action weight components, and denoising step weight components, respectively. The visual spatial weight components, action weight components, and denoising step weight components are then added together and normalized to obtain the activation weights of each expert network in the corresponding expert group. Each visual spatial weight component, action weight component, and denoising step weight component contains multiple dimensions corresponding to each expert network. In step S3, the visual features are empty, and the visual spatial weight components are 0.

6. The hybrid-expert and visual feedback based three-dimensional hand-object interaction action sequence generation method of claim 1, wherein, The initial text description is refined to generate refined text descriptions corresponding to the hand level, finger level, and joint level, respectively, specifically including: Obtain a hierarchical prompt template containing hierarchical decomposition constraints, coarse-to-fine constraints, and interaction-specific constraints. Among them, the hierarchical decomposition constraints guide the large language model to refine the description into three levels: hand, finger, and joint; the coarse-to-fine constraints guide the large language model to transition from overall posture description to local detail description; and the interaction-specific constraints guide the large language model to describe the specific object components of the interaction and the coordination between the hands.

7. The hybrid-expert and visual feedback based three-dimensional hand-object interaction action sequence generation method of claim 4, wherein, The loss functions used when training the generative network include diffusion reconstruction loss, geometric reconstruction loss, distance loss, contact loss, penetration loss, smoothing loss, and expert balance loss. wherein the diffusion reconstruction loss L diff is calculated based on the following equation: in, This is a sequence of actual hand movements. It is a noisy hand motion sequence. The parameter is represented as The hand movement sequence output by the generative network. t For the diffusion model time step, c For task data, This is the expected value; The geometric reconstruction loss L rec It is calculated based on the following formula: in, J and V These represent the actual hand joints and the hand mesh vertices, respectively. and The hand joints and hand mesh vertices are obtained by using the MANO model to generate the hand motion sequence output by the network. The distance loss L dist It is calculated based on the following formula: in, Point cloud for objects, This represents the distance between two vertices; The contact loss L con It is calculated based on the following formula: in, and These are the set of mutually contacting object points and the set of hand mesh vertices, determined based on a distance threshold, respectively. The penetration loss L pen It is calculated based on the following formula: in, It is the set of object points that penetrate the grid of the hand; The smoothing loss L acc It is calculated based on the following formula: where v i , v i-1 , v i-2 are respectively the consecutive three hand mesh vertices in the hand action sequence output by the generation network. The expert balance loss L balance It is calculated based on the following formula: wherein w a , w t and w v are the action weight component, the denoising step weight component and the visual space weight component respectively, M is the number of expert networks in the generation network, f i is the number of times the i-th expert network is retained in the current batch training process, P i is the average value of the activation weight of the i-th expert network calculated by the perception router in the current batch training process.

8. A three-dimensional hand object interactive action sequence generation system based on hybrid expert and visual feedback, characterized in that, include: The data acquisition unit is used to acquire task data for hand-object interaction tasks. The task data includes an initial text description, object point cloud, object motion trajectory, and action category label. The initial text description is refined to generate refined text descriptions corresponding to the hand level, finger level, and joint level, respectively. The encoding unit is used to encode initial text features, hierarchical refined text features corresponding to each level of refined text description, object geometric features, action label features, and time step embedding based on the task data and the diffusion model time step. An action sequence generation unit is used to input the initial text features, the hierarchical refined text features corresponding to each level of refined text description, the object geometric features, the action label features, the time step embedding, the object motion trajectory, and the noisy hand action sequence into a diffusion-based generative network to obtain a coarse hand action sequence output by the generative network; the generative network includes expert groups corresponding to each level, and each expert group contains multiple expert networks of different sizes used to generate hand action sequences. Based on the rough hand motion sequence and the object point cloud, a differentiable rendering is performed to obtain an interactive image. The visual features of the interactive image are extracted. Based on the interactive image, physical correction text corresponding to each level is generated and the physical correction text features are extracted. The initial text features, the physically corrected text features corresponding to each level, the object geometric features, the action label features, the time step embedding, the object motion trajectory, the noisy hand action sequence, and the visual features of the interactive image are input again into the generator network to obtain the denoised hand action sequence output by the generator network. The physically corrected text features are used to guide the generator network to perform hand action correction, and the visual features are used to determine the activation weights of each expert network in each expert group, so as to perform weighted fusion of the outputs of the corresponding expert networks based on the activation weights of each expert network in each expert group.

9. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-7.

Images