A method and system for generating an operation sequence guided by an object motion sequence

Abstract

The present invention relates to a method and system for generating a two-handed operation sequence guided by an object motion sequence. The system includes an object three-dimensional model library module, an object motion sequence editing module, an initial grasping posture generation module, an initial frame grasping posture screening module, a two-handed operation sequence generation module, a two-handed operation sequence screening module, and a post-processing optimization module; the method is: given a three-dimensional model of an operated object, the object space motion sequence is set, an initial grasping posture is generated based on the object shape, a suitable initial grasping posture is selected, and a two-handed operation sequence is generated in batches in conjunction with the object motion sequence, a suitable two-handed operation sequence is selected, and finally, the spatial distance between the object shape and the hand model is used to further optimize using a gradient method. The system solves the problems of low working efficiency and poor versatility of the current operation sequence generation methods, and can effectively improve the working efficiency of the production of digital content such as mixed reality and animated videos, and reduce the complexity of operations.

Description

Technical Field

[0001] The present invention relates to an operation sequence generation method, and more particularly to an object motion sequence guided operation sequence generation method and system. The present invention can be used to generate virtual operation content of surrounding objects in virtual reality / augmented reality, and can also be used for digital content production and AI-assisted editing such as animation and games, and can also be applied to the field of intelligent robot operation strategy generation. Background Art

[0002] In daily activities, humans interact with objects around them in a large number of scenarios, completing tasks with specific intentions by grabbing / moving / flipping the target objects. Whether in the field of robot operation, mixed reality digital content technology, or intelligent animation generation, there is a need to generate reasonable anthropomorphic grabbing and operation sequences for given target object movements. Although these operation sequence generation methods can complete some specific tasks, they are highly complex, prone to collisions when grabbing with both hands, low work efficiency, and poor versatility, making them difficult to produce digital content such as mixed reality and animated videos. Summary of the invention

[0003] In view of the problems of high complexity, easy collision of two-handed grasping, low work efficiency and poor versatility of the current operation sequence generation methods, a method and system for generating an operation sequence guided by an object motion sequence is proposed. The present invention mainly relates to a user giving an operated object motion sequence and generating a corresponding two-handed operation sequence under the object motion sequence. The present invention uses the object motion sequence and the contact mark between the two hands and the object as driving information, adopts a diffusion model in multiple stages to realize the generation of two-handed operation sequences, first generates an initial frame grasping posture based on the given information, and then generates a complete two-handed operation sequence based on this.

[0004] The technical solution of the present invention is:

[0005] A method for generating an operation sequence guided by an object motion sequence, using a given target object posture sequence to generate a two-handed humanoid dexterous hand operation sequence that conforms to human operation behavior habits; the module includes an object three-dimensional model library module, an object motion sequence editing module, an initial grasping posture generation module, an initial frame grasping posture screening module, an object sequence guided two-handed operation sequence generation module, a two-handed operation sequence screening module, and a post-processing optimization module; firstly, given a three-dimensional model of an operated object, a user edits and sets the object's spatial motion sequence based on task requirements, generates an initial grasping posture based on the object shape, selects a suitable initial grasping posture, and generates two-handed operation sequences in batches in conjunction with the object motion sequence, and the user selects a suitable two-handed operation sequence, and finally further optimizes the object shape and the spatial distance between the hand model using a gradient method;

[0006] Step 1. Train the initial grasping posture generation module, calculate the given hand metacarpal bone size as the hand feature, and use the MANO human hand model to represent the hand model. Based on the sequence point cloud composed of the object Mesh grid vertices, use BPS for feature encoding, and connect it in parallel with the object posture sequence, hand-object contact mark, and hand metacarpal bone size features as the conditional feature vector;

[0007] Step 2. Train the two-handed operation sequence generation module. The object's three-dimensional shape information is based on the posture changes in the given motion sequence of the operated object. The object shapes of different postures are encoded using BPS, and the hand-object contact identification vector and the object posture sequence are combined to form a conditional vector. The initial frame of the operation sequence in the training set is fixed, and only the remaining two-handed operation sequences except the initial frame are predicted;

[0008] Step 3. After the above two network modules are trained, the process paradigm of this method can be used to realize the generation of a two-handed operation sequence under a given object motion sequence, and finally a two-handed operation sequence of the full object motion sequence is obtained.

[0009] Furthermore, the specific operation sequence generation method is as follows:

[0010] Step 1. Train the initial grasping posture generation module, calculate the given hand metacarpal bone size as the hand feature, and use the MANO human hand model to represent the hand model. Based on the sequence point cloud composed of the object Mesh mesh vertices, use BPS for feature encoding, and connect it with the object posture sequence, hand-object contact mark, and hand metacarpal bone size features in parallel as the conditional feature vector; the diffusion model network training loss term includes the state encoding L1 norm as the loss function L x , L x It can be expressed as:

[0011]

[0012] Where X i Represents the true value of the state encoding vector of the two-handed operation sequence, Represents the predicted value of the state encoding vector of the two-handed operation sequence;

[0013] And the closest point x from the hand joint to the object surface obj The distance d from the hand joint to the object surface obj The vector sum should be the finger joint position x p , based on this relationship, the object surface point offset loss I is set xyz It is expressed as:

[0014]

[0015] where α 1 Represents the weight coefficient of the loss function term;

[0016] Since the state of the two-handed operation sequence is encoded as the joint position, in order to increase the coordination of the hand joints, the finger bone length constraint L is added sk :

[0017]

[0018] where α 2 Represents the weight coefficient of the loss function term;

[0019] in Represents the hand joint coordinates, i represents a joint index subscript in the finger joint sequence; Indicates the length of the bone represented by the predicted joint, sk gt Represents the true value of the corresponding finger bone length in the corresponding test set;

[0020] Step 2. Train the two-handed operation sequence generation module. The object's three-dimensional shape information is based on the posture changes in the given motion sequence of the operated object. The object shapes of different postures are encoded using BPS, and the hand-object contact identification vector and the object posture sequence are combined to form a conditional vector. The initial frame of the operation sequence in the training set is fixed, and only the remaining two-handed operation sequences except the initial frame are predicted; the relevant loss function is the same as the previous module;

[0021] Step 3. After the two network modules are trained, the process paradigm of the present invention can be used to realize the generation of a two-handed operation sequence under a given object motion sequence. The specific process paradigm is as follows:

[0022] Step 3.1, select the 3D model of the operated object and set the hand-object contact identification information sequence;

[0023] Step 3.2, given the object pose sequence, manually edit and adjust it to set the wrist orientation in the initial frame;

[0024] Step 3.3, based on the given initial object posture, object three-dimensional shape, initial frame hand-object contact identification information, wrist orientation of both hands, and finger skeleton vectors, the initial grasping posture generation module batch obtains the initial frames of the operation sequence of grasping the object with the right hand;

[0025] Step 3.4, select one frame from the initial frames of the batch-generated operation sequence as the output of the module;

[0026] Step 3.5, based on the known three-dimensional shape information of the object, the given posture sequence of the operated object, the given hand-object contact identification information, and the operation sequence initial frame obtained by the operation sequence initial frame generation module, a conditional diffusion model is used to batch generate a two-handed operation sequence;

[0027] Step 3.6, selecting an operation sequence from the batch-generated two-handed operation sequences as the output of the module;

[0028] Step 3.7: The two-handed operation sequence outputted in steps 3.5 and 3.6 is represented by a sequence of finger joint positions. Through inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist, i.e., the wrist position, are obtained. Combined with the object position information, the wrist motion sequence in the global coordinates can be obtained.

[0029] Step 3.8, fix the wrist motion sequence, and further optimize the finger rotation angle of the MANO model by using the gradient method according to the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model in the MANO hand model to the object point cloud; utilize the differentiability characteristics of the MANO model, based on the signed distance parameter between the output hand posture and the object to be grasped, the hand posture includes the wrist posture and the rotation angle of each finger, and further optimize the rotation angle of each finger of the output hand posture; select the vertices of the inner area of ​​the ring finger, middle finger, index finger and thumb as the contact area between the hand and the object, calculate the distance between the vertices of the corresponding area and the vertices of the object, and optimize the distance by gradient to make it close to zero; at the same time, it must also meet the requirements that all vertices of the hand should be outside the mesh surface of the object, and the vertices of the object mesh should also be outside the mesh surface of the hand;

[0030] It should be pointed out that the post-processing optimization module only optimizes the first and second joints of the fingers, and the wrist posture and the end joint remain fixed during the optimization process; the total optimization loss includes the contact loss term and the penetration loss term, and the contact loss function L f for:

[0031] L f =∑minD(P f ,P obj ),f∈{index finger, middle finger, ring finger, thumb}

[0032] Where P f Indicates the mesh vertex of the selected finger contact area, P obj Represents the mesh vertex of the object. This loss function represents the sum of the minimum distances between the vertices in the finger contact area and the vertices on the surface of the object. The penetration loss function L intp It is expressed as:

[0033] L intp =abs(min(0,(L dis +δ)))

[0034] L dis =min{D(P obj ,Phand )*Sign(N hand,nst ·(P obj -P hand,nst ))}

[0035] Where abs represents the absolute value, D(P obj ,P hand ) represents the distance between the object point cloud and the hand model point cloud, Sign represents the sign function, positive is 1, negative is -1, N hand,nst Represents the normal vector corresponding to each point in the hand point cloud closest to the object point cloud, P hand,nst Represents the point in the hand point cloud that is closest to the object point cloud;

[0036] Where L dis Indicates the signed distance between the object point cloud and the hand mesh vertex. The distance between the object surface vertex outside the hand mesh is positive, and the distance between the object surface vertex inside the hand mesh is negative. δ represents the distance threshold. The hand mesh vertex can invade a small amount inside the object mesh. Here, δ is empirically set to 4mm, indicating that the hand vertex can invade 4mm inside the object surface. In order to avoid the problem of penetration between different fingers during the gradient optimization process, different point cloud groups are selected from the finger joints and the midpoint positions of the joints. The distance between each finger point cloud group and other finger point cloud groups is calculated respectively. The related constraint function L fc It is expressed as:

[0037]

[0038] Where g represents the index number of the five fingers;

[0039] Among them, H f,g Indicates the point cloud group corresponding to the selected finger, H f,Ω represents the point cloud composed of other fingers except the selected finger; δ 2 represents the minimum distance between different fingers, which is 1.2 cm here; finally, the finger joint velocity in the wrist coordinate system is added as the smoothing loss term L jv :

[0040]

[0041] v k =‖J k -J k-1 ‖ 2

[0042] J k , J k-1 Respectively represent the hand joint position coordinates of the kth frame and the k-1th frame;

[0043] where v k represents the velocity of the finger joint in the wrist coordinate system, Represents the predicted joint position speed; the total optimization loss function L opt It is expressed as:

[0044] L opt =ω 1 L f +ω 2 L intp +ω 3 L fc +ω 4 L jv .

[0045] where ω 1 ,ω 2 ,ω 3 ,ω 4 are the weight coefficients of each loss item respectively;

[0046] The above-mentioned gradient optimization loss function is used to achieve local optimization of the rotation angle of each finger in the two-handed operation sequence after 30-50 gradient iterations;

[0047] After the process of step 3, a two-handed operation sequence of a full object motion sequence is finally obtained.

[0048] An object motion sequence guided operation sequence generation system is used to implement the object motion sequence guided operation sequence generation method as described above, including an object three-dimensional model library module, an object motion sequence editing module, an initial grasping posture generation module, an initial frame grasping posture screening module, an object sequence guided two-handed operation sequence generation module, a two-handed operation sequence screening module, and a post-processing optimization module; the main functions of each module are as follows:

[0049] The object 3D model library module contains the 3D shape information of the operated object, which is mainly represented by Mesh;

[0050] Object motion sequence editing module, in which the user sets the motion sequence of the operated object based on specific task requirements, including the position of the object in the world coordinate system and the rotation posture of the object;

[0051] The initial grasping posture generation module generates the initial frame grasping posture of both hands based on the given object 3D shape information, the set initial frame object posture, the set wrist orientation, and the hand-object contact identification information using the conditional diffusion model; the state encoding of the two-handed operation action sequence includes the finger joint position The closest point from the hand joint to the object surface Distance from hand joint to object surface The above vectors are measured in the local coordinate system centered on the object. The state encoding of the two-handed operation sequence can be expressed as X = [x p ,x obj ,d obj ];in, The hands contain 42 joints in total, and the spatial position of each joint contains 3 dimensions, xyz, so the total dimension is 42*3;

[0052] The initial frame generation module of the operation sequence is mainly implemented by the diffusion model. The corresponding conditional quantities are the object's three-dimensional shape information, the hand-object contact identification vector, the wrist orientation vector of both hands, and the finger skeleton vector. The finger skeleton identifies the size of the operating finger. The object shape information is encoded using the basic point set BPS feature. The output of the model is the state code X of the two-handed operation action sequence, and the finger joint position is encoded. After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist can be obtained;

[0053] Initial frame grasping posture screening module, the operation sequence initial frame generation module based on the diffusion model can batch generate various operation sequence initial frames. For the batch generated operation sequence initial frames, this module uses certain two-handed operation rules, such as wrist direction, wrist height, object contact area, or the user manually selects the appropriate operation sequence initial frame according to specific application requirements;

[0054] The object sequence guided two-handed operation sequence generation module is the core module of the present invention. It mainly uses the known three-dimensional shape information of the object, the given posture sequence of the operated object, the given hand-object contact identification information and the operation sequence initial frame obtained by the operation sequence initial frame generation module, and uses the conditional diffusion model to generate a natural and smooth two-handed operation sequence that conforms to human operation habits; similarly, the output of the model is a two-handed operation sequence composed of state codes X, and the finger joint positions are After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist can be obtained. Combined with the object position information, the wrist motion sequence in the global coordinate system can be obtained. In the motion sequence of the operated object in this module, the position change of the object is ignored. In the local coordinate system centered on the object, the three-dimensional shape sequence of different postures is encoded with BPS. The obtained object three-dimensional shape feature sequence represented by BPS, combined with the hand-object contact identification information and the object posture sequence, constitutes the conditional information of this network module. At the same time, the initial frame of the operation sequence remains fixed in the model inference stage, and only the operation sequence except the first frame is subjected to denoising and denoising operations.

[0055] The two-handed operation sequence screening module can generate various two-handed operation sequences in batches based on the above conditional diffusion model. This module uses certain two-handed operation rules, such as wrist direction, wrist height, object contact area, or the user manually selects the appropriate two-handed operation sequence according to specific application requirements;

[0056] Post-processing optimization module: This module fixes the wrist motion sequence and uses the gradient method to further optimize the finger posture according to the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model to the object point cloud, in order to obtain a more realistic two-handed operation sequence.

[0057] The beneficial effects of the present invention are:

[0058] The present invention mainly proposes a two-handed operation sequence generation method guided by an object motion sequence, which can effectively improve the work efficiency of the production of digital content such as mixed reality and animation videos, reduce operation complexity, and improve industrial upgrading. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural flow chart of the present invention;

[0060] Figure 2 It is the state coding diagram of the operation sequence of the present invention;

[0061] Figure 3 It is a module diagram for generating an initial frame of a two-hand operation sequence of the present invention;

[0062] Figure 4 A network structure diagram of a model for generating a two-handed operation sequence guided by an object motion sequence of the present invention;

[0063] Figure 5 A schematic diagram of contact and mold penetration in the gradient optimization of the present invention;

[0064] Figure 6 The schematic diagram is a diagram of the effect of the operation sequence generation of the present invention. DETAILED DESCRIPTION

[0065] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0066] The present invention provides a two-handed operation sequence generation method flow guided by an object motion sequence, which uses a given target object posture sequence to generate a two-handed humanoid dexterous hand operation sequence that conforms to human operation behavior habits. The main modules are: object three-dimensional model library module, object motion sequence editing module, initial grasping posture generation module, initial frame (rule-based / user) grasping posture screening module, object sequence guided two-handed operation sequence generation module, two-handed operation sequence screening module, and post-processing optimization module. The main process is as follows Figure 1 As shown, first, given a three-dimensional model of the object to be operated, the user edits and sets the object's spatial motion sequence based on task requirements, generates an initial grasping posture based on the object shape, selects a suitable initial grasping posture, and batch generates two-handed operation sequences based on the object motion sequence. The user selects a suitable two-handed operation sequence, and finally uses the gradient method to further optimize using the spatial distance between the object shape and the hand model.

[0067] Among them, the main functions of each module are as follows:

[0068] 1) Object 3D model library module, which contains the 3D shape information of the operated object, mainly represented by Mesh.

[0069] 2) Object motion sequence editing module, in which the user sets the motion sequence of the operated object based on specific task requirements, including the position of the object in the world coordinate system and the rotation posture of the object.

[0070] 3) Initial grasping posture generation module, based on the object 3D shape information given in module 1, the set initial frame object posture, the set wrist orientation, and the hand-object contact identification information, this module uses the conditional diffusion model to generate the initial frame grasping posture of both hands. The state encoding of the two-handed operation action sequence includes the finger joint position ( The hands contain 42 joints in total, and the spatial position of each joint contains 3 dimensions (xyz), so the total dimension is 42*3), the closest point from the hand joint to the object surface Distance from hand joint to object surface The above vectors are measured in the local coordinate system centered on the object. The state encoding of the two-handed operation sequence can be expressed as X = [x p ,x obj ,d obj ], such as Figure 2 shown.

[0071] Operation sequence initial frame generation module, the structure is as follows Figure 3As shown in the figure, the diffusion model is mainly used to implement it. The corresponding conditional quantities are the object's three-dimensional shape information, the hand-object contact identification vector, the wrist orientation vector of both hands, and the finger skeleton vector. The finger skeleton identifies the size of the operating finger. The object shape information is encoded using the basic point set BPS (Basis Point Set, BPS) feature. The output of the model is the state code X of the two-handed operation action sequence, and the finger joint position After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist can be obtained.

[0072] 4) Initial frame grasping posture screening module. The operation sequence initial frame generation module based on the diffusion model can batch generate various operation sequence initial frames. For the batch generated operation sequence initial frames, the module uses certain two-handed operation rules (such as wrist direction, wrist height, object contact area, etc.) or the user manually selects the appropriate operation sequence initial frame according to specific application requirements.

[0073] 5) Object sequence guided two-handed operation sequence generation module, which is the core module of the present invention, mainly uses the known object three-dimensional shape information, the given operated object posture sequence, the given hand-object contact identification information and the operation sequence initial frame obtained by the operation sequence initial frame generation module, and uses the conditional diffusion model to generate a natural and smooth two-handed operation sequence that conforms to human operation habits. Similarly, the output of the model is a two-handed operation sequence composed of state codes X, and the finger joint positions are After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist can be obtained. Combined with the object position information, the wrist motion sequence in the global coordinates can be obtained. The network structure diagram of this module is shown in the figure below. Figure 4 As shown in the figure, in the motion sequence of the operated object, the position change of the object is ignored, and the three-dimensional shape sequences of different postures are encoded with BPS in the local coordinate system centered on the object. The obtained three-dimensional shape feature sequence of the object represented by BPS is combined with the hand-object contact identification information and the object posture sequence to form the conditional information of this network module. At the same time, the initial frame of the operation sequence remains fixed in the model inference stage, and only the operation sequence except the first frame is subjected to denoising and denoising operations.

[0074] 6) A two-handed operation sequence screening module can generate a variety of two-handed operation sequences in batches based on the above-mentioned conditional diffusion model. This module uses certain two-handed operation rules (such as wrist direction, wrist height, object contact area, etc.) or the user to manually select a suitable two-handed operation sequence according to specific application requirements.

[0075] 7) Post-processing optimization module: This module fixes the wrist motion sequence and uses the gradient method to further optimize the finger posture according to the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model to the object point cloud, so as to obtain a more realistic two-handed operation sequence.

[0076] The specific operation sequence generation method is as follows:

[0077] Step 1. Train the initial grasping posture generation module, calculate the given hand metacarpal bone size as the hand feature, and use the MANO (Metric-Affine Hand Model) hand model to represent the hand model. Based on the sequence point cloud composed of the object mesh vertices, BPS is used for feature encoding, and it is connected in parallel with the object posture sequence, hand-object contact mark, and hand metacarpal bone size features as the conditional feature vector. The diffusion model network training loss term includes the state encoding L1 norm as the loss function L x , L x It can be expressed as:

[0078]

[0079] Where X i Represents the true value of the state encoding vector of the two-handed operation sequence, Represents the predicted value of the state encoding vector of the two-handed operation sequence.

[0080] And the closest point x from the hand joint to the object surface obj The distance d from the hand joint to the object surface obj The vector sum should be the finger joint position x p , based on this relationship, the object surface point offset loss L is set xyz It is expressed as:

[0081]

[0082] where α 1 Represents the weight coefficient of the loss function term.

[0083] Since the state of the two-handed operation sequence is encoded as the joint position, in order to increase the coordination of the hand joints, the finger bone length constraint L is added sk :

[0084]

[0085] where α 2 Represents the weight coefficient of the loss function term.

[0086] in Represents the hand joint coordinates, and i represents a joint index subscript in the finger joint sequence. Indicates the length of the bone represented by the predicted joint, sk gt Represents the true value of the corresponding finger bone length in the test set.

[0087] Step 2. Train the two-handed operation sequence generation module. The object 3D shape information is based on the posture changes in the given motion sequence of the operated object. The object shapes of different postures are encoded using BPS, and the hand-object contact identification vector and the object posture sequence are combined to form a conditional vector. The initial frame of the operation sequence in the training set is fixed, and only the remaining two-handed operation sequences except the initial frame are predicted. The relevant loss function is the same as the previous module.

[0088] Step 3. After the two network modules are trained, the process paradigm of the present invention can be used to realize the generation of a two-handed operation sequence under a given object motion sequence. The specific process paradigm is as follows:

[0089] Step 3.1, select the 3D model of the operated object, such as an apple, and set the hand-object contact identification information sequence to right hand contact;

[0090] Step 3.2, given the object pose sequence, manually edit and adjust it to set the wrist orientation in the initial frame;

[0091] Step 3.3, based on the given initial object posture, object three-dimensional shape, initial frame hand-object contact identification information, wrist orientation of both hands, and finger skeleton vectors, the initial grasping posture generation module batch obtains the initial frames of the operation sequence of grasping the object with the right hand;

[0092] Step 3.4, select one frame from the initial frames of the batch-generated operation sequence as the output of the module;

[0093] Step 3.5, based on the known three-dimensional shape information of the object, the given posture sequence of the operated object, the given hand-object contact identification information and the operation sequence initial frame obtained by the operation sequence initial frame generation module, the conditional diffusion model is used to batch generate two-handed operation sequences.

[0094] Step 3.6, selecting an operation sequence from the batch-generated two-handed operation sequences as the output of the module;

[0095] Step 3.7: The two-handed operation sequence outputted in steps 3.5 and 3.6 is represented by a sequence of finger joint positions. Through inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist (i.e., wrist position) are obtained. Combined with the object position information, the wrist motion sequence in the global coordinates can be obtained.

[0096] Step 3.8, fix the wrist motion sequence, and use the gradient method to further optimize the finger rotation angle of the MANO model based on the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model in the MANO hand model to the object point cloud. Utilizing the differentiability of the MANO model, the output hand posture (including wrist posture and each finger rotation angle) and the signed distance parameter between the object to be grasped are used to further optimize the finger rotation angle of the output hand posture. Figure 5 As shown in (a), the vertices of the inner area of ​​the ring finger, middle finger, index finger and thumb are selected as the contact area between the hand and the object, and the distance between the vertices in the corresponding area and the vertices of the object is calculated. The distance is optimized by gradient to make it close to zero. At the same time, all vertices of the hand should be outside the mesh surface of the object, and the vertices of the object mesh should also be outside the mesh surface of the hand. Figure 5 shown.

[0097] It should be noted that the post-processing optimization module only optimizes the first and second joints of the fingers, and the wrist posture and end joints remain fixed during the optimization process. The total optimization loss includes the contact loss term and the penetration loss term. The contact loss function L f for:

[0098] L f =∑minD(P f ,P obj ),f∈{index finger, middle finger, ring finger, thumb}

[0099] Where P f Indicates the mesh vertex of the selected finger contact area, P obj Represents the mesh vertex of the object. This loss function represents the sum of the minimum distances between the vertices in the finger contact area and the vertices on the object surface. intp It is expressed as:

[0100] L intp =abs(min(0,(L dis +δ)))

[0101] L dis =min{D(P obj ,P hand )*Sign(N hand,nst ·(P obj -P hand,nst ))}

[0102] Where abs represents the absolute value, D(P obj ,P hand ) represents the distance between the object point cloud and the hand model point cloud, Sign represents the sign function (positive is 1, negative is -1), Nhand,nst Represents the normal vector corresponding to each point in the hand point cloud closest to the object point cloud, P hand,nst Represents the point in the hand point cloud that is closest to the object point cloud.

[0103] Where L dis Indicates the signed distance between the object point cloud and the hand mesh vertex. The distance of the object surface vertex outside the hand mesh is positive, and the distance of the object surface vertex inside the hand mesh is negative. δ represents the distance threshold. The hand mesh vertex can intrude a small amount into the object mesh. Here, δ is empirically set to 4mm, indicating that the hand vertex can intrude 4mm into the object surface. In order to avoid the problem of penetration between different fingers during the gradient optimization process, we choose Figure 5 The finger joints and joint midpoints shown in (c) form different point cloud groups. The distances between each finger point cloud group and other finger point cloud groups are calculated respectively. The related constraint function L fc It is expressed as:

[0104]

[0105] Where g represents the index number of the five fingers;

[0106] Among them, H f,g Indicates the point cloud group corresponding to the selected finger, H f,Ω Represents the point cloud composed of other fingers except the selected finger. 2 Indicates the minimum distance between different fingers, which is 1.2cm here. Finally, add the finger joint velocity in the wrist coordinate system as the smoothing loss term L jv :

[0107]

[0108] v k =‖J k -J k-1 ‖ 2

[0109] J k , J k-1 They represent the hand joint position coordinates of the kth frame and the k-1th frame respectively.

[0110] where v k represents the velocity of the finger joint in the wrist coordinate system, Represents the predicted joint position velocity. The total optimization loss function L opt It is expressed as:

[0111] L opt =ω 1 L f +ω2 L intp +ω 3 L fc +ω 4 L jv .

[0112] where ω 1 ,ω 2 ,ω 3 ,ω 4 are the weight coefficients of each loss term.

[0113] The above-mentioned gradient optimization loss function is used to achieve local optimization of the rotation angle of each finger in the two-handed operation sequence after 30-50 gradient iterations.

[0114] After the process processing in step 3, a two-handed operation sequence of the full object motion sequence is finally obtained. Figure 6 shown.

[0115] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A method for generating an operation sequence guided by an object motion sequence, characterized in that: Using the given target object posture sequence, a two-handed humanoid dexterous hand operation sequence that conforms to human operation behavior habits is generated; the module includes an object three-dimensional model library module, an object motion sequence editing module, an initial grasping posture generation module, an initial frame grasping posture screening module, an object sequence-guided two-handed operation sequence generation module, a two-handed operation sequence screening module, and a post-processing optimization module; first, given the three-dimensional model of the operated object, the user edits and sets the object's spatial motion sequence based on task requirements, generates an initial grasping posture based on the object shape, selects a suitable initial grasping posture, and combines the object motion sequence to batch generate two-handed operation sequences. The user selects a suitable two-handed operation sequence, and finally uses the gradient method to further optimize the spatial distance between the object shape and the hand model; Step 1. Train the initial grasping posture generation module, calculate the given hand metacarpal bone size as the hand feature, and use the MANO human hand model to represent the hand model. Based on the sequence point cloud composed of the object Mesh grid vertices, use BPS for feature encoding, and connect it in parallel with the object posture sequence, hand-object contact mark, and hand metacarpal bone size features as the conditional feature vector; Step 2. Train the two-handed operation sequence generation module. The object's three-dimensional shape information is based on the posture changes in the given motion sequence of the operated object. The object shapes of different postures are encoded using BPS, and the hand-object contact identification vector and the object posture sequence are combined to form a conditional vector. The initial frame of the operation sequence in the training set is fixed, and only the remaining two-handed operation sequences except the initial frame are predicted; Step 3. After the initial grasping posture generation module and the two-handed operation sequence generation module are trained, the process paradigm of this method is used to realize the generation of two-handed operation sequence under a given object motion sequence, and finally a two-handed operation sequence of the full object motion sequence is obtained.

2. The method for generating an operation sequence guided by an object motion sequence according to claim 1, characterized in that: The specific operation sequence generation method is as follows: Step 1. Train the initial grasping posture generation module, calculate the given hand metacarpal bone size as the hand feature, the hand model is represented by the MANO human hand model, based on the sequence point cloud composed of the object Mesh grid vertices, use BPS for feature encoding, and connect it with the object posture sequence, hand-object contact mark, and hand metacarpal bone size features in parallel as the conditional feature vector; initial The training loss term of the grasping posture generation module includes the L1 norm of the state encoding as the loss function L x , L x It is expressed as: Where X i Represents the true value of the state encoding vector of the two-handed operation sequence, Represents the predicted value of the state encoding vector of the two-handed operation sequence; And the closest point x from the hand joint to the object surface obj The distance d from the hand joint to the object surface obj The vector sum should be the finger joint position x p , based on this relationship, the object surface point offset loss l is set xyz It is expressed as: Where α1 represents the weight coefficient of the loss function term; Since the state of the two-handed operation sequence is encoded as the joint position, in order to increase the coordination of the hand joints, the finger bone length constraint L is added sk : Where α2 represents the weight coefficient of the loss function term; in Represents the hand joint coordinates, i represents a joint index subscript in the finger joint sequence; Indicates the length of the bone represented by the predicted joint, sk gt Represents the true value of the corresponding finger bone length in the corresponding test set; Step 2. Train the two-handed operation sequence generation module. The object's three-dimensional shape information is based on the posture changes in the given motion sequence of the operated object. The object shapes of different postures are encoded using BPS, and the hand-object contact identification vector and the object posture sequence are combined to form a conditional vector. The initial frame of the operation sequence in the training set is fixed, and only the remaining two-handed operation sequences except the initial frame are predicted; the related loss function is the same as that of the initial grasping posture generation module; Step 3. After the initial grasping posture generation module and the two-handed operation sequence generation module are trained, the process paradigm of the present invention is used to realize the generation of the two-handed operation sequence under the given object motion sequence. The specific process paradigm is as follows: Step 3.1, select the 3D model of the operated object and set the hand-object contact identification information sequence; Step 3.2, given the object pose sequence, manually edit and adjust it to set the wrist orientation in the initial frame; Step 3.3, based on the given initial object posture, object three-dimensional shape, initial frame hand-object contact identification information, wrist orientation of both hands, and finger skeleton vectors, the initial grasping posture generation module batch obtains the initial frames of the operation sequence of grasping the object with the right hand; Step 3.4, select one frame from the initial frames of the batch-generated operation sequence as the output of the module; Step 3.5, based on the known three-dimensional shape information of the object, the given posture sequence of the operated object, the given hand-object contact identification information, and the operation sequence initial frame obtained by the operation sequence initial frame generation module, a conditional diffusion model is used to batch generate a two-handed operation sequence; Step 3.6, selecting an operation sequence from the batch-generated two-handed operation sequences as the output of the module; Step 3.7: The two-handed operation sequence outputted in steps 3.5 and 3.6 is represented by a sequence of finger joint positions. Through inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist, i.e., the wrist position, are obtained. Then, the wrist motion sequence in the global coordinates is obtained in combination with the object position information. Step 3.8, fix the wrist motion sequence, and further optimize the finger rotation angle of the MANO model by using the gradient method according to the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model in the MANO hand model to the object point cloud; utilize the differentiability characteristics of the MANO model, based on the signed distance parameter between the output hand posture and the object to be grasped, the hand posture includes the wrist posture and the rotation angle of each finger, and further optimize the rotation angle of each finger of the output hand posture; select the vertices of the inner area of ​​the ring finger, middle finger, index finger and thumb as the contact area between the hand and the object, calculate the distance between the vertices of the corresponding area and the vertices of the object, and optimize the distance by gradient to make it close to zero; at the same time, it must also meet the requirements that all vertices of the hand should be outside the mesh surface of the object, and the vertices of the object mesh should also be outside the mesh surface of the hand; It should be pointed out that the post-processing optimization module only optimizes the first and second joints of the fingers, and the wrist posture and the end joint remain fixed during the optimization process; the total optimization loss includes the contact loss term and the penetration loss term, and the contact loss function L f for: L f =∑minD(P f ,P obj ),f∈{index finger, middle finger, ring finger, thumb} Where P f Indicates the mesh vertex of the selected finger contact area, P obj Represents the mesh vertex of the object. This loss function represents the sum of the minimum distances between the vertices in the finger contact area and the vertices on the surface of the object. The penetration loss function L intp It is expressed as: L intp =abs(min(0,(L dis +δ))) L dis =min{D(P obj ,P hand )*Sign(N hand,nst ·(P obj -P hand,nst ))} Where abs represents the absolute value, D(P obj ,P hand ) represents the distance between the object point cloud and the hand model point cloud, Sign represents the sign function, positive is 1, negative is -1, N hand,nst Represents the normal vector corresponding to each point in the hand point cloud closest to the object point cloud, P hand,nst Represents the point in the hand point cloud that is closest to the object point cloud; Where L dis Indicates the signed distance between the object point cloud and the hand mesh vertex. The distance between the object surface vertex outside the hand mesh is positive, and the distance between the object surface vertex inside the hand mesh is negative. δ represents the distance threshold, which is a small amount of the hand mesh vertex invading the interior of the object mesh. Here, δ is empirically set to 4mm, indicating that the hand vertex intrudes 4mm into the object surface. In order to avoid the problem of penetration between different fingers during the gradient optimization process, different point cloud groups are selected from the finger joints and the midpoint positions of the joints. The distance between each finger point cloud group and other finger point cloud groups is calculated respectively. The related constraint function L fc It is expressed as: Where g represents the index number of the five fingers; Among them, H f,g Indicates the point cloud group corresponding to the selected finger, H f,Ω represents the point cloud composed of other fingers except the selected finger; δ2 represents the minimum distance between different fingers, which is selected as 1.2cm here; finally, the finger joint velocity in the wrist coordinate system is added as the smoothing loss term L jv : in k =||J k -J k-1 ||2 J k , J k-1 Respectively represent the hand joint position coordinates of the kth frame and the k-1th frame; where v k represents the velocity of the finger joint in the wrist coordinate system, Represents the predicted joint position speed; the total optimization loss function L opt It is expressed as: L opt =ω1L f +ω2L intp +ω3L fc +ω4L jv . Among them, ω1, ω2, ω3, and ω4 are the weight coefficients of each loss term; Using L opt The loss function is gradient optimized, and after 30-50 gradient iterations, the rotation angle of each finger in the two-handed operation sequence is locally optimized; After the process of step 3, a two-handed operation sequence of a full object motion sequence is finally obtained.

3. An operation sequence generation system guided by object motion sequence, characterized in that: The method for generating an operation sequence guided by an object motion sequence as claimed in any one of claims 1 or 2 comprises an object three-dimensional model library module, an object motion sequence editing module, an initial grasping posture generation module, an initial frame grasping posture screening module, an object sequence guided two-handed operation sequence generation module, a two-handed operation sequence screening module, and a post-processing optimization module; the main functions of each module are as follows: The object 3D model library module contains the 3D shape information of the operated object, which is mainly represented by Mesh; Object motion sequence editing module, in which the user sets the motion sequence of the operated object based on specific task requirements, including the position of the object in the world coordinate system and the rotation posture of the object; The initial grasping posture generation module generates the initial frame grasping posture of both hands based on the given object 3D shape information, the set initial frame object posture, the set wrist orientation, and the hand-object contact identification information using the conditional diffusion model; the state encoding of the two-handed operation action sequence includes the finger joint position The closest point from the hand joint to the object surface Distance from hand joint to object surface The above vectors are measured in the local coordinate system centered on the object. The state encoding of the two-handed operation sequence is represented by the symbol X = [x p ,x obj ,d obj ];in, The hands contain 42 joints in total, and the spatial position of each joint contains 3 dimensions, xyz, so the total dimension is 42*3; The initial frame generation module of the operation sequence is mainly implemented by the diffusion model. The corresponding conditional quantities are the object's three-dimensional shape information, the hand-object contact identification vector, the wrist orientation vector of both hands, and the finger skeleton vector. The finger skeleton identifies the size of the operation finger. The object shape information is encoded using the basic point set BPS feature. The output of the model is the state code X of the two-handed operation action sequence, and the finger joint position is encoded. After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist are obtained; The initial frame grasping posture screening module, the operation sequence initial frame generation module based on the diffusion model batch generates various operation sequence initial frames. For the batch generated operation sequence initial frames, this module uses the wrist orientation, wrist height, object contact area of ​​the two-handed operation rules, or the user manually selects the operation sequence initial frame according to specific application requirements; The object sequence guided two-handed operation sequence generation module is the core module of the present invention. It mainly uses the known three-dimensional shape information of the object, the given posture sequence of the operated object, the given hand-object contact identification information and the operation sequence initial frame obtained by the operation sequence initial frame generation module, and uses the conditional diffusion model to generate a natural and smooth two-handed operation sequence that conforms to human operation habits; similarly, the output of the model is a two-handed operation sequence composed of state codes X, and the finger joint positions are After the inverse kinematics IK transformation, the rotation angle of each finger, the position of the wrist relative to the center of the object, and the global posture of the wrist are obtained, and then the wrist motion sequence in the global coordinate is obtained in combination with the object position information; in the motion sequence of the operated object in this module, the position change of the object is ignored, and the three-dimensional shape sequence of different postures is encoded with BPS in the local coordinate system centered on the object. The obtained three-dimensional shape feature sequence of the object represented by BPS, combined with the hand-object contact identification information and the object posture sequence, constitutes the conditional information of this network module. At the same time, the initial frame of the operation sequence remains fixed in the model inference stage, and only the operation sequence except the first frame is subjected to denoising and denoising operations; The two-handed operation sequence screening module generates various two-handed operation sequences in batches based on the above conditional diffusion model. This module uses certain two-handed operation rules, such as wrist direction, wrist height, object contact area, or the user manually selects the appropriate two-handed operation sequence according to specific application requirements; Post-processing optimization module: This module fixes the wrist motion sequence and uses the gradient method to further optimize the finger posture according to the distance from the hand model vertex to the point cloud, the distance from the object point cloud to the hand model vertex, and the distance from the vertex of each finger sub-model to the object point cloud; To obtain a more realistic two-handed operation sequence.

Images