Method and device for obtaining dexterous operation teaching data set based on visual observation

Abstract

The invention provides a smart operation teaching data set acquisition method and device based on visual observation, and the method comprises the steps: preliminarily estimating the three-dimensional poses of a human hand and an object in each frame of a human hand teaching video according to a color image and a depth image in the human hand teaching video; and taking the initial pose estimation result as an initial value, performing joint optimization on the pose tracks of the human hand and the objects by utilizing an optimization objective function, and outputting the three-dimensional pose tracks of the human hand and the objects. And performing heterogeneous mapping on the three-dimensional posture track of the hand to generate a dexterous hand action sequence. And performing spatial transformation on the original track teaching data set by using the three-dimensional attitude track of each object to obtain a dexterous operation teaching data set. The dexterous hand operation teaching data is automatically extracted without an external wearable device, the teaching data acquisition efficiency is remarkably improved, and the method is suitable for various complex operation tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot learning and operation, and particularly relates to a dexterous operation teaching data set acquisition method and device based on visual observation. BACKGROUND

[0002] With the development of robot autonomous operation capability, multi-fingered dexterous hand has broad application prospects in complex operation tasks due to its high degree of freedom and anthropomorphic structure. However, the performance of imitation learning as an important method to realize dexterous operation strategy depends heavily on teaching data. How to efficiently acquire teaching data is a key problem faced by the robot field.

[0003] Current teaching methods can be divided into two categories: direct teaching and indirect teaching. Direct teaching directly acquires teaching information from the robot itself, but it depends on data gloves or motion capture systems, and has problems such as expensive equipment, complex operation, low data acquisition efficiency, etc., which limits its practical application in dexterous hand imitation learning. Indirect teaching records the task teaching process by wearing sensors or using third-party perception devices by the teacher, and converts it into data that can be used by the robot through teaching data processing. This method does not require the robot to participate in the acquisition process, which can greatly reduce the data acquisition cost and improve the data acquisition efficiency.

[0004] In the field of robot imitation learning, teaching data is mostly collected by human operation of the robot, which is costly and inefficient. Although the existing indirect teaching method can reduce the cost, it still has problems such as dependence on high-precision equipment and difficulty in extracting effective features. SUMMARY

[0005] The present application provides a dexterous operation teaching data set acquisition method and device based on visual observation, which solves the defects of existing indirect teaching methods that rely on high-precision equipment and have difficulty in extracting effective features, and automatically extracts dexterous hand operation teaching data without the need for external wearable devices, significantly improving the teaching data acquisition efficiency and being suitable for various complex operation tasks. The technical solution of the present application is as follows: In a first aspect, the present application provides a dexterous operation teaching data set acquisition method based on visual observation, comprising: acquiring a human hand teaching video, the human hand teaching video being collected by an RGB-D visual observation device; estimating the three-dimensional pose of the human hand and the object in each frame of the human hand teaching video based on the color image and the depth image in the human hand teaching video, to obtain a preliminary pose estimation result; using the preliminary pose estimation result as an initial value, and using a pre-constructed optimization objective function to jointly optimize the pose trajectory of the human hand and the object, to output the three-dimensional pose trajectory of the human hand and each object; Heterogeneous mapping is performed based on the 3D pose trajectory of the human hand to generate dexterous hand action sequences; the 3D pose trajectories of each object and the dexterous hand action sequences constitute the original trajectory teaching dataset; The original trajectory teaching dataset is spatially transformed using the three-dimensional attitude trajectories of each object to obtain a dexterous manipulation teaching dataset.

[0006] Optionally, the preliminary pose estimation result includes the object pose estimation result; the object pose estimation result is obtained in the following manner: The pre-established object detection network is used to detect objects using object names as prompts. The detected objects are then processed using a pre-established object segmentation model to obtain the segmentation mask of the objects. Construct an object information map based on the segmentation mask of the manipulated object; The differential renderer is used to render segmentation masks and depth images based on the object's pose, generating a rendered mask image and a rendered depth map. The segmentation loss is determined based on the rendered mask image and the object information map, and the first depth loss is determined based on the rendered depth map and the actual depth map; The object pose is estimated by minimizing the segmentation loss and the first depth loss, and the object pose estimation result is obtained.

[0007] Optionally, the optimization objective function includes a hand optimization loss, which is a weighted sum of joint point loss, joint angle constraints, and standard hand prior constraints. The joint point loss, joint angle constraint, and standard hand prior constraint are determined in the following ways: ; in, Indicates joint loss. The first number representing the number of people The three-dimensional position of each joint. The first number representing the number of people The two-dimensional position of each key point on the image. This indicates a projection operation. N Indicates the number of joints in a human hand; ; in, This indicates joint angle constraints. Represents the i-th dimension of the joint parameters. and These represent the minimum and maximum limits for the joint parameter, respectively. ; in, This represents the standard hand prior constraint. represent human hand pose parameters, represent average pose.

[0008] Optionally, the optimization objective function further comprises a hand-object joint optimization loss, which is a weighted sum of a mask loss, a second depth loss, an interaction repulsion loss, an interaction attraction loss, and a sequence smoothness loss. The mask loss is determined by rendering the three-dimensional model of the hand and the object onto a two-dimensional image plane, obtaining the corresponding mask image generated by the rendering, and calculating the coincidence loss with the corresponding segmentation mask image to obtain the mask loss. The second depth loss is determined by generating a rendered depth image by rendering the three-dimensional model of the hand and the object, comparing with the depth image, and determining the depth alignment of the hand and the object to obtain the second depth loss. The interaction repulsion loss is determined by the following way: for each vertex of the hand determining the closest object surface vertex in the Euclidean distance traversal manner calculating the dot product between the vector difference and the normal of the vertex . In the case where the dot product is greater than zero, it is determined that penetration occurs between the hand and the object, the closest distance from the penetrated hand vertex to the object surface vertex is calculated, and the closest distance corresponding to all penetrated hand vertices is accumulated to obtain the interaction repulsion loss. The interaction attraction loss is determined by the following way: ; wherein, represents the interaction attraction loss, represents the dot product corresponding to the i-th vertex, and tanh represents the hyperbolic tangent function, represents the gap between the hand and the object, represents the distance parameter, which controls the relationship between the attraction force and the distance. The sequence smoothness loss is determined by the following way: ; wherein, represents the sequence smoothness loss, respectively represent the hand and object parameters of the t-th frame, the t 1 frame, and the t 2 frame that need to be involved in the calculation.

[0009] Optionally, the optimization objective of the heterogeneous mapping comprises minimizing the relative vector similarity of the hand and the dexterous hand and the joint angle regularization term, as follows:​ ; wherein, represents a minimization function, and respectively represent the current time and the previous time of the dexterous hand joint angle, represents a joint angle regularization term, respectively represent the human hand shape parameter and the human hand pose parameter, r represents a hand global translation vector, represents a regularization coefficient, represents the relative vector similarity of the human hand and the dexterous hand; ; wherein, and respectively represent the relative vector of the human hand and the dexterous hand, represents the number of relative vectors, represents the dexterous hand joint angle.

[0010] Optionally, each original teaching trajectory in the original trajectory teaching data set is composed of a sequence of unit groups of dexterous hand poses, joint angles and object poses; The spatial transformation of the three-dimensional pose trajectory of each object on the original trajectory teaching data set to obtain the dexterous operation teaching data set, comprising: Divide the operation task into multiple sub-tasks according to the contact continuity principle, and each sub-task is associated with one or more objects, and the one or more objects constitute an associated object set of the sub-task; For each sub-task, a reference object is randomly selected from the associated object set thereof and a new initial pose is generated for the reference object, wherein: if there is an associated object that has been processed in the previous sub-task, the associated object is selected as the reference object and the generated pose is inherited; Otherwise, a reference object is randomly selected and a new pose that meets the task constraints is generated; Randomly extract an original teaching trajectory from the original trajectory teaching data set, and based on the new initial pose of the reference object, spatially transform the dexterous hand pose and the pose of the associated object in each frame of the original teaching trajectory to obtain a teaching segment in a new scene: wherein, the relative pose relationship between the dexterous hand and the reference object is kept unchanged during the spatial transformation, and the global pose of the dexterous hand and the associated object in the new scene is calculated through the rigid body transformation formula; Splice the teaching segments of each sub-task in the task logic order to obtain the dexterous operation teaching data set.

[0011] In a second aspect, the present application also provides a dexterous operation teaching data set acquisition device based on visual observation, comprising the following modules: The video acquisition module is used to acquire a human hand teaching video, which is obtained by an RGB-D vision observation device; The pose estimation module is used to preliminarily estimate the three-dimensional pose of the hand and the object in each frame of the hand teaching video based on the color image and depth image in the hand teaching video, and obtain the preliminary pose estimation result. The joint optimization module is used to jointly optimize the pose trajectories of the human hand and objects using the preliminary pose estimation results as initial values ​​and a pre-built optimization objective function, and outputs the three-dimensional pose trajectories of the human hand and each object. The heterogeneous mapping module is used to perform heterogeneous mapping based on the three-dimensional pose trajectory of the human hand to generate dexterous hand action sequences; the three-dimensional pose trajectories of each object and the dexterous hand action sequences constitute the original trajectory teaching dataset; The spatial transformation module is used to perform spatial transformation on the original trajectory teaching dataset using the three-dimensional attitude trajectories of each object, so as to obtain a dexterous operation teaching dataset.

[0012] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the method for acquiring a dexterous operation teaching dataset based on visual observation as described in the first aspect above.

[0013] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for acquiring a dexterous manipulation teaching dataset based on visual observation as described in the first aspect above.

[0014] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for acquiring a dexterous manipulation teaching dataset based on visual observation as described in the first aspect above.

[0015] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The application provides a dexterous operation teaching data set acquisition method and device based on visual observation, which uses a preliminary pose estimation result as an initial value, iteratively optimizes a global trajectory by using an optimization objective function, automatically corrects initial estimation errors without manual intervention, and significantly improves the accuracy and robustness of human hand and object pose estimation, especially in scenes such as occlusion and rapid motion. Existing methods usually design a teaching process for a specific manipulator structure, and have poor generalization ability. The application converts a human three-dimensional pose trajectory into a joint space action sequence of a dexterous hand through heterogeneous mapping, combines the spatial transformation of the object pose, and can adapt to dexterous hands (such as five-finger bionic hands and parallel grippers) of different degrees of freedom and different grasping modes, so that the generated teaching data can be directly used for reinforcement learning or imitation learning training, and the applicability of the method in complex operation tasks (such as fine assembly and tool use) is significantly expanded. Compared with traditional teaching which needs to be labeled frame by frame or repeated experiments, the application realizes an end-to-end automatic process from video input to teaching data output.

[0016] Moreover, the application only relies on ordinary RGB-D visual observation equipment, estimates the three-dimensional pose of the human hand and the object in the color image and the depth image directly, avoids the need for special hardware, reduces the system cost, and improves the convenience and scalability of data acquisition. The human hand teaching video is collected by using ordinary RGB-D visual observation equipment, without the need for wearable motion capture equipment or force feedback gloves, thereby reducing the hardware cost and operation constraints.

[0017] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description and the appended drawings.

[0018] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0020] Figure 1 is one of the flowcharts of the dexterous operation teaching data set acquisition method based on visual observation in the embodiments of the present application; Figure 2 is the second flowchart of the dexterous operation teaching data set acquisition method based on visual observation in the embodiments of the present application; Figure 3 is a hand shape parameter based sketch filtering method provided by an embodiment of the present application; Figure 4 is a before-and-after effect comparison diagram using an object information map in an embodiment of the present application; Figure 5 is a vertex diagram of a high-frequency contact area of a human hand in an embodiment of the present application; Figure 6 is a relative vector diagram of a human hand and a dexterous hand in an embodiment of the present application; Figure 7 is a teaching data augmentation process diagram in an embodiment of the present application; Figure 8 is a structure diagram of a dexterous operation teaching data set acquisition device based on visual observation in an embodiment of the present application; Figure 9 is a structure diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] Referring to FIG. Figure 1 The dexterous operation teaching data set acquisition method based on visual observation includes the following steps. S110, a human hand teaching video is acquired, and the human hand teaching video is collected by an RGB-D visual observation device.

[0023] The RGB-D visual observation device is an RGB-D camera (such as Intel RealSense or Microsoft Kinect). The RGB-D camera is fixed on a collection platform, task objects (such as tools and daily necessities) are placed in the scene, and an operator completes an operation task (such as object grabbing, rotating and placing) through natural operation. The RGB-D camera synchronously collects color images (RGB) and depth images (Depth) to capture continuous action details.

[0024] S120, a three-dimensional pose of a human hand and an object in each frame of the human hand teaching video is preliminarily estimated according to color images and depth images in the human hand teaching video, and a preliminary pose estimation result is obtained.

[0025] The human hand three-dimensional pose estimation method is: first, the MANO model and the HandOccNet network structure are used to estimate the three-dimensional pose of the human hand, then the Mediapipe is used to detect the wrist joint, and the spatial position of the wrist point is calculated combined with the camera internal parameter and the depth map. Finally, the unreliable prediction is removed according to the human hand shape parameter.

[0026] Figure 3 The process of filtering based on the human hand shape parameter in the application and the embodiment is shown. The filtering method is based on a core assumption: in the same video sequence, the shape of the human hand should remain relatively stable. In the video S, the reference shape parameter of the human hand in the video is calculated and the standard deviation : In the formula, represents the shape parameter of the human hand in the t-th frame, represents the total number of frames of the video. If the estimated shape parameter is greater than three times the standard deviation (i.e. 3 ) compared with the reference shape, that is, the shape parameter of the frame is considered to be beyond the acceptable range, the model estimation result of the frame is removed, and the pose estimation result of the adjacent frame is tracked.

[0027] The object pose estimation process is: first, using a target detection model (such as Grounding Dino) with the object name (such as "cup", "wrench") as the prompt word, the operating object is detected, and Grounding Dino locates the operating object in the RGB image, and outputs the bounding box. Then input the image in the bounding box into Mask R-CNN or Segment Anything Model (SAM) to generate the segmentation mask of the operating object. And based on the segmentation mask, the object information graph is constructed to reduce the influence of occlusion on pose estimation. The object pose is estimated by minimizing the segmentation loss and depth loss.

[0028] S130, using the preliminary pose estimation result as the initial value, the optimization objective function constructed in advance is used to jointly optimize the pose trajectory of the human hand and the object, and the three-dimensional pose trajectory of the human hand and each object is output.

[0029] The preliminary pose estimation result obtained in S120 is used as the initial value, and multiple constraints (including joint node loss, joint angle constraint, standard hand prior constraint, 2D mask error, depth consistency, contact constraint and time sequence smoothing) are introduced to jointly optimize the pose trajectory of the human hand and the object. According to the defined optimization objective function, the Adam optimizer is used for optimization, and the robust and continuous three-dimensional pose trajectory is output.

[0030] S140, heterogeneous mapping based on the three-dimensional pose trajectory of the human hand to generate the dexterous hand action sequence.

[0031] A heterogeneous mapping model between the human hand and the dexterous hand is established. The model defines relative vectors on the human hand and the dexterous hand, and solves the corresponding joint angle configuration of the multi-fingered dexterous hand by minimizing the vector error to generate a structurally feasible dexterous hand action. By using the Sequential Least-Squares Quadratic Programming (SLSQP) method for optimization, the problem can be effectively solved within the limit range of the joint angle. The neutral position q0=(q min q max ) / 2 is taken as the initial solution when the first frame is optimized, which can avoid local optimization near the joint limit. q0 represents the initial joint angle when the first frame is optimized, q min , and q max represent the minimum and maximum limit of the joint angle, respectively. For subsequent frames, the results of the previous frame are initialized, i.e. , and the optimized joint angle is passed to ensure the spatiotemporal consistency of the motion sequence. represents the initial joint angle when the t-th frame is optimized. The three-dimensional pose trajectory of each object and the dexterous hand action sequence constitute the original trajectory teaching data set . The original trajectory teaching data set contains N original teaching trajectories, each of which is composed of a sequence of unit groups of dexterous hand poses, joint angles, and object poses, and can be represented as: wherein, represents the i-th original teaching trajectory, represents the specific composition of the teaching trajectory data at time k. represents the specific composition of the teaching trajectory data at time k. represents the global pose of the dexterous hand at time k. represents the joint angle of the dexterous hand at time k. represents the object pose of the i-th object at time k. n represents the number of objects related to the teaching trajectory. is the pose set of n target objects.

[0032] S150, spatially transforming the original trajectory teaching data set using the three-dimensional pose trajectory of each object to obtain a dexterous operation teaching data set.

[0033] The operation task is decomposed into subtasks (such as "grab object A" and "move to position B"), and the contact continuity principle is followed (avoiding segmentation in the contact phase). Then, scene reconstruction is performed, specifically, an object associated with a subtask is randomly selected to generate a new initial pose (such as translating object 5 cm). If the object pose has been updated in a previous subtask, the pose is inherited to maintain consistency. Then, trajectory transformation is performed, specifically, a trajectory is randomly extracted from the original trajectory teaching data, and the pose of the dexterous hand and the object is transformed to the new scene. The uniform interpolation method is used to smooth the pose jumps. The enhanced teaching data set is output, that is, the above-mentioned dexterous operation teaching data set, which includes N trajectories, and each trajectory is composed of a sequence of dexterous hand poses, joint angles, and object poses.

[0034] The existing indirect teaching method (such as based on an inertial measurement unit (IMU) or an electromagnetic tracking device) needs to rely on high-precision external sensors, which is costly and complex to deploy. The present application only relies on ordinary RGB-D visual observation equipment, estimates the three-dimensional pose of the human hand and the object directly in the color image and the depth image, avoids the need for special hardware, reduces the system cost, and improves the convenience and scalability of data acquisition. The human hand teaching video is collected by using ordinary RGB-D visual observation equipment, without the need for wearable motion capture equipment or force feedback gloves, reducing the hardware cost and operational constraints.

[0035] The traditional method relies on manual annotation or pre-defined feature templates to extract key points, which is difficult to adapt to object deformation, occlusion or illumination changes in complex operation tasks. The present application uses the preliminary pose estimation result as the initial value, iteratively optimizes the global trajectory using the optimization objective function, automatically corrects the initial estimation error without human intervention, significantly improves the accuracy and robustness of the human hand and object pose estimation, and performs well in occlusion, fast motion and other scenes. The existing method usually designs a teaching process for a specific robot structure, which has poor generalization ability. The present application converts the human three-dimensional pose trajectory into a joint space motion sequence of the dexterous hand through heterogeneous mapping, and combines the spatial transformation of the object pose, which can adapt to dexterous hands of different degrees of freedom and different gripping modes (such as five-finger bionic hands, parallel grippers, etc.), and the generated teaching data can be directly used for reinforcement learning or imitation learning training, significantly expanding the applicability of the method in complex operation tasks (such as fine assembly, tool use). Compared with traditional teaching which needs to be annotated frame by frame or repeated experiments, the present application realizes an end-to-end automated process from video input to teaching data output.

[0036] Optionally, the preliminary pose estimation result includes an object pose estimation result; the object pose estimation result is obtained by: S210, using a pre-established target detection network to perform target detection with the object name as a prompt word, using a pre-established target segmentation model to process the detected operation object to obtain a segmentation mask of the operation object.

[0037] The Grounding DINO model is used as the target detection network, and the RGB image and the prompt word are input to output the bounding box of the operation object (format: [x_min, y_min, x_max, y_max]). The bounding box is input into a target segmentation model, such as the Segment Anything Model (SAM), to generate a binary segmentation mask of the operation object (pixel value is 0 or 1).

[0038] S220, construct an object information map based on the segmentation mask of the operation object. The object information map is defined as: wherein, represents the object information of the pixel point , represents that the pixel point belongs to the object category, represents that the pixel point belongs to the hand category, and the category of each pixel point is determined according to the segmentation mask. represents the normalized distance of the pixel point to the center of the object mask. The greater the distance, the weaker the correlation of the pixel point with the object, and the smaller the value assigned. is a magnitude parameter for controlling the influence of the change of the distance on the final result; is a scaling coefficient of the sigmoid function, which controls the range of the hand pixel value. and functions are used to control the range of the hand pixel value, to ensure that these values remain within a reasonable range; is a lower limit value, which ensures the minimum weight of the hand information in the optimization. It is used to ensure that the hand information can play a role in the object pose optimization process, and to avoid unnecessary influence of low-weight areas on the optimization result. is used to limit the value within the interval ,1].

[0039] S230, use a differential renderer to render the segmentation mask and the depth image according to the object pose, to generate a rendered mask image and a rendered depth image.

[0040] In the aspect of object initial pose calculation, the application uses the first frame picture information of the video, first calculates the translation vector T by operating the center point of the segmented mask of the object and its depth information, and combines the camera intrinsic parameters. At the same time, a plurality of rotation matrices R are uniformly sampled in the SO(3) space to generate a plurality of candidate initial pose schemes of rotation and translation vectors {R, T}. Each candidate scheme is optimized, and the pose scheme with the minimum loss after optimization is selected as the initial pose of the first frame. For subsequent frames, the optimization result of the previous frame is used as the initial value to ensure the consistency and continuity of the pose estimation in the time sequence.

[0041] S240, determine a segmentation loss according to the rendered mask image and the object information map, and determine a first depth loss according to the rendered depth map and the actual depth map.

[0042] The rendered mask image is compared with the object information map to calculate the segmentation loss wherein, the segmentation loss is represented by Lseg, the rotation matrix and the translation vector are represented by R and T respectively, the object information map is represented by I, the rendered mask image generated according to the object pose (the rotation matrix R and the translation vector T) is represented by I R T.

[0043] The first depth loss is calculated by comparing the rendered depth map generated by rendering and the actual depth map: wherein, the first depth loss is represented by Ldepth, the rendered depth map is represented by D R T, the actual depth map is represented by D.

[0044] S250, estimate the object pose by minimizing the segmentation loss and the first depth loss to obtain the object pose estimation result.

[0045] The objective function to be minimized is the weighted sum of the segmentation loss and the first depth loss: wherein, the minimized function is represented by L, and are weight coefficients of the segmentation loss and the first depth loss respectively.

[0046] Minimize the comprehensive loss by gradient descent , iteratively update the pose parameters (R and T), until convergence (such as the comprehensive loss is less than a preset threshold or the number of iterations reaches a maximum number of iterations), and output the optimized rotation matrix and translation vector.

[0047] The present application is based on object detection, instance segmentation and differential rendering for object pose estimation. Through a multi-stage optimization strategy, the accuracy and robustness of object pose estimation are significantly improved, overcoming the limitations of traditional methods in complex scenes. Traditional methods rely on manual annotation or pre-defined feature matching, making it difficult to adapt to object deformation, occlusion or category diversification, resulting in large positioning errors. The present application uses a target detection network to preliminarily locate the object, reducing the search space. Combined with a target segmentation model, an accurate segmentation mask is generated, avoiding background interference and improving object boundary recognition accuracy. Compared with traditional color or edge detection-based segmentation methods, the segmentation mask of the present application can more accurately describe the shape of the object, especially in complex scenes such as occlusion and light changes, still maintaining high robustness. An object information map is constructed as a supervision signal. A differential renderer is used to render the segmentation mask and depth map in real time, and compared with the object information map to calculate the segmentation loss and first depth loss. By jointly optimizing (minimizing segmentation loss first depth loss) to iteratively adjust the object pose, the local optimum problem is avoided. The differential renderer can optimize the pose end-to-end, converging faster and more accurately than traditional ICP methods. Simultaneous optimization of segmentation loss (to ensure alignment of object boundaries) and depth loss (to ensure depth consistency) forms multi-modal supervision, reducing the influence of a single error source. The differential renderer can provide real-time feedback on the rendering results, dynamically adjusting the optimization direction to avoid getting stuck in local optima.

[0048] Optionally, the optimization objective function in the above S130 includes two parts: hand optimization loss and hand-object joint optimization loss.

[0049] The hand optimization loss is the weighted sum of joint loss, joint angle constraint and standard hand prior constraint.

[0050] The joint loss uses camera intrinsic parameters to project three-dimensional joints onto the image plane and compares them with two-dimensional detected joints, defined as: wherein, represents the joint loss, represents the three-dimensional position of the th joint of the hand, which can be obtained through the MANO model. represents the two-dimensional position of the th joint of the hand on the image, which can be obtained by Mediapipe detection. represents the projection operation, N represents the number of joints of the hand, which includes 21 joints.

[0051] The joint angle constraint is defined to ensure the physiological reasonableness of the hand pose as: where, represents joint angle constraints, represents the i-th dimension of joint parameters (e.g. finger bending angle), and represents the minimum and maximum limit of the joint parameter, respectively.

[0052] The standard hand prior constraint, using the average pose from MANO model statistics As a prior, providing guidance for the estimation of hand pose, defined as: where, represents the standard hand prior constraint, represents the hand pose parameters, represents the average pose, which is the average hand pose prior learned from MANO statistics.

[0053] The hand optimization loss defined as the weighted sum of the joint center loss, joint angle constraints, and standard hand prior constraint: where, are the weight coefficients of the joint center loss, joint angle constraints, and standard hand prior constraint, respectively.

[0054] By minimizing the joint center loss, i.e. minimizing the error between the projected three-dimensional joint centers and the two-dimensional detected joint centers, it ensures that the estimated three-dimensional pose is consistent with the actual observed two-dimensional image. Two-dimensional detection may have noise or occlusion (such as fingers being occluded by objects), through multi-frame joint optimization and three-dimensional model constraints (MANO), the deviation of single-frame detection can be corrected. For example, if the index finger tip is misdetected in the two-dimensional image (e.g. shifted by 10 pixels), the optimization process will automatically correct this error through the geometric consistency constraints of the three-dimensional model (MANO) combined with the correct detection of other joint centers.

[0055] By joint angle constraints, the joint angles (e.g. finger bending, wrist rotation) are forced to be within the human kinematic limits (e.g. fingers cannot be bent in reverse), avoiding the generation of anti-joints or unnatural robot poses. Preventing numerical instability or divergence caused by joint angle out-of-bounds (e.g. exceeding ±90°) during optimization. Standard hand prior constraints take the average pose learned from statistics as a benchmark, guiding the optimization results towards common human hand poses, avoiding overfitting to noise or extreme poses. When part of the hand is severely occluded (e.g. fingers are not visible when fists are clenched), the standard hand prior constraints can provide reasonable pose speculation, avoiding incorrect estimation due to missing observations. The three loss functions work together, balancing observation alignment, physical feasibility, and naturalness, outputting three-dimensional poses that conform to both image evidence and biomechanics.

[0056] Optionally, the human hand-object joint optimization loss is a weighted sum of the mask loss, the second depth loss, the interaction repulsion loss, the interaction attraction loss, and the sequence smoothness loss.

[0057] The mask loss is determined by rendering the three-dimensional model of the human hand and the object onto a two-dimensional image plane, obtaining the corresponding mask image generated by rendering, and calculating the overlap loss with the corresponding segmentation mask image to obtain the mask loss. It is defined as: wherein, represents the mask loss, represents the segmentation mask image of the hand and the object, represents the corresponding mask image generated by rendering.

[0058] Further, the importance of each frame mask is evaluated by calculating the number of pixel points of each frame mask and the degree of overlap with the previous and next frame masks (using the intersection over union IoU as a measure), and an adaptive weight is provided for the mask loss. The importance calculation formula is defined as follows: wherein, represents the importance evaluation value of the i-th frame mask. This importance evaluation value is used to weight the i-th frame mask according to its importance when calculating the mask loss, to ensure that important frames play a greater role in the optimization process. represents the mask image of the i-th frame, represents the number of non-zero elements in the i-th frame mask, i.e. the number of pixels belonging to the object and the hand. represents the mask image of the j-th frame, represents the total number of frames of the video. represents the i-th 1frame and the intersection over union of the i-th frame mask, which measures the degree of overlap between the two frame masks, reflecting the continuity and stability of the position of the object and the hand between adjacent frames. is the importance evaluation value of the i-th frame mask. 1-frame mask, which is used to measure the continuity and stability of object and hand position between adjacent frames.

[0059] To ensure that the adaptive weight remains within a reasonable range, the importance evaluation value is normalized, defined as: wherein, is the adaptive weight of the i-th frame mask, represents the importance evaluation value of the j-th frame mask. By normalizing , the importance evaluation value is obtained.

[0060] The second depth loss is determined by rendering a three-dimensional model of the hand and the object to generate a rendered depth map, comparing it with the depth image, and determining the depth alignment of the hand and the object to obtain the second depth loss. The second depth loss is defined as: wherein, is the depth image, represents the rendered depth map, r represents the global translation vector of the hand (palm center coordinates).

[0061] The interaction repulsion loss can limit unreasonable penetration between models. The interaction repulsion loss is determined by: for each vertex of the hand , the closest object surface vertex is determined in a Euclidean distance traversal manner , the vector difference between and is calculated, and the dot product of the normal of the vertex is calculated , the formula is as follows wherein, represents the normal of the object surface vertex .

[0062] In the case where the dot product is greater than zero, it is determined that penetration occurs between the hand and the object, the closest distance from the penetrated hand vertex to the object surface vertex is calculated, and the closest distance corresponding to all penetrated hand vertices is accumulated to obtain the interaction repulsion loss.

[0063] wherein, represents the interaction repulsion loss, is the closest distance from the penetrated hand vertex to the object surface vertex.

[0064] In the process of human hand operating objects, there is a contact area between the human hand and the object, compared with the gap between the human hand and the object, and the human hand model and the object model have a small penetration. Therefore, this section introduces an interactive attraction term to reduce the excessive gap between the human hand and the object. Hasson et al. gave a vertex area through data statistics, and the hand vertex in this area often contacts the object. For the points in the contact area of the fingertips, first, use the same method as before to judge whether the vertex penetrates, if not, find the nearest vertex on the object, and calculate the distance (d) between the two. wherein, represents the interactive attraction loss, represents the number product corresponding to the i-th vertex, and tanh represents the hyperbolic tangent function, represents the distance measure of the gap between the human hand and the object, represents a parameter for controlling the hyperbolic tangent function tanh. It affects the change rate and output range of the tanh function, thereby adjusting the sensitivity of the interactive attraction loss to the gap distance.

[0065] The present application sets to 0.01. Unlike the interactive repulsion loss, it is unreasonable for the human hand to penetrate the object at any time, and in some scenarios, the hand and the object are actually not in contact, at which time the attraction is too large and will lead to inconsistency with the actual situation. Therefore, for the human hand vertex exceeding 2cm does not participate in the calculation of the interactive attraction loss . Moreover, the tanh function exhibits ideal gradient self-adaptive characteristics: when the fingertip vertex is close to the object surface, the gradient is large, providing a significant directional guide for the optimization process; when it is far away from the object surface, the gradient presents a smooth decay trend, which can reduce the attraction effect when the contact point exceeds the reasonable interaction range.

[0066] The sequence smoothing loss considers the continuity and correlation between frames, and by limiting the amplitude of parameter changes between adjacent frames, it ensures the smoothness and consistency of hand posture and object motion in time, and is defined as: wherein, represents the sequence smoothing loss. respectively represent the parameter change amount of the t-th frame relative to the t 1 frame, and the parameter change amount of the t 1 frame relative to the t 2 frame. respectively represent the t-th frame, and the t ​1 frame, t 2 frames require the hand and object parameters of the person participating in the calculation, which include the pose of the hand and the object.

[0067] The hand-object joint optimization loss is defined as the weighted sum of the mask loss, the second depth loss, the interaction repulsion loss, the interaction attraction loss, and the sequence smoothness loss: wherein, represents the hand-object joint optimization loss, respectively represents the weight coefficient of the mask loss, the second depth loss, the interaction repulsion loss, the interaction attraction loss, and the sequence smoothness loss, represents the adaptive weight.

[0068] The hand-object joint optimization loss proposed in the application significantly improves the pose estimation accuracy and physical rationality in the hand-object interaction scene through the weighted sum of multiple losses. The mask loss renders the three-dimensional model of the hand and the object to the two-dimensional image plane, obtains the corresponding mask generated by rendering, and calculates the coincidence loss with the corresponding segmentation mask. This loss can effectively improve the boundary alignment accuracy, even in the case of occlusion, it can reduce the boundary error caused by occlusion or light, enhance the occlusion robustness, and make the pose estimation more accurate in complex environments. The second depth loss generates a rendered depth map by rendering the three-dimensional model of the hand and the object, and compares it with the depth image to determine the depth alignment of the hand and the object. It can improve the depth alignment accuracy, reduce the depth error caused by sensor noise, and further avoid the error caused by depth jitter through the smooth constraint of adjacent frames in the video sequence.

[0069] The interaction repulsion loss determines the nearest object surface vertex to each vertex of the hand in a Euclidean distance traversal manner, calculates the dot product of the vector difference between the two and the vertex normal. When the dot product is greater than zero, it is determined that penetration occurs between the hand and the object, and the interaction repulsion loss is obtained by accumulating the nearest distance of all penetrated hand vertices to the object surface vertex. This loss can reduce unreasonable penetration, ensure that the hand and the object have no physical penetration, and improve the physical rationality of the pose, making the grasping action more stable in the grasping task. The interaction attraction loss focuses on the gap between the hand and the object, and uses the hyperbolic tangent function to constrain the gap, enhancing the contact tightness, avoiding excessive separation of the hand and the object, and avoiding unnecessary contact.

[0070] The sequence smoothness loss minimizes the difference of the pose changes between adjacent frames by calculating the pose changes between adjacent frames, improves motion smoothness, reduces temporal jitter, makes the motion sequence smoother, enhances temporal consistency, and can improve motion smoothness in dynamic operation tasks. These losses are jointly optimized by weighted combination, which can balance the influence of different constraints and adapt to different scenarios. For example, in tasks such as grasping and assembly, the interaction between the hand and the object is more natural, avoiding penetration or excessive separation, and improving the matching degree of simulation and reality. This provides a high-precision and high-robustness hand-object pose estimation scheme for dexterous operation teaching, significantly improving the quality and efficiency of teaching data acquisition.

[0071] In summary, the optimization objective function is defined as: wherein, r represents the global translation vector of the hand. represents the hand optimization loss for all frames and the hand-object joint optimization loss are accumulated.

[0072] The initial pose estimation result of S120 is used as the optimization initial value, while minimizing the hand optimization loss and the hand-object joint loss , the parameters are updated iteratively by gradient descent method, and the robust and continuous three-dimensional hand and object pose trajectory is obtained.

[0073] Optionally, the optimization objective of the heterogeneous mapping includes minimizing the relative vector similarity of the hand and the dexterous hand and the joint angle regularization term, realizing high-precision and smooth mapping from hand motion to dexterous hand control. The optimization objective is as follows: wherein, represents the minimization function, and represent the dexterous hand joint angles at the current time and the previous time, respectively, represents the joint angle regularization term, represents the hand shape parameter and the hand pose parameter, respectively, r represents the global translation vector of the hand, and c represents the regularization coefficient, used to balance the similarity optimization and the joint angle smoothness. represents the relative vector similarity of the hand and the dexterous hand; wherein, and represent the relative vectors of the hand and the dexterous hand (such as the vector from the wrist to the fingertip, the interphalangeal joint vector, etc.), The quantity of the relative vector is represented.The MANO model can be used to calculate the position coordinates of each joint by The position coordinates of each joint are calculated by the parameters, and the relative vector can be obtained by subtracting the corresponding coordinates. The joint angle of the dexterous hand is represented.

[0074] By optimizing the similarity of the relative vector, the joint movement direction of the human hand and the dexterous hand is ensured to be consistent. By optimizing the joint angle regularization term, the change of the joint angle of the dexterous hand is constrained to avoid mechanical shaking or over-limit. The three-dimensional pose trajectory of the human hand is input into the heterogeneous mapping model, and the output is the minimum .

[0075] The present application uses a heterogeneous mapping model based on the similarity of the relative vector, which is used to map the human hand motion to the dexterous hand motion, and solves the motion conversion problem of the traditional method in the case of large structural difference and non-matching degree of freedom. The model realizes the generation of the feasible dexterous hand motion by optimizing the similarity of the relative vector and the joint angle regularization term. The core optimization objective of the heterogeneous mapping model is to minimize the similarity of the relative vector of the human hand and the dexterous hand, and the joint angle regularization term is introduced to ensure the smoothness of the motion. The mapping relationship is established by the similarity of the relative vector (rather than the absolute joint angle), so that the dexterous hand can imitate the movement mode (such as grabbing and pinching) of the human hand, rather than mechanically copying the joint angle.

[0076] The generation process of the dexterous operation teaching data set of the present application is realized by task decomposition, sub-task space transformation and segment splicing, and the core is to use the relative pose relationship of the original teaching trajectory, combined with the randomized object pose transformation, to expand the data diversity and maintain the rationality of the operation logic. The above-mentioned S150 uses the three-dimensional pose trajectory of each object to perform spatial transformation on the original trajectory teaching data set to obtain the dexterous operation teaching data set, which comprises: S1501, the operation task is divided into a plurality of sub-tasks according to the contact continuity principle, each sub-task is associated with one or more objects, and the one or more objects constitute an associated object set of the sub-task.

[0077] The complex operation task is divided into multiple sub-tasks, each sub-task corresponding to one or more associated objects (such as grabbing a cup, moving a kettle). As in the task of pouring water, it can be divided into three sub-tasks of grabbing the kettle, moving to the water cup, and tilting to pour water, each sub-task being associated with the kettle or the water cup. When dividing, avoid splitting the task at the stage of contacting the object with the dexterous hand to ensure the smoothness of the motion trajectory. Each sub-task defines its associated object set (such as the "move the kettle" sub-task only associated with the kettle). The contact continuity principle prevents the dexterous hand from having a sudden change in pose when contacting the object, improving the physical reasonableness of the generated data. If there is a shared object between sub-tasks (such as the kettle appearing in multiple sub-tasks), the subsequent sub-task directly inherits its generated pose to avoid scene parameter conflicts. The shared object pose inheritance mechanism avoids repeated generation and improves efficiency.

[0078] S1502, for each sub-task, randomly select a reference object from its associated object set and generate a new initial pose for it.

[0079] For each sub-task, randomly select an object from its associated object set as a reference (such as selecting the kettle in the "move the kettle" sub-task). The pose generation rule is: if there is an associated object that has been processed in the previous sub-task (such as the kettle has generated a pose in the "grab the kettle" sub-task), select the associated object as the reference object and inherit the generated pose. Otherwise, randomly select a reference object and generate a new pose that meets the task constraints (such as the initial position of the kettle needs to be on the operation table). The pose constraint is that the generated pose needs to meet the task logic (such as the kettle mouth needs to be upwards) and the physical feasibility (such as the object cannot be suspended). By randomly selecting a reference object and generating a new pose, the coverage of the operation scene is expanded. The shared object pose inheritance ensures the parameter consistency of the multi-sub-task scene, avoiding unreasonable configurations (such as the kettle suddenly floating).

[0080] S1503, randomly select an original demonstration trajectory from the original trajectory demonstration dataset, based on the new initial pose of the reference object, perform spatial transformation on the dexterous hand pose and the associated object pose in each frame of the original demonstration trajectory to obtain a demonstration segment in the new scene: where the relative pose relationship between the dexterous hand and the reference object is kept unchanged during spatial transformation, and the global pose of the dexterous hand and the associated object in the new scene is calculated through the rigid body transformation formula.

[0081] Randomly select a trajectory from the original trajectory demonstration dataset, which contains the sequence of dexterous hand pose, joint angle and associated object pose. The pose of the dexterous hand relative to the reference object in the new scene is required to remain consistent with the original demonstration trajectory. Keep the relative pose relationship between the dexterous hand and the reference object unchanged, and convert the global pose of the dexterous hand in each frame of the original demonstration trajectory to the pose in the new scene through rigid body transformation, the formula is as follows: wherein, represents the transformation matrix from the object in the original scene to the object in the new scene. It represents the transformation of the object's position and orientation in the new scene relative to the original scene. represents the global pose of the dexterous hand at time k in the new scene. It represents the position and orientation of the dexterous hand relative to the world coordinate system in the new scene. is the global pose of the dexterous hand at time k. It represents the position and orientation of the dexterous hand relative to the world coordinate system in the original scene.

[0082] Transform the pose of the m associated objects of this subtask in each frame to the pose in the new scene. The pose of the mth object after transformation is calculated as follows: wherein, represents the global pose of the mth associated object at time k in the new scene. represents the global pose of the mth object at time k.

[0083] The pose of the non-associated object is determined by the previous or subsequent subtask (such as the water cup pose in the "pouring water" subtask inherited from the "moving water cup" subtask). The relative pose invariance ensures that the interaction relationship (such as the grasping point) between the dexterous hand and the object remains effective in the new scene. By generating a large amount of new scene data through random transformation, the adaptability of the model to different object configurations is improved.

[0084] S1504, splice the demonstration segments of each subtask in the order of task logic to obtain the dexterous operation demonstration dataset.

[0085] Splice the demonstration segments of each subtask in the order of task logic (such as "grasp the water jug" first, then "move to the water cup above"). Check continuity when splicing to ensure that the pose of the dexterous hand and the object does not have sudden changes at the junction of subtasks. Linear interpolation is performed on the pose of the dexterous hand at the splicing point to eliminate possible discontinuities. By smoothing the splicing noise, the demonstration trajectory is more consistent with the real operation process. Through subtask combination, complex operation data (such as assembly task) of long sequence and multiple stages can be generated.

[0086] The present application generates teaching data covering different scene configurations through random reference object selection and pose transformation, and improves the model generalization capability. The contact continuity principle and rigid body transformation ensure that the generated data conforms to the operation logic and physical constraints, avoiding invalid data. The shared object pose inheritance and subtask decomposition mechanism reduces the computational cost and supports large-scale data generation. The method is suitable for various dexterous hand structures and operation tasks (such as grasping, moving, and assembling), and has wide adaptability. The method provides high-quality and diversified teaching data for robot imitation learning, and significantly improves the training efficiency and robustness of complex operation tasks.

[0087] Figure 2 The flow of the dexterous operation teaching data set acquisition method based on visual observation in the present application and embodiments is shown. The method flow is as follows: first, hand and object pose estimation is performed from the hand teaching video collected by the RGB-D camera. When performing pose estimation, the segmentation mask and depth image are rendered according to the object pose by using a differential renderer, the segmentation loss and the first depth loss are calculated by comparing the rendered mask image, the rendered depth map and the object information map, and the pose estimation is performed by minimizing the weighted sum of the segmentation loss and the first depth loss. Then, sequence joint optimization is performed on the hand and object pose to obtain accurate three-dimensional pose information of the hand and object; subsequently, a heterogeneous mapping model is used to map the three-dimensional pose information of the hand to the dexterous hand domain to obtain the teaching data of the dexterous hand operation. Finally, an object-centered teaching data enhancement method is introduced, and the diversity of the teaching data is increased by aligning and transforming the object pose.

[0088] Sequence joint optimization is performed by using a joint optimization objective function, which includes a hand optimization loss and a hand-object joint optimization loss. The hand optimization loss includes a joint loss (re-projection) of the hand joint, a joint angle constraint, and a standard hand prior constraint. The hand-object joint optimization loss includes a mask loss, a depth loss, an interaction repulsion loss, an interaction attraction loss, and a sequence smoothness loss.

[0089] Figure 4 The effect comparison before and after using the object information map in the present embodiment is shown. Figure 4 (a) in the figure is the effect of fitting only by the segmentation mask and the rendered mask, Figure 4 (b) in the figure is the fitting effect after introducing the object information map. In order to more clearly show the comparison results, the first to fourth columns in the figure show, in order: the original image of the scene, the object pose estimation result image after projecting the estimated object pose to the original image, the visualization result image of the object mask and the object information map, and the object mask image generated by rendering. The introduction of the object information map can effectively solve the pose estimation error problem caused by hand occlusion.

[0090] Figure 5 The vertex of the high-frequency contact area of ​​the human hand in this embodiment is shown. For example... Figure 5 As shown, the ObMan dataset identifies the vertices where a person's hand frequently comes into contact with objects, and only considers these vertices when calculating the interaction attraction loss. Figure 5 The vertices shown are used for calculation.

[0091] Figure 6 The diagram illustrates the relative vectors defined in the human hand and the dexterous hand. The multi-finger dexterous hand used in this embodiment is an Allegro Hand, with only four fingers, corresponding to the thumb, index finger, middle finger, and ring finger of the human hand. To ensure structural correspondence and motion consistency, heterogeneous mapping is performed using the four fingers of the human hand, which have the closest structure. Specifically, relative vectors are defined in both the human hand and the dexterous hand, from the wrist to the fingertip (yellow line) and the intermediate joint (green line), and similarity mapping is performed based on these relative vectors. Represents the relative vector of a human hand. Represents the relative vector of a dexterous hand, i=1,2, ,8.

[0092] Figure 7 The process of performing teaching data augmentation in this embodiment is illustrated. For example... Figure 7 As shown, for each subtask, an object is randomly selected from the set of objects associated with the current subtask, and a new initial pose is generated for it to construct a new scene. After the scene is constructed, an original teaching trajectory is randomly extracted from the original teaching trajectory data (e.g., ...). Figure 7 (As shown in the yellow box). The global poses of the dexterous hand and the object in the new scene are calculated, generating new teaching segments for each subtask. The final dexterous manipulation teaching dataset is constructed by stitching together the teaching segments from each subtask. Uniform interpolation is used to smooth the dexterous hand's pose during initial position transformations and subtask stitching.

[0093] The following describes the device for acquiring dexterous manipulation teaching datasets based on visual observation provided by the present invention. The device for acquiring dexterous manipulation teaching datasets based on visual observation described below can be referred to in correspondence with the method for acquiring dexterous manipulation teaching datasets based on visual observation described above.

[0094] The present invention provides a visual observation-based dexterous manipulation teaching dataset acquisition device, referring to... Figure 8 As shown, it includes: The video acquisition module 310 is used to acquire a human hand teaching video, which is obtained by an RGB-D vision observation device; The pose estimation module 320 is configured to preliminarily estimate the three-dimensional pose of the human hand and the object in each frame of the human hand demonstration video according to the color image and the depth image in the human hand demonstration video, to obtain a preliminary pose estimation result. The joint optimization module 330 is configured to take the preliminary pose estimation result as an initial value, and perform joint optimization on the pose trajectory of the human hand and the object by using a pre-constructed optimization objective function, to output the three-dimensional pose trajectory of the human hand and each object. The heterogeneous mapping module 340 is configured to perform heterogeneous mapping based on the three-dimensional pose trajectory of the human hand, to generate a dexterous hand action sequence; the three-dimensional pose trajectory of each object and the dexterous hand action sequence constitute an original trajectory demonstration data set. The spatial transformation module 350 is configured to perform spatial transformation on the original trajectory demonstration data set by using the three-dimensional pose trajectory of each object, to obtain a dexterous operation demonstration data set.

[0095] Figure 9 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 9 As shown in FIG. 1, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can communicate with each other through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute a dexterous operation demonstration data set acquisition method based on visual observation.

[0096] In addition, the logical instruction in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0097] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the method for acquiring dexterous operation teaching data set based on visual observation.

[0098] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method for acquiring dexterous operation teaching data set based on visual observation.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0100] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A vision-observation-based dexterous manipulation teaching data set acquisition method, characterized by comprising the following steps: The method comprises the following steps: acquire a human hand demonstration video, which is collected by an RGB-D vision observation device; estimate the three-dimensional pose of the human hand and the object in each frame of the human hand demonstration video based on the color image and the depth image in the human hand demonstration video, and obtain a preliminary pose estimation result; use the preliminary pose estimation result as an initial value, and use a pre-constructed optimization objective function to jointly optimize the pose trajectory of the human hand and the object, and output the three-dimensional pose trajectory of the human hand and each object; perform heterogeneous mapping based on the three-dimensional pose trajectory of the human hand to generate a dexterous hand action sequence; the three-dimensional pose trajectory of each object and the dexterous hand action sequence constitute an original trajectory demonstration data set; perform spatial transformation on the original trajectory demonstration data set using the three-dimensional pose trajectory of each object to obtain a dexterous operation demonstration data set. 2.The vision-observation-based dexterous operation teaching data set acquisition method according to claim 1, characterized in that, The preliminary pose estimation result includes an object pose estimation result; the object pose estimation result is obtained by the following method: use a pre-established target detection network to perform target detection with the object name as a prompt word, and use a pre-established target segmentation model to process the detected operation object to obtain a segmentation mask of the operation object; construct an object information graph based on the segmentation mask of the operation object; use a differential renderer to render the segmentation mask and the depth image according to the object pose to generate a rendered mask image and a rendered depth image; determine a segmentation loss according to the rendered mask image and the object information graph, and determine a first depth loss according to the rendered depth image and the actual depth image; estimate the object pose by minimizing the segmentation loss and the first depth loss to obtain the object pose estimation result. 3.The vision-observation-based dexterous manipulation teaching data set acquisition method according to claim 1, characterized in that, The optimization objective function includes a human hand optimization loss, which is a weighted sum of a joint node loss, a joint angle constraint and a standard hand prior constraint; The joint node loss, the joint angle constraint and the standard hand prior constraint are determined by the following methods respectively: ； wherein, represents a joint loss, represents a three-dimensional position of a joint of the human hand, represents a two-dimensional position of a joint of the human hand on the image, represents a projection operation, N represents a number of joints of the human hand; ； wherein, denotes a joint angle constraint, denotes the i-th dimension of a joint parameter, and denotes the minimum limit and the maximum limit of the joint parameter, respectively; ； wherein, represents a standard hand prior constraint, represents a human hand pose parameter, represents an average pose. 4.The vision-observation-based dexterous operation teaching data set acquisition method according to claim 1, wherein, The optimization objective function further includes a human hand-object joint optimization loss, which is a weighted sum of a mask loss, a second depth loss, an interaction repulsion loss, an interaction attraction loss and a sequence smoothing loss; The mask loss is determined by the following method: rendering the three-dimensional model of the human hand and the object onto a two-dimensional image plane, obtaining the corresponding mask image generated by rendering, and calculating the coincidence loss with the corresponding segmentation mask image to obtain the mask loss; The second depth loss is determined by the following method: generating a rendered depth image by rendering the three-dimensional model of the hand and the object, comparing with the depth image, and determining the depth alignment of the hand and the object to obtain the second depth loss; The interaction repulsion loss is determined by the following way: for each vertex of the hand determining the closest object surface vertex in a euclidean distance traversal way calculating the dot product of the vector difference between and the normal of the vertex ; In the case where the scalar product is greater than zero, it is determined that penetration occurs between the hand and the object, the nearest distance from the hand vertex to the object surface vertex is calculated, and the nearest distance corresponding to all the hand vertices that have penetrated is accumulated to obtain the interaction repulsion loss; The interaction attraction loss is determined by the following method: ； wherein, represents the interactive attraction loss, represents the quantity product corresponding to the ith vertex, tanh represents the hyperbolic tangent function, represents the gap between the human hand and the object, represents the distance parameter, controlling the relationship between the attraction force and the distance; The sequence smoothing loss is determined by the following method: ； wherein, denotes a sequence smoothness loss, denotes the t-th frame, the t-th 1frame, the t-th 2frame, the hand and object parameters that need to be involved in the computation. 5.The vision-observation-based dexterous manipulation teaching data set acquisition method according to claim 1, wherein, The optimization objective of the heterogeneous mapping includes minimizing the relative vector similarity of the human hand and the dexterous hand and the joint angle regularization term, as follows: ； wherein, denotes a minimization function, and denotes a current time and a previous time, respectively, denotes a joint angle regularization term, denotes a human hand shape parameter and a human hand pose parameter, respectively, denotes a regularization coefficient, denotes a relative vector similarity of the human hand and the dexterous hand; ； wherein, and represents a relative vector of a human hand and a dexterous hand, respectively, represents a magnitude of the relative vector, represents a dexterous hand joint angle.

6. The vision-observation-based dexterous manipulation teaching data set acquisition method according to claim 1, characterized in that, Each original teaching trajectory in the original trajectory teaching data set is composed of a sequence of unit groups of dexterous hand poses, joint angles and object poses; The space transformation of the original trajectory teaching data set by the three-dimensional pose trajectory of each object obtains a dexterous operation teaching data set, which comprises: Dividing the operation task into a plurality of sub-tasks according to the contact continuity principle, each sub-task being associated with one or more objects, the one or more objects constituting an associated object set of the sub-task; For each sub-task, a reference object is randomly selected from the associated object set and a new initial pose is generated for the reference object, wherein: if an associated object has been processed in a previous sub-task, the associated object is selected as the reference object and the generated pose is inherited; otherwise, a reference object is randomly selected and a new pose conforming to the task constraint is generated; A raw teaching trajectory is randomly extracted from the original trajectory teaching data set, and the dexterous hand pose and the pose of the associated object in each frame of the raw teaching trajectory are space transformed based on the new initial pose of the reference object to obtain a teaching segment in a new scene: wherein the relative pose relationship between the dexterous hand and the reference object is kept unchanged during the space transformation, and the global pose of the dexterous hand and the associated object in the new scene is calculated by the rigid body transformation formula; The teaching segments of each sub-task are spliced in the order of task logic to obtain the dexterous operation teaching data set. 7.A dexterous operation teaching data set acquisition device based on visual observation, characterized in that, It comprises: A video acquisition module is configured to acquire a human hand teaching video, which is collected by an RGB-D vision observation device; A pose estimation module is configured to preliminarily estimate the three-dimensional pose of a human hand and an object in each frame of the human hand teaching video according to the color image and the depth image in the human hand teaching video, and obtain a preliminary pose estimation result; A joint optimization module is configured to use the preliminary pose estimation result as an initial value, and use a pre-constructed optimization objective function to jointly optimize the pose trajectory of the human hand and the object, and output the three-dimensional pose trajectory of the human hand and each object; A heterogeneous mapping module is configured to perform heterogeneous mapping based on the three-dimensional pose trajectory of the human hand to generate a dexterous hand action sequence; the three-dimensional pose trajectory of each object and the dexterous hand action sequence constitute an original trajectory teaching data set; A space transformation module is configured to use the three-dimensional pose trajectory of each object to perform space transformation on the original trajectory teaching data set to obtain a dexterous operation teaching data set.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the dexterous operation teaching data set acquisition method based on visual observation according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the dexterous operation teaching data set acquisition method based on visual observation according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the dexterous operation teaching data set acquisition method based on visual observation according to any one of claims 1 to 6.

Images