Robot teleoperation method, device, equipment, medium and product

Abstract

The invention relates to a robot teleoperation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring hand posture data of key points of a hand of a human body; performing pose calculation on the hand pose data to obtain a hand joint angle of the robot; performing pose conversion on the wrist pose data of the hand pose data to obtain wrist joint pose data of the hand of the robot; resolving the wrist joint pose data according to a preset inverse kinematics model to obtain a two-arm joint angle of the robot; and performing teleoperation on the robot according to the hand joint angle and the double-arm joint angle to obtain a hand operation result of the robot. By means of the method, the joint angles of the hands and the two arms of the robot can be calculated in real time at the plate end of the robot to achieve teleoperation, and balance of precision, usability and cost is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a robot teleoperation method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of robot technology, especially in recent years, the breakthrough of humanoid / semi-humanoid robots equipped with dual arms and dexterous hands in realizing autonomous operation through VLA (Vision-Language-Action Model), the robot teleoperation system has played an important role in the training data collection of the robot autonomous operation method represented by VLA. However, in the prior art, the monocular / stereoscopic vision scheme is cheap but the finger and wrist pose accuracy obtained is low, and pose drift and jitter are prone to occur. In order to realize the dual-arm and dexterous hand teleoperation scheme, an additional master arm and finger joint pose capture device are needed, which is not user-friendly. In addition, the pose accuracy obtained by optical motion capture is high but the price is expensive. At present, there is an urgent need for a robot teleoperation method that can accurately obtain the finger and wrist pose, and at the same time, solve the joint angles of the dexterous hand and dual arms at the robot board end to realize teleoperation, and improve the balanced performance of the overall scheme in terms of accuracy, ease of use and cost. SUMMARY

[0003] Therefore, it is necessary to provide a robot teleoperation method, device, computer equipment, computer readable storage medium and computer program product that can accurately obtain the finger and wrist pose, and at the same time, solve the joint angles of the dexterous hand and dual arms at the robot end to realize teleoperation, in order to solve the above technical problems.

[0004] In a first aspect, the present application provides a robot teleoperation method, comprising:

[0005] obtaining hand pose data of human hand key points;

[0006] performing pose solving on the hand pose data to obtain hand joint angles of a robot;

[0007] performing pose conversion on wrist pose data of the hand pose data to obtain wrist joint pose data of the hand of the robot;

[0008] solving the wrist joint pose data according to a preset inverse kinematics model to obtain dual arm joint angles of the robot;

[0009] teleoperating the robot according to the hand joint angles and the dual arm joint angles to obtain a hand operation result of the robot.

[0010] In one of the embodiments, the pose solving of the hand pose data to obtain the hand joint angle of the robot comprises:

[0011] The vector calculation according to the hand pose data obtains hand vector data;

[0012] The vector processing of the hand vector data obtains target vector data;

[0013] The optimization target calculation of the target vector data obtains target vector optimization data, and the joint angle optimization processing according to the target vector optimization data obtains the hand joint angle.

[0014] In one of the embodiments, the vector processing of the hand vector data to obtain the target vector data comprises:

[0015] The vector classification of the hand vector data obtains inter-finger vector data and wrist-to-finger vector data;

[0016] The vector length distance determination of the inter-finger vector data obtains a length distance determination result;

[0017] The projection of the inter-finger vector data according to the distance determination result obtains projection vector data;

[0018] The weight distribution according to the projection vector data and the wrist-to-finger vector data obtains the target vector data.

[0019] In one of the embodiments, the optimization target calculation of the target vector data to obtain target vector optimization data, and the joint angle optimization processing according to the target vector optimization data to obtain the hand joint angle comprises:

[0020] The robot state calculation of the target vector data obtains robot state data;

[0021] The forward kinematics calculation of the robot state data obtains current vector data;

[0022] The loss function optimization calculation according to the current vector data obtains the target vector optimization data;

[0023] The gradient calculation according to the target vector optimization data obtains target gradient data;

[0024] The joint angle optimization processing according to the target gradient data obtains the hand joint angle.

[0025] In one of the embodiments, the inverse kinematics model comprises a pseudo-inverse solution sub-model and a null space sub-model, and the wrist joint pose data is calculated according to the preset inverse kinematics model to obtain the dual-arm joint angle of the robot, comprising:

[0026] The pseudo-inverse solution sub-model is used to calculate the pseudo-inverse solution data of the wrist joint pose data;

[0027] The null space sub-model is used to optimize the pseudo-inverse solution data to obtain the dual-arm joint angle.

[0028] In one of the embodiments, the method further comprises:

[0029] Obtaining the joint angle control effect of the robot;

[0030] Generating a target operation trajectory according to the joint angle control effect and the hand operation result;

[0031] Generating a training operation trajectory sample and a training operation trajectory sample image based on the target operation trajectory, so as to train a target visual language action model.

[0032] In a second aspect, the present application further provides a robot teleoperation device, comprising:

[0033] An acquisition module is configured to acquire hand pose data of human hand key points;

[0034] A pose calculation module is configured to calculate the hand pose data to obtain the hand joint angle of the robot;

[0035] A pose conversion module is configured to convert the wrist pose data of the hand pose data to obtain the wrist joint pose data of the hand of the robot;

[0036] An inverse calculation module is configured to calculate the wrist joint pose data according to a preset inverse kinematics model to obtain the dual-arm joint angle of the robot;

[0037] A teleoperation module is configured to teleoperate the robot according to the hand joint angle and the dual-arm joint angle to obtain the hand operation result of the robot.

[0038] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0039] Acquiring hand pose data of human hand key points;

[0040] solving pose of the hand pose data to obtain a hand joint angle of the robot;

[0041] converting pose of wrist pose data of the hand pose data to obtain wrist joint pose data of the robot hand;

[0042] solving the wrist joint pose data according to a preset inverse kinematics model to obtain a double-arm joint angle of the robot;

[0043] teleoperating the robot according to the hand joint angle and the double-arm joint angle to obtain a hand operation result of the robot.

[0044] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0045] obtaining hand pose data of human hand key points;

[0046] solving pose of the hand pose data to obtain a hand joint angle of the robot;

[0047] converting pose of wrist pose data of the hand pose data to obtain wrist joint pose data of the robot hand;

[0048] solving the wrist joint pose data according to a preset inverse kinematics model to obtain a double-arm joint angle of the robot;

[0049] teleoperating the robot according to the hand joint angle and the double-arm joint angle to obtain a hand operation result of the robot.

[0050] In a fifth aspect, the present application also provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the following steps:

[0051] obtaining hand pose data of human hand key points;

[0052] solving pose of the hand pose data to obtain a hand joint angle of the robot;

[0053] converting pose of wrist pose data of the hand pose data to obtain wrist joint pose data of the robot hand;

[0054] solving the wrist joint pose data according to a preset inverse kinematics model to obtain a double-arm joint angle of the robot;

[0055] teleoperating the robot according to the hand joint angle and the double-arm joint angle to obtain a hand operation result of the robot.

[0056] The robot teleoperation method, device, computer equipment, computer readable storage medium and computer program product, by acquiring the hand pose data of the human hand key points; the hand pose data is solved, and the hand joint angle of the robot is obtained; the wrist pose data of the hand pose data is converted, and the wrist joint pose data of the robot hand is obtained; the wrist joint pose data is solved according to the preset inverse kinematics model, and the double-arm joint angle of the robot is obtained; the robot is teleoperated according to the hand joint angle and the double-arm joint angle, and the hand operation result of the robot is obtained. In specific implementation, the hand pose data of the human hand key points is acquired through the VR head-mounted display worn by the operator, the hand pose data is solved, and the hand joint angle of the robot is obtained; the wrist pose data of the hand pose data is converted, and the wrist joint pose data is obtained, and further, the wrist joint pose data is solved according to the preset inverse kinematics model, and the double-arm joint angle is obtained, the movement of the robot hand and the double arm is controlled through the obtained hand joint angle and the double-arm joint angle, the robot is teleoperated, and the hand operation result is obtained, and according to the hand operation result, the playback viewing in simulation and the training of visual language action model (VLA) can also be used. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0058] Figure 1 The application environment diagram of the robot teleoperation method in one embodiment;

[0059] Figure 2 The flowchart of the robot teleoperation method in one embodiment;

[0060] Figure 3 The model training flowchart of the robot teleoperation method in one embodiment;

[0061] Figure 4 The wrist joint pose data output by the VR head-mounted display coordinate system in the robot teleoperation method in another embodiment;

[0062] Figure 5 The wrist pose data of the left and right wrists in the robot base coordinate system in the robot teleoperation method in another embodiment;

[0063] Figure 6Fig. 2 is a schematic diagram of the complete flow of hand joint angle reorientation from the operator's hands to the robot's hand joints for the robot teleoperation method in another embodiment;

[0064] Figure 7 Fig. 5 is a schematic diagram of the hand key points output by the process of obtaining the human hand key points from the VR API for the robot teleoperation method in another embodiment;

[0065] Figure 8 Fig. 6 is a schematic diagram of the overall architecture of the robot teleoperation method in another embodiment;

[0066] Figure 9 Fig. 7 is a schematic diagram of the vector processing flow for the robot teleoperation method in another embodiment;

[0067] Figure 10 Fig. 8 is a structural block diagram of the robot teleoperation device in an embodiment;

[0068] Figure 11 Fig. 9 is an internal structural diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0069] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0070] The robot teleoperation method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains the hand pose data of the human hand key points; the hand pose data is calculated to obtain the joint angle of the robot hand; the wrist pose data of the hand pose data is converted to obtain the wrist joint pose data of the robot hand; the wrist joint pose data is calculated according to the preset inverse kinematics model to obtain the joint angle of the robot arm; the robot is remotely operated according to the hand joint angle and the arm joint angle, and the hand operation result of the robot is obtained. Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0071] In an exemplary embodiment, as shown in Figure 2 , a robot remote operation method is provided, and the method is applied to Figure 1 the server 104 in the figure for illustration, including the following steps 202 to step 210. Among them:

[0072] Step 202, obtaining hand pose data of human hand key points.

[0073] Among them, the human hand key points can include finger joint key points and wrist joint key points, wherein the human hand key points can include 26 key points as shown in Figure 7 Figure 7 26 hand key points contained in an embodiment are shown, which are not limited thereto.

[0074] ​In some embodiments, the hand pose data of the hand key points can be obtained by calling the interface of the VR headset system to output the finger joint pose and wrist joint pose of the operator. In this embodiment, the wearer can see the real surrounding environment by using the multi-camera perspective mode of the VR at the same time, which ensures that the wearer has an open operation field of view. Since the VR headset system only needs to transmit a small amount of lightweight pose data such as hand pose (finger joint pose) and wrist pose, it does not need to transmit data such as video stream that occupies a large bandwidth, thereby ensuring the real-time performance of data acquisition.

[0075] Specifically, as shown in Figure 4 and Figure 5 , wherein, Figure 4 shows the output wrist joint pose and finger joint pose in the VR base link in the VR headset coordinate system Figure 4 , wherein the left / right wrist pose in Figure 4 is the wrist joint pose of the left and right hands, and the left / right hand finger pose in Figure 4 is the finger joint pose of the left and right hands. Figure 5 shows the wrist joint pose of the left and right hands in the base_link in the robot coordinate system Figure 5 , which corresponds to the pose of left_link7 and right_link7 in Figure 5 , wherein each hand key point in Figure 4 and Figure 5 is pose data constructed by x, y, and z coordinate axes. The pose data includes position (x, y, z) and attitude (Orientation). The attitude can be represented as the angle of roll, pitch, and yaw, or a set of quaternions, or a 3x3 rotation matrix, without limitation.

[0076] In step 204, the hand pose data is pose-solved to obtain the hand joint angle of the robot.

[0077] The pose solving specifically refers to the specific pose processing process of the redirection from the hands of the operator to the hand joint angle of the robot, which can be pose solving by remapping algorithm, without limitation. In some embodiments, the specific process framework is shown in Figure 6 , wherein Figure 6 shows the complete process of pose solving the hand pose data to obtain the hand joint angle.

[0078] For example, the finger pose data can be pose-solved to obtain the hand joint angle by remapping, hand key point constraint, etc.

[0079] Step 206, pose conversion is performed on the wrist pose data of the hand pose data to obtain wrist joint pose data of the robot hand.

[0080] Wherein, the pose conversion on the wrist pose data refers to converting the pose data in the VR system coordinate system into the pose data in the robot coordinate system.

[0081] Exemplarily, the wrist pose data hand pose can be pose-converted through the conversion matrix of the two coordinate systems of the VR system coordinate system and the robot coordinate system to obtain the robot double-arm target wrist joint pose robot wrist pose.

[0082] Step 208, according to the preset inverse kinematics model, the wrist joint pose data is solved to obtain the joint angle of the robot double arm.

[0083] Wherein, the inverse kinematics model can adopt the double-arm inverse solution PINK IK model, and the solving refers to solving the corresponding joint angle of the double arm using the inverse kinematics model after obtaining the wrist joint pose data, that is, the inverse solution is performed to obtain the joint angle of the robot double arm.

[0084] Exemplarily, the inverse kinematics model PINK IK can be used to divide the solving into primary task pseudo-inverse solution calculation and zero space secondary optimization: the primary task solution is obtained by using the pseudo-inverse algorithm with weight or damping for the wrist / terminal pose, and then the secondary target is superimposed under the zero space projection, so as to output the joint angle of the double arm.

[0085] Wherein, the pseudo-inverse algorithm with weight or damping is used for solving, which refers to improving the motion stability of the robot near the singular point by exposing the damping coefficient; at the same time, the weight matrix is introduced to distinguish the priority of different tasks or the importance of each joint; during the solving and projection process, the constraints of joint position, velocity, acceleration / degree are considered synchronously, and the joint parameters are saturated or clipped if necessary.

[0086] Step 210, remote operation is performed on the robot according to the hand joint angle and the double-arm joint angle to obtain the hand operation result of the robot.

[0087] Wherein, the remote operation can refer to the operation of the simulation robot in the simulation environment, or the operation of the robot in the real environment, and the hand operation result is a kind of feedback data for feeding back the joint angle control effect.

[0088] Exemplarily, according to the hand joint angle and the double-arm joint angle, the hand operation result is obtained by sending to the simulation to drive the motion of the simulation robot through the network. For example, Figure 8As shown, the hand joint angle and the double-arm joint angle obtained by processing the pose data of the VR through the network are sent to the robot in the simulation scene or the real scene to drive the robot to move, and the hand operation result is obtained.

[0089] In the robot teleoperation method, the hand pose data of the human hand key points is obtained, the hand pose data is solved to obtain the hand joint angle of the robot, the wrist pose data of the hand pose data is converted to obtain the wrist joint pose data of the robot hand, the wrist joint pose data is solved according to the preset inverse kinematics model to obtain the double-arm joint angle of the robot, and the robot is teleoperated according to the hand joint angle and the double-arm joint angle to obtain the hand operation result of the robot. Therefore, the hand pose data of the human hand key points is obtained by wearing the VR head-mounted display of the operator, the hand pose data is solved to obtain the hand joint angle of the robot, the wrist joint pose data is obtained by converting the wrist pose data of the hand pose data, and the double-arm joint angle is further obtained by solving the wrist joint pose data according to the preset inverse kinematics model. The movement of the robot hand and the double arm is controlled by the obtained joint angle, the robot is teleoperated to obtain the hand operation result, and according to the hand operation result, the playback viewing in simulation and the training of the visual language action model (VLA) can also be used.

[0090] In some embodiments, the hand pose data is solved to obtain the hand joint angle, including: performing vector calculation on the hand pose data to obtain hand vector data; performing vector processing on the hand vector data to obtain target vector data; performing optimization target calculation on the target vector data to obtain target vector optimization data, and performing joint angle optimization processing on the target vector optimization data to obtain the hand joint angle.

[0091] In this embodiment, the finger pose data is solved by remapping, the pose solving is realized by vector calculation, vector processing and optimization target calculation of pose conversion, the hand joint angle of the robot is finally solved by joint angle optimization processing, and the movement of the robot hand is controlled according to the solved hand joint angle, so that the relative relationship between the fingers is accurately captured, the control strategy can be dynamically adjusted by optimization, and the smoothness and stability of the movement are maintained, which is suitable for different grasping and operation tasks.

[0092] It should be noted that vector processing is a detailed processing process of a series of vector redirection processing. In some embodiments, the detailed process of vector processing can be referred to in Figure 9 As shown, Figure 9 The detailed flowchart of vector processing is shown.

[0093] In some embodiments, the hand vector data is vector-processed to obtain target vector data, including: vector-classifying the hand vector data to obtain inter-finger vector data and wrist-to-finger vector data; performing vector length distance determination on the inter-finger vector data to obtain a length distance determination result; projecting the inter-finger vector data according to the distance determination result to obtain projected vector data; and performing weight allocation according to the projected vector data and the wrist-to-finger vector data to obtain the target vector data.

[0094] It should be noted that the hand vector data can be vector-classified according to the hand key points corresponding to the hand vector data to obtain the inter-finger vector data and the wrist-to-finger vector data. The inter-finger vector data is the vector data between the hand key points of different fingers.

[0095] Specifically, the inter-finger vector data is the spatial vector between two finger key points in the hand key points, for example, index_tip-middle_tip, which is used to represent the direction and distance from the index finger tip key point to the middle finger tip key point, and the inter-finger vector data is used to describe the relative position relationship between the fingers. The wrist-to-finger vector data is the spatial vector from the wrist key point in the hand key points to a finger tip key point in the hand key points, for example, index_tip-wrist, which is used to represent the direction and distance from the wrist key point to the index finger tip key point, and the wrist-to-finger vector data is used to describe the position and posture (relative position relationship) of the finger tip relative to the wrist.

[0096] However, when the length distance determination result determines that the fingers are close, i.e., the modulus of the inter-finger vector data is less than the projection determination distance (project_dist), the inter-finger vector data is projected, so that the vector length can be fixed to a preset value, which can make the vector shape match the kinematic constraint of the robot gripping action, and avoid the invalidation or distortion of the gripping instruction due to the too small original vector modulus. When the length distance determination result determines that the fingers are not so close, i.e., the modulus of the inter-finger vector data is greater than or equal to the projection determination distance (project_dist), the vector can be directly used to optimize the target calculation, which guarantees the data effectiveness and calculation efficiency. When the length distance determination result determines that the fingers are far away, i.e., the modulus of the inter-finger vector data is greater than the abnormal determination distance, in combination with the physiological motion constraint of the finger key points, the situation is beyond the normal working condition range, and it is determined that the key point detection algorithm is abnormal, so the data is directly discarded to avoid the negative impact of invalid data on the control accuracy.

[0097] Thus, this vector processing mechanism can prevent fingers from getting too close or separating; at the same time, the fixed vector length strategy implemented by projection transformation can ensure a gradual and smooth transition of the vector shape in the critical state, which helps to ensure the stability of the overall vector data and thus improve stable grasping and precise control.

[0098] In addition, based on the length and distance determination results of the vectors between a single group of related fingers (such as the thumb and index finger, the index finger and the middle finger, etc.), when the actual distance between the fingers reaches a preset extreme proximity threshold, the length of the vector between the fingers can be forcibly corrected to a preset safe length as a basic constraint mechanism. When the threshold is not reached, the original vector length is retained, thereby achieving the core function of avoiding excessive proximity or separation of a single group of fingers and ensuring the basic stability of the movement of a single group of fingers.

[0099] In other embodiments, such as Figure 9 As shown, in the projection processing, in addition to the first-order projection based on the preset distance, a second-order projection can also be performed according to the activation state of the first-order projection of the finger. For example, it is checked whether the relevant fingers have been activated for projection. Only when all the first-order projections of the relevant finger vectors (i.e., the relevant finger pairs) have been activated will the second-order projection be performed. If the relevant finger pairs have not been projected, the vector corresponding to the second-order projection will remain the original vector and will not be processed by the second-order projection, thereby further ensuring the coordination of finger movements.

[0100] First-order projection is a projection mechanism used to describe the projection of the target vectors of two finger keypoints in the human hand into a fixed-length vector (usually a constant with a small magnitude) when the distance between the two finger keypoints is very close. Second-order projection is a projection optimization performed after first-order projection (strengthening the weight of the projection vector in the overall optimization and realizing the coordinated projection optimization between multiple fingers). It forces the pair of fingers that have completed the first-order projection of the robot hand to maintain a fixed-length vector distance, thereby achieving the "second-order projection" effect to further enhance the stability and coordination of grasping.

[0101] It should be noted that first-order projection and second-order projection are performed simultaneously and work together in the optimization objective calculation and loss function. That is, second-order projection is performed at the same time as first-order projection. When first-order projection is triggered, the activation state and corresponding weights of the finger vector are updated. When second-order projection is triggered (second-order projection is only triggered when first-order projection is activated), projection optimization continues after the first-order projection vector is processed, and the corresponding weights are updated again.

[0102] The weight distribution according to the projection vector data and the wrist-to-finger vector data can be 200 when projecting in a first-order connection (between fingers), 400 when projecting in a second-order connection (finger coordination), and a special weight for the wrist-to-finger connection. The base weight is 1 in a non-projection state. The target vector data is obtained by weight distribution according to the above weight conditions. The weight system ensures that important vectors are processed preferentially. The weight distribution reflects the importance of different vectors, and the weight system can avoid abrupt changes.

[0103] It should be further explained that in the optimizer, the weight of the wrist-to-finger connection (i.e., the vector from the wrist to the tip of each finger) is set to a special value, i.e., a special weight, to enhance the importance of these connections in optimization. The specific implementation process is as follows. In the optimizer, the weight of the wrist-to-finger connection is first set to 200 (high_weight) to distinguish ordinary connections (weight 1), but in the subsequent code implementation process, this high weight parameter is further spliced into an independent weight component that specifically acts on the wrist-to-finger vector optimization calculation, allowing this type of vector to have a higher weight proportion in the loss function and further improving the fitting priority. This is done to prevent pose abnormalities caused by hand detection errors and to ensure that the spatial relationship between the wrist and the fingertips is more stable.

[0104] In some embodiments, the target vector data is optimized to obtain target vector optimization data, and the joint angle is optimized based on the target vector optimization data to obtain the hand joint angle, including: performing robot state calculation on the target vector data to obtain robot state data; performing forward kinematics calculation on the robot state data to obtain current vector data; performing loss function optimization calculation on the current vector data to obtain target vector optimization data; performing gradient calculation on the target vector optimization data to obtain target gradient data; and performing joint angle optimization processing based on the target gradient data to obtain the hand joint angle.

[0105] The optimization target calculation refers to a series of calculations and processing of the target vector data. It should be noted that the robot state data is obtained by calculating the target vector data. First, the current robot base data is obtained, which can be obtained by inputting the human hand data or the detection result. Then, the distance of each finger-tip to finger-tip line target_vec_dist is calculated, and it is determined whether it needs to be "projected". If the distance is less than the preset projection trigger threshold project_dist, it is marked as projected, and if it is greater than the preset projection escape threshold escape_dist, it is removed from the projection. Further, the weight of the corresponding vector is generated: the weight of the line in the projection state is high, that is, the weight of the finger-tip to finger-tip line in the robot base data is higher than that of the second-order combination. The second-order combination can be a derived vector synthesized by a plurality of robot base data (first-order) finger-tip to finger-tip lines through operation, and the unprojected line is a regular weight, and the wrist-finger-tip line is additionally added with a uniform weight. Finally, the reference vector is generated: for the projected line, the direction vector is multiplied by the fixed length projected_dist to form the "robot target projection vector", and the unprojected line keeps the original value. The finger-tip to finger-tip vector after projection is spliced with the original wrist-finger-tip vector to obtain the final robot state data reference_vec, which is used as the target of optimization alignment.

[0106] It should be further noted that the state of the robot needs to be loaded before optimization. The specific process is as follows: first, the full joint array qpos of the robot is constructed, wherein the fixed joint that does not need to participate in the optimization iteration is directly assigned with fixed_qpos (fixed joint parameter preset value), and the target joint to be optimized is written and filled by the optimization variable x. If the kinematic adaptor (kinematic adaptor, processing mimic / mapping) is configured in the robot model, the current joint data needs to be expanded into the real robot joint sequence by calling adaptor forward_qpos. The kinematic adaptor is used to make bidirectional mapping between the "optimization parameter space" and the "real robot joint space", forward completion constraint, and backward gradient summary, to avoid directly optimizing the mimic joint which cannot be independent.

[0107] In one specific embodiment, the forward kinematics calculation is performed on the robot state data to obtain the current vector data, including: calling the forward kinematics function compute_forward_kinematics(qpos), inputting the robot full joint array qpos for operation, extracting the pose translation information of the relevant link in the robot state data from the operation result to form the position list body_pos; then taking out the link positions of origin and task according to the start / endpoint index respectively, calculating the current robot connection vector robot_vec = task - origin (the vector covers all the finger-finger and wrist-finger connection lines), so as to obtain the current vector data.

[0108] Then, by calculating the norm error ||robot_vec - reference_vec|| of each connection line, the error is processed by Huber loss to reduce the sensitivity to outliers. The scalar objective value is obtained by weighting, averaging / summing according to the weight in the robot state calculation step, and the scalar objective value is returned to the optimizer as the target vector optimization data. If the optimizer requests gradient, gradient calculation is performed according to the target vector optimization data, the local Jacobian of each link involved is calculated and rotated to the world coordinate system; if there is a kinematics adaptor, the backward_jacobian method is called to rearrange the Jacobian matrix to the target joint order. The Huber loss is back-propagated to obtain the gradient of each link position, which is multiplied by the Jacobian to be summarized to the joint gradient. The time regularization term 2 * norm_delta *(x - last_qpos) is added to encourage the joint angle to approach the last time, so as to avoid jitter. The target gradient data result is written into grad for NLopt to use.

[0109] Finally, the NLopt (SLSQP) uses the target gradient data and gradient to iteratively update the joint angle vector x, stops after meeting the tolerance, so as to realize the optimization of the joint angle, output the optimized target joint angle, and obtain the hand joint angle, which together with the fixed joint forms the final robot state.

[0110] In this embodiment, the target vector optimization data can be used to further calculate the Jacobian matrix and perform back propagation for joint angle optimization processing. The specific optimization process can use the SLSQP (Sequential Least Squares Programming) optimizer to adapt to the processing of nonlinear optimization problems with constraints. In the optimization process, a constraint condition of joint angle range limitation is set. Finally, the target function value and gradient are calculated to update the joint angle, and the convergence condition is checked. Through post-processing, the kinematic constraints of the simulated joint are implemented, the trajectory is smoothed, and the jitter is reduced.

[0111] In this embodiment, the core idea of remapping is to achieve more accurate finger control by dynamically adjusting the target and weight. Specifically, in the projection mechanism, when the finger is close (distance < project_dist), the projection is activated, and the projection will fix the target vector length to a preset value, which helps to stabilize the grasp and precise control. The hierarchical weight system ensures that the coordination priority between fingers is higher, and the control strategy is dynamically adjusted according to the distance between fingers, providing more accurate control in fine operations and maintaining flexibility in large-scale movements. Finally, the Huber loss function is used to reduce the influence of outliers, and a regularization term is added to avoid sudden changes in joint angles, and dynamic weights are used to ensure smooth transitions. This design is particularly suitable for precise grasping tasks, finger coordination operations, dynamically adjusting grasping strength scenes, and continuous actions that require stable transitions. Compared with ordinary vector redirection, this adaptive dynamic adjustment mechanism can better handle the relationship between fingers and achieve more accurate and stable control.

[0112] In some embodiments, the inverse kinematics model includes a pseudo-inverse solution sub-model and a null space sub-model. The wrist joint pose data is calculated according to the preset inverse kinematics model to obtain the double-arm joint angle, including: pseudo-inverse calculation of the wrist joint pose data according to the pseudo-inverse solution sub-model to obtain pseudo-inverse solution data; and optimization of the pseudo-inverse solution data according to the null space sub-model to obtain the double-arm joint angle.

[0113] It needs to be further explained that, according to the pseudo-inverse solution sub-model, the wrist joint pose data is calculated by pseudo-inverse, and pseudo-inverse solution data is obtained. For a redundant robot arm (for example, a 7-DOF robot arm), there are infinitely many joint angle combinations that can reach the same end pose. The pseudo-inverse solution data itself provides a "minimum norm" solution (i.e., the solution with the smallest joint movement). However, the pseudo-inverse solution sub-model of PINK IK can use zero space projection to determine the priority of the primary task (end pose) and the secondary task through the task priority mechanism. The secondary target only takes effect in the zero space to "not interfere with the primary task, so as to optimize a secondary task without affecting the primary task (end pose).

[0114] Specifically, in the process of pseudo-inverse calculation of wrist joint pose data according to the pseudo-inverse solution sub-model, the pseudo-inverse solution of the redundant robot arm (such as 7-DOF) is provided with the minimum norm primary solution, and the pseudo-inverse solution data is obtained. On the basis of this pseudo-inverse solution data, secondary optimization can be introduced through zero space projection of the zero space sub-model without destroying the primary task.

[0115] In some embodiments, the pseudo-inverse solution data is optimized according to the zero space sub-model to obtain the joint angles of the dual-arm. It needs to be explained that the zero space sub-model refers to the part of the joint speed change that does not cause any movement of the end effector. The zero space sub-model of PINK IK allows another optimization target (such as joint limit obstacle avoidance, avoidance of singular points, maintenance of a comfortable posture, and gaze target, etc.) to be projected into this zero space. It needs to be further explained that the primary task is "the hand reaches a certain position", and the secondary task can be "the elbow of the robot arm is lifted as much as possible" or "all joints are as far away from their movement limits as possible". PINK IK can handle both tasks simultaneously without interfering with the completion of the primary task, and optimize the secondary task as much as possible.

[0116] In addition, since the zero space part of the zero space sub-model carries the secondary target (joint limit obstacle avoidance, avoidance of singular points, comfortable posture, elbow lifting, gaze / pointing, etc.), and can add time smoothing regularization, the solution is continuous and stable. In the scene where the robot adopts dual arms, the left and right arms can be independently solved for the primary task, and symmetric / collaborative constraints can be added in the zero space of the zero space sub-model respectively or coupled.

[0117] Among them, adding time smoothing regularization means that "distance from the previous time solution minimization" or "minimum joint speed / acceleration" regularization can be added to reduce jitter. More specific examples of secondary targets can be: joint centralization (away from soft / hard limits), elbow lifting / shoulder retraction, obstacle avoidance (human / device), orientation / gaze constraints.

[0118] In this embodiment, since it is a numerical method based on pseudo-inverse, PINK IK is solved by a small iteration step, and the resulting motion is usually continuous and very smooth, without abrupt jumps. When combined with reasonable secondary goals (such as joint velocity minimization), the resulting motion trajectory looks very natural and efficient, similar to human motion, and unlike analytical methods, PINK IK is a numerical method that does not depend on the specific configuration of the robot, as long as the Jacobian matrix of the robot can be calculated, the same algorithm can be applied to robots of various structures (robot arms, humanoid robots, snake arms, etc.), which makes it highly versatile and reusable, although the robustness of singular point processing is not as "strong" as DLS, but PINK IK can actively avoid singular points through its null space optimization. The "distance from singular points" can be put into the secondary task as an optimization index, so that the robot arm can automatically move away from singular configurations that may cause high-speed motion while completing the primary task. Therefore, the robot wrist pose is obtained by inverse kinematics. The inverse kinematics solving method used here ensures high precision, smooth trajectory, and robustness of end following.

[0119] In order to more easily understand the scheme of the present application, the specific process of transforming the operator wrist pose to the robot wrist pose in the actual application process is further described herein in combination with Figure 4 and Figure 5

[0120] First, the transformation from the operator wrist pose in the VR coordinate system (such as the right wrist pose in Figure 4 ) to the robot wrist joint pose in the robot coordinate system (such as the right link7 in Figure 5 ) mainly involves the following two aspects. On the one hand, there are differences in the definitions of the two coordinate systems base and the orientations of the coordinate axes representing the wrist pose. On the other hand, due to the differences in the configurations of the operator and the robot, such as different arm lengths and heights, there is a fixed offset between the operator's wrist to the VR base and the robot's wrist to the robot base.

[0121] Specifically, the right hand wrist pose in VR is represented as (obtained from the VR api), and the VR base link to the robot base link is represented as It should be noted that this first considers rotation and does not consider translation, assuming that the origins of the two bases coincide, and the rotation is obtained by defining the orientations of the two coordinate systems. Then, the operator's right hand wrist pose in the robot base link coordinate system is ​, a formulaic definition of the process of pose transformation according to the above defined expression is given as follows:

[0122] Robot_base_link T right_wrist = Robot_base_link T VR_base_link VR_base_link T right_wrist ;

[0123] It should be further explained that the pose transformation from the operator's right hand wrist pose in the base link coordinate system of the robot to the pose of the right hand wrist of the robot is represented as (mainly rotation, defined by the axis direction of the local coordinate system), and the pose of the right hand wrist right_link7 of the robot in the base link of the robot is obtained as A formulaic definition of the process of pose transformation according to the above defined expression is given as follows:

[0124] Robot_base_link T right_link7 = Robot_base_link T right_wrist VR_base_link T right_wrist ;

[0125] Finally, in combination with the actual application effect, a fixed offset offset is added to the solved pose to determine the final pose of the right hand wrist (left_link7) of the robot in the base coordinate system (base link coordinate system) of the robot. The solving process of the left hand wrist pose is consistent with that of the right hand, which will not be repeated here. It should be noted that the pose of the operator's wrist in the VR base in the VR field of view and the pose of the robot wrist relative to its own base coordinate system (robot base) have a fixed offset offset. Due to the differences in configuration between the operator and the robot, such as different arm lengths and heights, the wrist pose obtained by inverse solution cannot accurately match the actual work requirements, so a fixed offset offset needs to be added to the solved pose to determine the real pose data of the left / right hand pose.

[0126] In an exemplary embodiment, as shown in Figure 3 , the model training step includes steps 302 to 306. Among them:

[0127] Step 302, get the joint angle control effect of the robot.

[0128] The joint angle control effect can be a description statement or accurate angle control feedback data, without limitation.

[0129] In some embodiments, the joint angle control effect of the robot can be obtained by recording the joint angles of the arms and hands, and synchronously recording the camera images of the robot head.

[0130] In some embodiments, the joint angle control effect can be viewed in the simulation after being issued, and the recorded joint angle can also be played back to view the joint angle control effect of the robot in the simulation or the real machine.

[0131] In step 304, the operation trajectory generation is performed according to the joint angle control effect and the hand operation result, and a target operation trajectory is obtained.

[0132] In some embodiments, the operation trajectory generation is a process of trajectory recognition based on the recorded images.

[0133] In some embodiments, the target operation trajectory can be recognized and generated in the simulation environment based on the collected joint angle control effect and hand operation result by using the action recognition model.

[0134] In step 306, a training operation trajectory sample and a training operation trajectory sample image are generated based on the target operation trajectory, and are used for training the target visual language action model.

[0135] In some embodiments, the training operation trajectory sample and the training operation trajectory sample image are corresponding, and each sample corresponds to a group of sample images.

[0136] In some embodiments, the simulation and the real machine are synchronously recorded, the data quality is quickly checked in the simulation, and a large number of new operation trajectories are generated based on the training operation trajectory sample and the training operation trajectory sample image.

[0137] Specifically, in the original demonstration trajectory recording stage, not only the relative position and attitude data of the target object and the robot end effector are collected, but also the hand key vector information (inter-finger vector, wrist-to-finger vector) and the corresponding operation characteristic parameters (such as the grasping force threshold, the finger motion coordination coefficient) are synchronously associated. When the scene changes (the initial position of the object, the target position changes), first, the position and attitude of the original trajectory are mapped to the new scene through homogeneous coordinate transformation to form the initial new trajectory; then, based on the size and spatial position of the object in the new scene, the preset safety length threshold of the first-order projection and the coordination verification range of the second-order projection are dynamically adjusted, and the hand vector data in the initial new trajectory is subjected to secondary constraint; finally, according to the operation priority in the new scene (such as precise grasping, rapid movement), the weight distribution rule is adaptively adjusted (such as increasing the second-order projection weight to 500 for precise operation, and keeping the basic weight 1 for rapid movement), and the finally generated new trajectory not only retains the original operation mode (such as the grasping angle, the finger opening and closing sequence) and the motion characteristics (such as the speed curve, the acceleration threshold), but also adapts to the spatial configuration of the new scene through dynamic constraint and weight adaptation, and improves the stability and precision of the operation, so as to be used for training the target visual language action model.

[0138] In the embodiment, the robot head camera synchronously records visual image data such as the position change process of the double arms and the dexterous hand joint angles. In the data collection stage, the joint angle control effect issued in the simulation system can be directly viewed; after the camera collects the data, the recorded joint angle data can be called to perform offline playback verification on the simulation environment or the real robot platform, so as to comprehensively evaluate the actual effect of the joint angle control strategy. In addition, based on the collected real operation trajectory data, a plurality of new operation trajectories and their matched visual images can be further generated in the simulation environment to build a rich and high-quality data set for the training of the VLA model.

[0139] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include a plurality of steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0140] Based on the same inventive concept, the embodiments of the present application also provide a robot teleoperation device for implementing the robot teleoperation method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more robot teleoperation device embodiments provided below can refer to the limitations of the robot teleoperation method described above, which will not be repeated here.

[0141] In one exemplary embodiment, as shown in Figure 10 A robot teleoperation device is provided, comprising: an acquisition module 1001, a pose solving module 1002, a pose conversion module 1003, an inverse solving module 1004, and a teleoperation module 1005, wherein:

[0142] The acquisition module 1001 is configured to acquire hand pose data of a human hand key point;

[0143] The pose solving module 1002 is configured to perform pose solving on the hand pose data to obtain hand joint angles of a robot;

[0144] The pose conversion module 1003 is configured to perform pose conversion on wrist pose data of the hand pose data to obtain wrist joint pose data of a robot hand;

[0145] The inverse solving module 1004 is configured to perform solving on the wrist joint pose data according to a preset inverse kinematics model to obtain double-arm joint angles of the robot;

[0146] The teleoperation module 1005 is configured to perform teleoperation on the robot according to the hand joint angles and the double-arm joint angles to obtain a hand operation result of the robot.

[0147] In some embodiments, the pose solving module 1002 is further configured to perform vector calculation on the hand pose data to obtain hand vector data; perform vector processing on the hand vector data to obtain target vector data; perform optimization target calculation on the target vector data to obtain target vector optimization data, and perform joint angle optimization processing on the target vector optimization data to obtain the hand joint angles.

[0148] In some embodiments, the pose solving module 1002 is further configured to perform vector classification on the hand vector data to obtain inter-finger vector data and wrist-to-finger vector data; perform vector length distance determination on the inter-finger vector data to obtain a length distance determination result; perform projection on the inter-finger vector data according to the distance determination result to obtain projected vector data; and perform weight distribution on the projected vector data and the wrist-to-finger vector data to obtain the target vector data.

[0149] In some embodiments, the pose solving module 1002 is further configured to perform robot state calculation on the target vector data to obtain robot state data, perform forward kinematics calculation on the robot state data to obtain current vector data, perform loss function optimization calculation on the current vector data to obtain target vector optimization data, perform gradient calculation on the target vector optimization data to obtain target gradient data, and perform joint angle optimization processing on the target gradient data to obtain hand joint angles.

[0150] In some embodiments, the inverse kinematics model includes a pseudo-inverse solution sub-model and a null space sub-model, and the inverse solving module 1004 is further configured to perform pseudo-inverse calculation on the wrist joint pose data according to the pseudo-inverse solution sub-model to obtain pseudo-inverse solution data, and perform optimization on the pseudo-inverse solution data according to the null space sub-model to obtain dual-arm joint angles.

[0151] In some embodiments, the device further includes a training module configured to obtain joint angle control effects of the robot, generate target operation trajectories based on the joint angle control effects and hand operation results, and generate training operation trajectory samples and training operation trajectory image based on the target operation trajectories, so as to train the target visual language action model.

[0152] In the above robot teleoperation device, the hand pose data of the human hand key points is obtained through the VR headset worn by the operator, the hand joint angles of the robot are obtained by performing pose solving on the hand pose data, the wrist joint pose data of the robot hand is obtained by performing pose conversion on the wrist pose data of the hand pose data, the dual-arm joint angles of the robot are obtained by performing solving on the wrist joint pose data according to the preset inverse kinematics model, and the hand operation result of the robot is obtained by teleoperating the robot according to the hand joint angles and the dual-arm joint angles. Therefore, the hand pose data of the human hand key points is obtained through the VR headset worn by the operator, the hand joint angles of the robot are obtained by performing pose solving on the hand pose data, the wrist joint pose data is obtained by performing pose conversion on the wrist pose data of the hand pose data, and the dual-arm joint angles are further obtained by performing solving on the wrist joint pose data according to the preset inverse kinematics model. The movement of the robot hand and the dual arm is controlled through the obtained joint angles, the teleoperation of the robot is realized, and the hand operation result is obtained. According to the hand operation result, the playback viewing in simulation and the training of the visual language action model (VLA) model can also be performed.

[0153] Each of the above modules in the robot teleoperation device can be realized by software, hardware, and a combination thereof in whole or in part. Each of the above modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to each of the above modules.

[0154] In an example embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 11 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store an inverse kinematics model. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a robot teleoperation method.

[0155] Those skilled in the art can understand that Figure 11 The structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0156] In an example embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the robot teleoperation method described above.

[0157] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the robot teleoperation method described above.

[0158] In an embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps of the robot teleoperation method described above.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant regulations.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0161] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0162] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method of robot teleoperation, characterized by, The method comprises: acquiring hand pose data of human hand key points; solving the hand pose data to obtain hand joint angles of a robot; converting wrist pose data of the hand pose data to obtain wrist joint pose data of the robot hand; solving the wrist joint pose data according to a preset inverse kinematics model to obtain double-arm joint angles of the robot; teleoperating the robot according to the hand joint angles and the double-arm joint angles to obtain hand operation results of the robot.

2. The method of claim 1, wherein, The solving of the hand pose data to obtain hand joint angles of a robot comprises: performing vector calculation according to the hand pose data to obtain hand vector data; performing vector processing on the hand vector data to obtain target vector data; performing optimization target calculation on the target vector data to obtain target vector optimization data, and performing joint angle optimization processing according to the target vector optimization data to obtain the hand joint angles.

3. The method of claim 2, wherein, The vector processing on the hand vector data to obtain target vector data comprises: performing vector classification on the hand vector data to obtain inter-finger vector data and wrist-to-finger vector data; performing vector length distance determination on the inter-finger vector data to obtain length distance determination results; projecting the inter-finger vector data according to the distance determination results to obtain projected vector data; performing weight distribution according to the projected vector data and the wrist-to-finger vector data to obtain the target vector data.

4. The method of claim 2, wherein, The optimization target calculation on the target vector data to obtain target vector optimization data, and the joint angle optimization processing according to the target vector optimization data to obtain the hand joint angles, comprise: performing robot state calculation on the target vector data to obtain robot state data; performing forward kinematics calculation on the robot state data to obtain current vector data; performing loss function optimization calculation according to the current vector data to obtain the target vector optimization data; performing gradient calculation according to the target vector optimization data to obtain target gradient data; performing joint angle optimization processing according to the target gradient data to obtain the hand joint angles.

5. The method of claim 1, wherein, The inverse kinematics model comprises a pseudo-inverse solution sub-model and a null space sub-model, and the solving of the wrist joint pose data according to a preset inverse kinematics model to obtain double-arm joint angles of the robot comprises: performing pseudo-inverse calculation on the wrist joint pose data according to the pseudo-inverse solution sub-model to obtain pseudo-inverse solution data; optimizing the pseudo-inverse solution data according to the null space sub-model to obtain the double-arm joint angles.

6. The method of claim 1, wherein, The method further comprises: acquiring joint angle control effects of the robot; generating operation trajectories according to the joint angle control effects and the hand operation results to obtain target operation trajectories; generating training operation trajectory samples and training operation trajectory image samples based on the target operation trajectories, for training of a target visual language action model.

7. A robotic teleoperation device, characterized by The device comprises: an acquisition module configured to acquire hand pose data of human hand key points; A pose solving module is configured to solve a hand pose of the hand pose data to obtain a joint angle of a hand of the robot. A pose conversion module is configured to convert a wrist pose of the hand pose data to obtain wrist joint pose data of the hand of the robot. An inverse solving module is configured to solve the wrist joint pose data according to a preset inverse kinematics model to obtain a joint angle of a double arm of the robot. A teleoperation module is configured to teleoperate the robot according to the joint angle of the hand and the joint angle of the double arm to obtain a hand operation result of the robot.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images