Upper limb motion imitation method and device, robot and computer program product

Abstract

The application is suitable for the field of robot technology, and particularly relates to an upper limb action imitation method and device, a robot and a computer program product. In the method, the robot can obtain a first pose corresponding to an upper limb key point of an imitated object, and can convert the first pose into a second pose corresponding to an upper limb key point of the robot. Based on the second pose corresponding to the upper limb key point of the robot, the target angle corresponding to each joint of the upper limb of the robot is accurately determined through hierarchical quadratic programming, so that the robot can be accurately controlled to perform action imitation according to the target angle corresponding to each joint of the upper limb of the robot, the similarity and stability of robot action imitation can be improved, and the imitation effect of the robot can be improved.

Description

Technical Field

[0001] This application belongs to the field of robotics technology, and in particular relates to a method, device, robot, and computer program product for imitating upper limb movements. Background Technology

[0002] Humanoid robots (or humanoid robots) resemble humans in appearance and possess multi-degree-of-freedom robotic arms, enabling them to move in a human-like manner, that is, to a certain extent, mimicking human movement. Currently, humanoid robots are generally created by acquiring motion images and then imitating those images. This method of motion image-based imitation suffers from low similarity and poor imitation quality. Summary of the Invention

[0003] This application provides an upper limb movement imitation method, device, robot, and computer program product, which can improve the similarity of upper limb movement imitation by the robot and enhance the imitation effect of the robot.

[0004] In a first aspect, embodiments of this application provide a method for imitating upper limb movements, applied to a robot, the method comprising:

[0005] Obtain the first pose corresponding to the key points of the upper limb of the object being imitated;

[0006] Based on the first pose, determine the second pose corresponding to the key points of the robot's upper limb;

[0007] Based on the second pose, the target angles corresponding to each joint of the robot's upper limb are determined through hierarchical quadratic programming;

[0008] The movement of the robot's upper limb is controlled according to the target angles corresponding to each joint of the robot's upper limb.

[0009] In the above-mentioned upper limb movement imitation method, the robot can obtain the first pose corresponding to the key points of the upper limb of the object being imitated, and can convert the first pose into the second pose corresponding to the key points of the robot's upper limb. Based on the second pose corresponding to the key points of the robot's upper limb, the target angles corresponding to each joint of the robot's upper limb can be accurately determined through hierarchical quadratic programming. Thus, the robot can be accurately controlled to imitate movements according to the target angles corresponding to each joint of the robot's upper limb, which can improve the similarity and stability of the robot's movement imitation and improve the imitation effect of the robot.

[0010] In some embodiments, determining the target angles corresponding to each joint of the robot's upper limb through hierarchical quadratic programming based on the second pose includes:

[0011] Based on the second pose, the initial angles of each joint of the robot's upper limb are determined through the first secondary planning.

[0012] Based on the second pose, determine the arm angle corresponding to the robot, and determine the angular velocity corresponding to the arm angle;

[0013] Based on the angular velocity corresponding to the arm angle, the initial angles of each joint of the robot's upper limb are updated through a second quadratic programming process to obtain the target angles of each joint of the robot's upper limb.

[0014] In one possible implementation, the second pose includes the target position and target orientation corresponding to the robot wrist;

[0015] The step of determining the initial angles of each joint of the robot's upper limb based on the second pose through a first quadratic programming process includes:

[0016] Based on the target position and target posture corresponding to the robot's wrist, the initial angles corresponding to each joint of the robot's upper limb are determined through the first secondary planning.

[0017] In one possible implementation, the second pose further includes a target position corresponding to the robot's shoulder and a target position corresponding to the robot's elbow;

[0018] Determining the corresponding arm angle of the robot based on the second pose includes:

[0019] The first plane corresponding to the robot is determined based on the target position corresponding to the robot's shoulder, the target position corresponding to the robot's elbow, and the target position corresponding to the robot's wrist.

[0020] The arm angle of the robot is determined based on the reference plane and the first plane corresponding to the robot. The arm angle of the robot is the angle between the reference plane and the first plane. The reference plane is determined based on the initial position of the robot's shoulder, the initial position of the robot's elbow, and the initial position of the robot's wrist.

[0021] In some embodiments, the method further includes:

[0022] Obtain the angles corresponding to each finger of the object being imitated;

[0023] The angles corresponding to each finger of the robot are determined based on the angles corresponding to each finger of the object being imitated.

[0024] The movement of each finger of the robot is controlled according to the angle corresponding to each finger.

[0025] In some embodiments, determining the second pose corresponding to the upper limb key points of the robot based on the first pose includes:

[0026] Based on the first pose, determine the arm size of the object being imitated;

[0027] Obtain the arm dimensions of the robot;

[0028] Based on the arm size of the object being imitated, the arm size of the robot, and the first pose, the second pose corresponding to the key points of the robot's upper limb is determined.

[0029] In one possible implementation, the arm size of the object being imitated includes a first size and a second size of the object being imitated, the first size being the size from the shoulder to the elbow of the object being imitated, and the second size being the size from the shoulder to the wrist of the object being imitated; the arm size of the robot includes a third size and a fourth size, the third size being the size from the shoulder to the elbow of the robot, and the fourth size being the size from the shoulder to the wrist of the robot.

[0030] The step of determining the second pose corresponding to the upper limb key points of the robot based on the arm size of the imitated object, the arm size of the robot, and the first pose includes:

[0031] Based on the first dimension, the third dimension, and the first pose, determine the second pose corresponding to the robot's elbow;

[0032] Based on the second dimension, the fourth dimension, and the first pose, the second pose corresponding to the robot's wrist is determined.

[0033] Secondly, embodiments of this application provide an upper limb movement mimicry device for use in robots, the device comprising:

[0034] The first pose acquisition module is used to acquire the first pose corresponding to the key points of the upper limb of the object being imitated;

[0035] The second pose determination module is used to determine the second pose corresponding to the upper limb key points of the robot based on the first pose.

[0036] The target angle determination module is used to determine the target angles corresponding to each joint of the robot's upper limb through hierarchical quadratic programming based on the second pose.

[0037] The motion control module is used to control the movement of the robot's upper limbs according to the target angles corresponding to each joint of the robot's upper limbs.

[0038] Thirdly, embodiments of this application provide a robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it enables the robot to perform the upper limb movement imitation method described in any of the first aspects above.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to implement the upper limb movement imitation method described in any one of the first aspects above.

[0040] Fifthly, embodiments of this application provide a computer program product, which stores a computer program. When the computer program is executed by a computer, it causes the computer to implement the upper limb movement imitation method described in any one of the first aspects above.

[0041] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the upper limb movement imitation method provided in the embodiments of this application. Figure 1 ;

[0044] Figure 2 This is an example diagram of the motion capture device for acquiring pose according to an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the application scenario provided in the embodiments of this application. Figure 1 ;

[0046] Figure 4 Flowchart of the upper limb movement imitation method provided in this application embodiment Figure 2 ;

[0047] Figure 5 and Figure 6 This is a schematic diagram of the application scenario provided in the embodiments of this application. Figure 2 ;

[0048] Figure 7This is a schematic diagram of the upper limb movement mimicry device provided in the embodiments of this application;

[0049] Figure 8 This is a schematic diagram of the robot provided in the embodiments of this application. Detailed Implementation

[0050] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0051] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0052] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0054] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0056] Humanoid robots, resembling humans in appearance, possess multi-degree-of-freedom robotic arms, such as a seven-degree-of-freedom arm, enabling them to achieve anthropomorphic movements—that is, they can mimic human movement patterns to a certain extent. To increase the complexity and human-likeness of multi-degree-of-freedom robot movements, enhance their interactivity, and reduce the difficulty of motion programming, motion teaching for robots has been widely developed. For example, robots can learn by imitation to mimic human movements. During this learning process, it is necessary to coordinate the movements of the robot's joints to maintain balance and ensure smooth motion trajectories.

[0057] Humanoid robots typically mimic actions by acquiring motion images of the object being imitated. Based on these images, the robot determines the angles of the corresponding joints and then performs the mimicry. However, this method of mimicking actions based on motion images suffers from low similarity and poor imitation quality.

[0058] To address the aforementioned problems, this application provides a method, apparatus, robot, and computer program product for imitating upper limb movements. In this method, the robot can acquire the first pose corresponding to the key points of the upper limb of the object being imitated, and determine the second pose corresponding to the key points of the robot's upper limb based on the first pose. After determining the second pose, the robot can determine the target angles corresponding to each joint of the robot's upper limb through hierarchical quadratic programming, and control the robot's upper limb movement based on the target angles corresponding to each joint of the robot's upper limb. In other words, this application can acquire the first pose corresponding to the key points of the upper limb of the object being imitated, and convert the first pose into the second pose corresponding to the key points of the robot's upper limb. Based on the second pose corresponding to the key points of the robot's upper limb, the target angles corresponding to each joint of the robot's upper limb are accurately determined through hierarchical quadratic programming. Therefore, the robot can be accurately controlled to imitate movements based on the target angles corresponding to each joint of the robot's upper limb, improving the similarity and stability of the robot's movement imitation, enhancing the imitation effect, and possessing strong usability and practicality.

[0059] The upper limb movement imitation method provided in this application embodiment will be described in detail below with reference to the accompanying drawings and specific application scenarios.

[0060] Please see Figure 1 , Figure 1 This application illustrates a flowchart of the upper limb movement imitation method provided in an embodiment. Figure 1 This method can be applied to robots. For example, it can be applied to humanoid robots with upper limbs (such as arms). Figure 1 As shown, the method may include:

[0061] S101, The robot acquires the first pose corresponding to the key points of the upper limb of the object being imitated.

[0062] It should be noted that the upper limb movement imitation method provided in this application can be used to imitate the single-arm movement of a target object, or it can be used to imitate the double-arm movement of a target object. Specifically, when imitating the single-arm movement of a target object, it can be achieved using a robot's single arm. For example, the robot's left arm can imitate the movement of the target object's left arm, or the robot's right arm can imitate the movement of the target object's right arm. When imitating the double-arm movement of a target object, it can be achieved using both arms of a robot; that is, the robot's left arm can imitate the movement of the target object's left arm, and the robot's right arm can imitate the movement of the target object's right arm.

[0063] In this embodiment, the key points of the upper limb can be specifically determined according to the actual scenario. For example, the key points of the upper limb can be determined to include key points such as the shoulder, elbow, and wrist, depending on the actual scenario. It should be understood that both the first pose and the subsequent second pose can include position and posture.

[0064] The following will use key points of the upper limbs, including the shoulder, elbow, and wrist, as examples for illustration.

[0065] In some embodiments, the robot can acquire the first pose corresponding to the key points of the upper limbs of the imitated object using a motion capture device. The motion capture device may include sensors such as an inertial measurement unit (IMU). The imitated object can wear the motion capture device to perform actions. When the imitated object wears the motion capture device and performs actions, the sensors such as the IMU in the motion capture device can acquire the position and posture of various body parts of the imitated object, and determine the first pose corresponding to the key points of the upper limbs based on the position and posture of each body part. For example, determining the first pose corresponding to the shoulder, the elbow, and the wrist.

[0066] It should be noted that the specific type of motion capture device is not limited in the embodiments of this application, and can be determined according to the actual scenario. The motion capture device can be any existing type of motion capture device.

[0067] In one embodiment, the first pose corresponding to the upper limb keypoint of the imitated object can be the pose of the upper limb keypoint relative to the shoulder. For example, the first pose corresponding to the elbow can be the pose of the elbow relative to the shoulder, and the first pose corresponding to the wrist can be the pose of the wrist relative to the shoulder.

[0068] For example, after sensors such as IMU in a motion capture device acquire the first pose (e.g., position A1 and posture B1) of the shoulder of the object being imitated, the position (e.g., position A2) and posture (e.g., posture B2) of the elbow of the object being imitated, and the position (e.g., position A3) and posture (e.g., posture B3) of the wrist of the object being imitated, the motion capture device can subtract the position A2 of the elbow of the object being imitated from the position A1 of the shoulder of the object being imitated, and can subtract the posture B2 of the elbow of the object being imitated from the posture B1 of the shoulder of the object being imitated, to obtain the position and posture of the elbow of the object being imitated relative to the shoulder of the object being imitated (i.e., the first pose of the elbow of the object being imitated).

[0069] Similarly, the motion capture device can subtract the position A3 corresponding to the wrist of the object being imitated from the position A1 corresponding to the shoulder of the object being imitated, and can subtract the pose B3 corresponding to the wrist of the object being imitated from the pose B1 corresponding to the shoulder of the object being imitated, to obtain the position and pose of the wrist of the object being imitated relative to the shoulder of the object being imitated (i.e., the first pose corresponding to the wrist of the object being imitated).

[0070] For example, please see Figure 2 , Figure 2 An example diagram of the motion capture device for obtaining pose provided in an embodiment of this application is shown.

[0071] like Figure 2 As shown, the motion capture device can be equipped with IMUs (Induction Units) on its shoulders (e.g., left and right shoulders), upper arms (e.g., left and right upper arms), forearms (e.g., left and right forearms), and hands (e.g., left and right hands). When the subject wearing the motion capture device performs actions, the IMU on the shoulder can acquire the position and posture of the subject's shoulders in real time; the IMU on the upper arm can acquire the position and posture of the subject's upper arm in real time; the IMU on the forearm can acquire the position and posture of the subject's forearm in real time; and the IMU on the hand can acquire the position and posture of the subject's hand in real time.

[0072] Subsequently, the motion capture device can determine the first pose corresponding to the shoulder, the first pose corresponding to the elbow, and the first pose corresponding to the wrist of the object being imitated, based on the position and posture of the shoulder, the upper arm, the forearm, and the hand. The specific method by which the motion capture device determines the first pose of the shoulder, the elbow, and the wrist can be determined according to the actual scenario, and this embodiment does not impose any limitations on this.

[0073] It should be understood that, such as Figure 2 As shown, the body parts of the object being imitated may also include the head, neck, T8 (chest), T12 (12th thoracic vertebra), L3 (3rd lumbar vertebra), L5 (5th lumbar vertebra), pelvis, upper left lower limb, lower left lower limb, left foot, left toes, upper right lower limb, lower right lower limb, right foot, and right toes. The motion capture device may also install sensors such as IMUs in one or more of these body parts to collect data.

[0074] It should be noted that the above-described method of using sensors such as IMUs in motion capture devices to acquire the position and posture of the shoulder, upper arm, forearm, and hand of the object being imitated, and then using the motion capture device to determine the first pose of the shoulder, elbow, and wrist of the object being imitated, is merely an illustrative explanation and should not be construed as a limitation on the embodiments of this application. In the embodiments of this application, other devices (such as robots) can also determine the first pose of the shoulder, elbow, and wrist of the object being imitated based on the position and posture of the shoulder, upper arm, forearm, and hand acquired by the motion capture device.

[0075] It should be understood that the above-described method of acquiring pose by having the subject wear a motion capture device is merely illustrative and should not be construed as a limitation on the embodiments of this application. In the embodiments of this application, pose can also be acquired in other ways. For example, IMUs or other sensors can be directly attached to the shoulders, elbows, and wrists of the subject being imitated to directly acquire the first pose corresponding to the shoulder, the elbow, and the wrist of the subject being imitated.

[0076] In other embodiments, the motion capture device may also include a glove. The glove may also be equipped with sensors such as an IMU. After the object being imitated puts on the glove, the sensors in the glove can acquire the angles corresponding to each finger of the object being imitated, such as the angles corresponding to each finger joint (e.g., the angles of bending and / or adduction), allowing the robot to mimic the hand movements of the object being imitated based on the angles corresponding to each finger joint.

[0077] S102. Based on the first pose, the robot determines the second pose corresponding to the key points of the robot's upper limbs.

[0078] It should be noted that since the size of the arm of the object being imitated is generally different from that of the robot's arm, the first pose corresponding to the upper limb key points of the object being imitated cannot be directly assigned to the upper limb key points of the robot. It is necessary to map the first pose corresponding to the upper limb key points of the object being imitated to the upper limb key points of the robot. That is, it is necessary to convert the first pose corresponding to the upper limb key points of the object being imitated into the second pose corresponding to the upper limb key points of the robot. This ensures that the action imitated by the robot is within the robot's arm operation space, that is, within the reachable range of the robot's arm. This ensures the reachability of the robot's action imitation and guarantees the consistency between the action imitated by the robot and the action performed by the object being imitated.

[0079] In some embodiments, after obtaining the first pose corresponding to the upper limb keypoints of the imitated object, the robot can determine the arm size of the imitated object based on the first pose corresponding to the upper limb keypoints. Additionally, the robot can also obtain its own arm size. Subsequently, the robot can perform a proportional mapping between its own arm size and the arm size of the imitated object to map the first pose corresponding to the upper limb keypoints of the imitated object to the robot's upper limb keypoints, thus determining the second pose corresponding to the robot's upper limb keypoints.

[0080] It should be noted that the arm dimensions of the imitated object can include the dimensions from the shoulder to the elbow (e.g., referred to as the first dimension) and the dimensions from the shoulder to the wrist (e.g., referred to as the second dimension). Similarly, the arm dimensions of the robot include the dimensions from the shoulder to the elbow (e.g., referred to as the third dimension) and the dimensions from the shoulder to the wrist (e.g., referred to as the fourth dimension).

[0081] In one embodiment, dimension can refer to length. Specifically, the first dimension can refer to the length from the shoulder to the elbow of the object being imitated. The second dimension can refer to the length from the shoulder to the wrist of the object being imitated. The third dimension can refer to the length from the shoulder to the elbow of the robot. The fourth dimension can refer to the length from the shoulder to the wrist of the robot.

[0082] For example, the robot can determine the first dimension of the imitated object based on the position of the shoulder (e.g., the position of the shoulder in the first pose) and the position of the elbow (e.g., the position of the elbow in the first pose). The robot can determine the second dimension of the imitated object based on the position of the shoulder and the position of the wrist (e.g., the position of the wrist in the first pose).

[0083] In one possible implementation, the robot can determine the second pose of the robot's elbow based on the first dimension of the object being imitated, the third dimension of the robot, and the first pose of the key points of the upper limb of the object being imitated (e.g., the elbow).

[0084] Specifically, the robot can determine the ratio (e.g., ratio A) between the first dimension corresponding to the imitated object and the third dimension corresponding to the robot. Based on ratio A and the position of the elbow corresponding to the imitated object (i.e., the position in the first pose of the imitated object's elbow), the robot can determine the position of its elbow. It should be understood that the pose of the robot's elbow is the same as the pose of the imitated object's elbow. Therefore, the robot can determine the second pose of its elbow based on the position of its elbow and the pose of the imitated object's elbow. The pose in the second pose of the robot's elbow is the same as the pose in the first pose of the imitated object's elbow. In other words, the pose in the first pose of the imitated object's elbow can be used as the pose in the second pose of the robot's elbow.

[0085] In another possible implementation, the robot can determine the second pose of the robot's wrist based on the second dimension of the object being imitated, the fourth dimension of the robot, and the first pose of the key points of the upper limb (e.g., the wrist) of the object being imitated.

[0086] Specifically, the robot can determine the ratio (e.g., ratio B) between the second dimension corresponding to the imitated object and the fourth dimension corresponding to the robot. Based on ratio B and the position of the imitated object's wrist (i.e., the position in the first pose corresponding to the imitated object's wrist), the robot can determine the position of its own wrist. It should be understood that the robot's wrist pose is the same as the imitated object's wrist pose. Therefore, the robot can determine its own wrist's second pose based on its wrist position and the imitated object's wrist pose. The pose in the robot's second wrist pose is identical to the pose in the imitated object's first wrist pose. In other words, the pose in the first pose of the imitated object's wrist can be used as the pose in the robot's second wrist pose.

[0087] It should be understood that the first pose corresponding to the elbow and the first pose corresponding to the wrist of the imitated object are poses relative to the shoulder of the imitated object. Therefore, the second pose corresponding to the elbow and the second pose corresponding to the shoulder of the robot can also be poses relative to the shoulder of the robot.

[0088] In some embodiments, after obtaining the first pose corresponding to the key points of the upper limb of the object being imitated, the first pose can be filtered to reduce the influence of environmental and / or sensor noise and other factors, making the first pose smoother, which can ensure the smoothness and stability of the robot's motion imitation.

[0089] It should be noted that the specific filtering method described in this application is not limited and can be determined according to the actual scenario. For example, any existing filtering algorithm can be used to filter the first pose corresponding to the key points of the upper limb of the imitated object.

[0090] S103. Based on the second pose, the robot determines the target angles corresponding to each joint of the robot's upper limb through hierarchical secondary programming.

[0091] It should be understood that, given the end-effector pose, the angles corresponding to each joint can be determined through inverse kinematics, for example, through iterative or analytical solutions. However, both iterative and analytical solutions suffer from problems such as long computation time and large computational load.

[0092] In this embodiment, the inverse problem can be transformed into a quadratic optimization problem. After determining the second pose corresponding to the key points of the robot's upper limb, hierarchical quadratic programming (HQP) can be used to determine the angles (e.g., target angles) corresponding to each joint of the robot's upper limb. Specifically, after solving for the angles corresponding to each joint, a second QP optimization can be performed, placing low-priority subtasks within the null space of high-priority tasks. This ensures that, while satisfying the angle constraints corresponding to each joint, the robot's wrist can follow a given wrist trajectory (i.e., the wrist trajectory corresponding to the wrist of the imitated object). In other words, within the subspace of the solved angles corresponding to each joint, the angles corresponding to each joint follow a given elbow trajectory. This human-like optimization improves the similarity and stability of the robot's motion imitation, enhancing the imitation effect.

[0093] In some embodiments, after obtaining the second pose corresponding to the robot's upper limb keypoints, such as the second pose (e.g., target position and target orientation) corresponding to the robot's wrist, the robot can input the target position and target orientation corresponding to the robot's wrist into the inverse kinematics algorithm to obtain the angles (e.g., initial angles) corresponding to each joint of the robot's upper limb through the first QP solution. Additionally, the robot can determine the corresponding arm angle based on the second pose corresponding to the robot's upper limb keypoints. The corresponding arm angle may include the position information of the robot's elbow. Subsequently, the robot can determine the angular velocity corresponding to the arm angle and input it as the null velocity of the third axis into the second QP solution to update the initial angles corresponding to each joint of the robot's upper limb, thereby obtaining the target angles corresponding to each joint of the robot's upper limb. After two QP solutions, the elbow can also follow the position of the real elbow while satisfying the tracking of the wrist's position and orientation, thus achieving true humanoid motion and improving the similarity and stability of the robot's action imitation.

[0094] In one embodiment, the corresponding arm angle of the robot can be determined based on the pose of the robot's shoulder, the pose of the robot's elbow, and the pose of the robot's wrist. For example, the corresponding arm angle of the robot can be determined based on its position in the pose of the robot's shoulder, the pose of the robot's elbow, and the pose of the robot's wrist.

[0095] For example, the robot can determine the plane (e.g., a first plane) formed by its shoulder, wrist, and elbow based on the target positions corresponding to its shoulder (i.e., its position in the second pose), its elbow (i.e., its position in the second pose), and its wrist (i.e., its position in the second pose). Additionally, the robot can determine the plane (e.g., a reference plane) formed by its shoulder, wrist, and elbow based on the initial positions of its shoulder, elbow, and wrist when in its initial state. Subsequently, the robot can determine its corresponding arm angle based on the first plane and the reference plane. The corresponding arm angle can be the angle between the first plane and the reference plane.

[0096] It should be noted that the embodiments of this application do not limit the specific method for determining the angle between the first plane and the reference plane, and can be determined according to the actual scenario. For example, the angle between the first plane and the reference plane can be determined based on the perpendicular vector of the first plane and the perpendicular vector of the reference plane.

[0097] In one embodiment, after determining the corresponding arm angle of the robot, the robot can determine the angular velocity corresponding to the arm angle using a differential method. It should be understood that the method for determining the angular velocity corresponding to the arm angle using a differential method can refer to relevant existing technologies, and this application embodiment does not limit this approach.

[0098] S104. The robot controls the movement of its upper limbs according to the target angles corresponding to the joints of each joint.

[0099] In this embodiment of the application, after determining the target angles corresponding to each joint of the robot's upper limb, the robot can control each joint of the robot's upper limb according to the target angles corresponding to each joint of the robot's upper limb, thereby controlling the movement of the robot's upper limb and realizing the imitation of the action of the object being imitated.

[0100] In some embodiments, the robot can also mimic the movements of the fingers of the object being imitated. For example, it can mimic actions such as grasping or making gestures.

[0101] For example, the robot can acquire the angles corresponding to each finger of the object being imitated, such as the angles corresponding to each finger joint. This can be achieved, for instance, by using a glove equipped with sensors such as an IMU to acquire the angles corresponding to each finger joint of the object being imitated. After acquiring these angles, the robot can determine the angles corresponding to each of its own fingers based on the angles of the object's finger joints. Subsequently, the robot can control the movement of its own fingers based on these angles to mimic the finger movements of the object being imitated.

[0102] For example, when the robot knows the finger joints of each of its fingers, for each finger joint (e.g., finger joint A of the left index finger), the robot can directly determine the angle corresponding to finger joint A of the left index finger of the object being imitated as the angle corresponding to the same finger joint (e.g., finger joint A) of the robot's left index finger. This allows the robot to directly map the movements of each finger of the object being imitated to each finger of the robot, thus enabling the robot to imitate the finger movements of the object being imitated.

[0103] For example, when the robot doesn't know the joints of its own fingers, for each finger (e.g., the left thumb), the robot can obtain the size (or length) of the target's left thumb and the position of each joint of the target's left thumb. Additionally, the robot can obtain the size of its own left thumb. Then, the robot can determine the ratio (e.g., ratio C) between the size of the target's left thumb and the robot's own left thumb. Based on ratio C and the positions of the target's left thumb joints, the robot can determine the positions of its own left thumb joints. After determining the positions of the robot's left thumb joints, the robot can directly assign the angles of the target's left thumb joints to its own left thumb joints. Based on these angles, the robot can control the movement of its own fingers, thus mimicking the finger movements of the target.

[0104] Please see Figure 3 , Figure 3 This application illustrates an application scenario provided by an embodiment of this application. Figure 1 .

[0105] For example, in a specific scenario, the robot can control its fingers to perform actions based on the movements of the fingers of the object being imitated. Figure 3 (a) to Figure 3 The action shown in (b) is shown in the image.

[0106] It's important to note that a robot's imitation of an object's movements is generally a dynamic process. That is, the robot mimics the movements based on multiple frames of the object's actions. For each frame of the object's movements, the robot can acquire the first pose corresponding to the key points of the object's upper limbs. Based on this first pose, the robot can determine the target angles for each joint of its upper limbs. By controlling these target angles, the robot can then mimic the movements of each frame of the object's actions, thus achieving dynamic imitation of the object's movements.

[0107] In some embodiments, to reduce the problem of jitter during control, the present application embodiments may employ an incremental method of position and attitude to determine the first pose corresponding to the key points of the upper limb of the imitated object, thereby determining the target angles corresponding to each joint of the robot's upper limb.

[0108] For example, a robot can acquire the first pose corresponding to the upper limb key points in the first frame of the imitated object's motion. Based on this first pose, the robot can determine the target angles of each joint of the upper limb when imitating the first frame of the motion. For the second frame of the imitated object's motion, the robot can acquire the incremental changes in pose corresponding to the upper limb key points in both the first and second frames. These incremental changes can be superimposed on the first pose corresponding to the upper limb key points in the first frame to obtain the first pose corresponding to the upper limb key points in the second frame. Therefore, based on this first pose, the robot can determine the target angles of each joint of the upper limb when imitating the second frame.

[0109] Similarly, the first pose corresponding to the upper limb key points in the third or fourth frame of the imitated object's action can also be determined through incremental changes.

[0110] The above description is used to illustrate the upper limb movement imitation method provided in the embodiments of this application.

[0111] Please see Figure 4 , Figure 4 This application illustrates a flowchart of the upper limb movement imitation method provided in an embodiment. Figure 2 .

[0112] like Figure 4 As shown, the upper limb movement imitation method provided in this application embodiment may include a motion capture data acquisition process, a data post-processing process, and a control optimization process.

[0113] The subject being imitated can wear motion capture equipment to perform actions. During the motion capture data acquisition process, sensors such as the IMU in the motion capture equipment can collect data. That is, when the subject is wearing the motion capture equipment and performing actions, the motion capture equipment can acquire the position and posture of various body parts of the subject. Subsequently, the motion capture equipment can perform first pose determination, that is, based on the position and posture acquired by the IMU and other sensors, the motion capture equipment can determine the first pose corresponding to the key points of the subject's upper limbs.

[0114] During data post-processing, the robot can perform pose mapping and determine the real-time trajectory corresponding to the upper limb key points. Specifically, the robot can map the first pose corresponding to the upper limb key points of the imitated object to the robot's own upper limb key points. Based on the first pose of the imitated object's upper limb key points, the robot can determine the second pose corresponding to the robot's upper limb key points, and based on the second pose, determine the real-time trajectory corresponding to the robot's upper limb key points.

[0115] During control optimization, after acquiring the real-time trajectories corresponding to the key points of the robot's upper limbs, the robot can perform Hierarchical Quadratic Programming (HQP) solutions. That is, the robot can determine the target angles for each joint of the upper limb based on the real-time trajectories corresponding to the key points through hierarchical quadratic programming. Subsequently, the robot can control each joint of the upper limb based on the target angles to achieve the imitation of the actions of the target object.

[0116] Please see Figure 5 and Figure 6 , Figure 5 and Figure 6 This application illustrates an application scenario provided by an embodiment of this application. Figure 2 .

[0117] For example, in a specific scenario, the robot can control its upper limbs to perform actions based on the actions of the object being imitated. Figure 5 (a) to Figure 5 The motion shown in (b) is shown in the figure.

[0118] For example, in another specific scenario, the robot can control its upper limbs to perform actions based on the actions of the object being imitated. Figure 5 (c) to Figure 5 The motion shown in (d) is shown in the figure.

[0119] For example, in another specific scenario, the robot can control its upper limbs to perform actions based on the actions of the object being imitated. Figure 6 (a) to Figure 6 The motion shown in (c) means that the robot can make its elbow follow the position of the elbow of the object being imitated, while ensuring that the position and posture of the robot's end (i.e., wrist) are constant.

[0120] It should be noted that the upper limb movement imitation method provided in the above-described embodiments of this application, applied to robots, is merely illustrative and should not be construed as a limitation of the embodiments of this application. It should be understood that the upper limb movement imitation method provided in the embodiments of this application can also be applied to other electronic devices. For example, it can be applied to electronic devices that are communicatively connected to a robot. That is, after the electronic device determines the target angles corresponding to each joint of the robot's upper limb using the upper limb movement imitation method provided in the embodiments of this application, it can send control commands to the robot to control each joint of the robot's upper limb to move according to the corresponding target angles, thereby achieving movement imitation of the robot.

[0121] This application embodiment can collect the first pose corresponding to the key points of the upper limb of the object being imitated, and can convert the first pose into the second pose corresponding to the key points of the upper limb of the robot. Based on the second pose corresponding to the key points of the upper limb of the robot, through hierarchical secondary programming, the target angles corresponding to each joint of the robot's upper limb can be accurately determined. Thus, based on the target angles corresponding to each joint of the robot's upper limb, the robot can be accurately controlled to perform motion imitation, which can improve the similarity and stability of the robot's motion imitation and improve the robot's imitation effect.

[0122] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0123] Corresponding to the upper limb movement imitation method described in the above embodiments, Figure 7 A structural block diagram of the upper limb movement mimicking device provided in an embodiment of this application is shown. For ease of explanation, only the parts relevant to the embodiment of this application are shown. This device can be applied to robots, such as humanoid robots.

[0124] Reference Figure 7 The device may include:

[0125] The first pose acquisition module 701 is used to acquire the first pose corresponding to the key points of the upper limb of the object being imitated;

[0126] The second pose determination module 702 is used to determine the second pose corresponding to the upper limb key points of the robot based on the first pose.

[0127] The target angle determination module 703 is used to determine the target angles corresponding to each joint of the robot's upper limb through hierarchical quadratic programming based on the second pose.

[0128] The motion control module 704 is used to control the movement of the robot's upper limbs according to the target angles corresponding to each joint of the robot's upper limbs.

[0129] In some embodiments, the target angle determination module 703 is specifically used to determine the initial angles corresponding to each joint of the robot's upper limb through a first quadratic programming based on the second pose; determine the arm angle corresponding to the robot based on the second pose, and determine the angular velocity corresponding to the arm angle; update the initial angles corresponding to each joint of the robot's upper limb through a second quadratic programming based on the angular velocity corresponding to the arm angle, and obtain the target angles corresponding to each joint of the robot's upper limb.

[0130] In one possible implementation, the second pose may include the target position and target orientation corresponding to the robot wrist.

[0131] The target angle determination module 703 is also used to determine the initial angles of each joint of the robot's upper limb through a first-stage secondary planning based on the target position and target posture corresponding to the robot's wrist.

[0132] In one possible implementation, the second pose may further include the target position corresponding to the robot's shoulder and the target position corresponding to the robot's elbow.

[0133] The target angle determination module 703 is further configured to determine a first plane corresponding to the robot based on the target position corresponding to the robot shoulder, the target position corresponding to the robot elbow, and the target position corresponding to the robot wrist; and to determine an arm angle corresponding to the robot based on a reference plane and the first plane corresponding to the robot; the arm angle corresponding to the robot is the angle between the reference plane and the first plane, and the reference plane is determined based on the initial position corresponding to the robot shoulder, the initial position corresponding to the robot elbow, and the initial position corresponding to the robot wrist.

[0134] In some embodiments, the apparatus may further include:

[0135] The finger angle acquisition module is used to acquire the angles of each finger of the object being imitated;

[0136] The finger angle determination module is used to determine the angle of each finger of the robot based on the angle of each finger of the object being imitated.

[0137] The finger control module is used to control the movement of each finger of the robot according to the corresponding angle of each finger.

[0138] In some embodiments, the second pose determination module 702 is specifically used to determine the arm size of the object being imitated based on the first pose; obtain the arm size of the robot; and determine the second pose corresponding to the upper limb key points of the robot based on the arm size of the object being imitated, the arm size of the robot, and the first pose.

[0139] In one possible implementation, the arm size of the object being imitated may include a first size and a second size of the object being imitated, wherein the first size may be the size from the shoulder to the elbow of the object being imitated, and the second size may be the size from the shoulder to the wrist of the object being imitated; the arm size of the robot may include a third size and a fourth size, wherein the third size may be the size from the shoulder to the elbow of the robot, and the fourth size may be the size from the shoulder to the wrist of the robot.

[0140] The second pose determination module 702 is further configured to determine a second pose corresponding to the robot's elbow based on the first size, the third size, and the first pose; and to determine a second pose corresponding to the robot's wrist based on the second size, the fourth size, and the first pose.

[0141] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0143] Figure 8 This is a schematic diagram of the structure of a robot provided in one embodiment of this application. Figure 8 As shown, the robot 8 in this embodiment includes: at least one processor 80 ( Figure 8 (Only one is shown in the diagram), memory 81, and computer program 82 stored in said memory 81 and executable on said at least one processor 80, wherein the processor 80 executes said computer program 82 to implement the steps in any of the above embodiments of the upper limb movement imitation method.

[0144] Robot 8 may include, but is not limited to, processor 80 and memory 81. Those skilled in the art will understand that... Figure 8 The example shown is merely of robot 8 and does not constitute a limitation on robot 8. It may include more or fewer parts than shown, or combine certain parts, or different parts, such as input / output devices, network access devices, etc.

[0145] The processor 80 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0146] In some embodiments, the memory 81 may be an internal storage unit of the robot 8, such as a hard disk or memory of the robot 8. In other embodiments, the memory 81 may be an external storage device of the robot 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the robot 8. Furthermore, the memory 81 may include both internal storage units and external storage devices of the robot 8. The memory 81 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of computer programs. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0147] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above embodiments of the upper limb movement imitation methods.

[0148] This application provides a computer program product that stores a computer program. When the computer program is run on a robot, the robot can perform the steps described in the above embodiments of the upper limb movement imitation methods.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a device / robot, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable storage media cannot be electrical carrier signals or telecommunication signals.

[0150] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] In the embodiments provided in this application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An upper limb motion mimicking method characterized by comprising: The method is applied to a robot, and comprises: obtaining a first pose corresponding to upper limb key points of an imitated object; determining a second pose corresponding to upper limb key points of the robot according to the first pose; the second pose comprises a target position and a target posture corresponding to a wrist of the robot; determining initial angles of joints of the upper limb of the robot by first quadratic programming according to the target position and the target posture corresponding to the wrist of the robot; determining an arm angle corresponding to the robot and an angular velocity corresponding to the arm angle according to the second pose; updating the initial angles of the joints of the upper limb of the robot by second quadratic programming according to the angular velocity corresponding to the arm angle to obtain target angles of the joints of the upper limb of the robot; controlling the upper limb of the robot to move according to the target angles of the joints of the upper limb of the robot.

2. The method of claim 1, wherein, The second pose further comprises a target position corresponding to a shoulder of the robot and a target position corresponding to an elbow of the robot; determining the arm angle corresponding to the robot comprises: determining a first plane corresponding to the robot according to the target position corresponding to the shoulder of the robot, the target position corresponding to the elbow of the robot and the target position corresponding to the wrist of the robot; determining the arm angle corresponding to the robot according to a reference plane and the first plane corresponding to the robot; the arm angle corresponding to the robot is an included angle between the reference plane and the first plane, and the reference plane is determined according to an initial position corresponding to the shoulder of the robot, an initial position corresponding to the elbow of the robot and an initial position corresponding to the wrist of the robot.

3. The method of claim 1, wherein, The method further comprises: obtaining angles of fingers of the imitated object; determining angles of fingers of the robot according to the angles of the fingers of the imitated object; controlling the fingers of the robot to move according to the angles of the fingers of the robot.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of the second pose corresponding to the upper limb key points of the robot according to the first pose comprises: determining an arm size of the imitated object according to the first pose; obtaining an arm size of the robot; determining the second pose corresponding to the upper limb key points of the robot according to the arm size of the imitated object, the arm size of the robot and the first pose.

5. The method of claim 4, wherein, The arm size of the imitated object comprises a first size and a second size, the first size is a size from a shoulder to an elbow of the imitated object, and the second size is a size from the shoulder to a wrist of the imitated object; the arm size of the robot comprises a third size and a fourth size, the third size is a size from a shoulder to an elbow of the robot, and the fourth size is a size from the shoulder to a wrist of the robot; The determination of the second pose corresponding to the upper limb key points of the robot according to the arm size of the imitated object, the arm size of the robot and the first pose comprises: determine a second pose of an elbow of the robot according to the first size, the third size, and the first pose; determine a second pose of a wrist of the robot according to the second size, the fourth size, and the first pose.

6. An upper limb motion mimicking apparatus characterized by comprising: The device is applied to a robot, and the device comprises: a first pose acquisition module configured to acquire a first pose of upper limb key points of an imitated object; a second pose determination module configured to determine a second pose of the upper limb key points of the robot according to the first pose, wherein the second pose comprises a target position and a target pose of a wrist of the robot; a target angle determination module configured to determine initial angles of joints of the upper limb of the robot by first quadratic programming according to the target position and the target pose of the wrist of the robot, determine an arm angle of the robot according to the second pose, and determine an angular velocity corresponding to the arm angle; update the initial angles of the joints of the upper limb of the robot by second quadratic programming according to the angular velocity corresponding to the arm angle, and obtain target angles of the joints of the upper limb of the robot; a motion control module configured to control motion of the upper limb of the robot according to the target angles of the joints of the upper limb of the robot.

7. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program, so that the robot implements the upper limb action imitation method in any one of claims 1 to 5.

8. A computer program product storing a computer program, characterized in that, The computer program is executed by a computer, so that the computer implements the upper limb action imitation method in any one of claims 1 to 5.

Images