Robot remote operation control method, device, robot and electronic device

Abstract

The present invention discloses a robot teleoperation control method, device, robot, and electronic device. The method includes: acquiring target action data and displacement data of a target object, where the target action data includes head action data and arm action data; controlling the target robot to act according to the target action data so that the target robot completes the action corresponding to the target action data; based on the displacement data, performing centroid trajectory planning on the target robot using the MPC algorithm to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory so that the target robot moves to the position corresponding to the displacement data. The present invention solves the technical problem that the prior art cannot achieve full-body teleoperation of a robot.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular, to a robot teleoperation control method, device, robot, and electronic device. Background Art

[0002] The humanoid robot Walker has an appearance and structure highly similar to that of humans, with structures such as a head, torso, arms, palms, legs, and feet, and can achieve anthropomorphic movements. The humanoid robot can replace humans to perform some high-risk activities, but the current technical level is not sufficient to support the robot to make fully autonomous decisions in complex environments, and human participation is still required to control the robot to complete complex tasks. Summary of the Invention

[0003] Embodiments of the present invention provide a robot teleoperation control method, device, robot, and electronic device to at least solve the technical problem that the prior art cannot achieve full-body teleoperation of a robot.

[0004] According to one aspect of the embodiments of the present invention, a robot teleoperation control method is provided, including: obtaining target action data and displacement data of a target object, where the target action data includes head action data and arm action data; controlling the actions of a target robot according to the target action data so that the target robot completes the actions corresponding to the target action data; and based on the displacement data, performing centroid trajectory planning on the target robot by using an MPC algorithm to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory so that the target robot moves to the position corresponding to the displacement data.

[0005] According to another aspect of the embodiments of the present invention, a robot teleoperation control device is provided, including: a first obtaining module, configured to obtain target action data and displacement data of a target object, where the target action data includes head action data and arm action data; a first control module, configured to control the actions of a target robot according to the target action data so that the target robot completes the actions corresponding to the target action data; and a moving module, configured to perform centroid trajectory planning on the target robot by using an MPC algorithm based on the displacement data to obtain a target centroid trajectory, and establish a spring-damping system to track the target centroid trajectory so that the target robot moves to the position corresponding to the displacement data.

[0006] As an alternative example, the above device further includes: a second acquisition module, configured to acquire the head calibration action data, arm calibration action data, and calibration position data of the above target object before acquiring the target action data of the target object; a second control module, configured to control the head action of the above target robot according to the above head calibration action data, so that the head of the above target robot completes the action corresponding to the above head calibration action data, and control the arm action of the target robot according to the above arm calibration action data, so that the arm of the above target robot completes the action corresponding to the above arm calibration action data; a first creation module, configured to create a human body coordinate system with the above calibration position data as the origin; a second creation module, configured to create a robot coordinate system with the calibration position data of the above target robot as the origin.

[0007] As an alternative example, the above first acquisition module includes: an acquisition unit, configured to acquire the motion posture data and bone data of the above target object; a first calculation unit, configured to calculate the joint rotation matrix of the above target object according to the data fusion algorithm, filtering algorithm, and the above motion posture data; a second calculation unit, configured to calculate the bone vector of the above target object according to the above bone data; a third calculation unit, configured to calculate the product of the above joint rotation matrix and the above bone vector to obtain the above displacement data.

[0008] As an alternative example, the above first control module includes: a first control unit, configured to control the action of the above target robot according to the head action data, so that the head of the above target robot completes the action corresponding to the above head action data; a second control unit, configured to control the action of the above target robot according to the arm action data, so that the arm of the above target robot completes the action corresponding to the above arm action data.

[0009] As an alternative example, the above displacement module includes: a mapping unit, configured to map the displacement data into the human body coordinate system to obtain the mapped displacement data, and map the above mapped displacement data into the robot coordinate system to obtain the above target centroid trajectory; a third control unit, configured to control the action of the above target robot according to the above target centroid trajectory, so that the above target robot moves to the position corresponding to the above displacement data.

[0010] As an alternative example, the above device further includes: a third control module, configured to control the head camera of the above target robot to take pictures to obtain field of view data; a sending module, configured to send the above field of view data to a target device.

[0011] According to another aspect of the embodiments of the present invention, a robot is provided, including: a first acquisition module, configured to acquire target action data and displacement data of a target object, where the target action data includes head action data and arm action data; an execution module, configured to execute a target action according to the target action data; a movement module, configured to perform centroid trajectory planning on the target robot based on the MPC algorithm according to the displacement data to obtain a target centroid trajectory, and establish a spring-damping system, and move to the position corresponding to the displacement data according to the target centroid trajectory.

[0012] As an optional example, the robot further includes: a second acquisition module, configured to acquire head calibration action data, arm calibration action data, and calibration position data of the target object before acquiring the target action data of the target object; a first control module, configured to control the head action according to the head calibration action data so that the head completes the action corresponding to the head calibration action data, and control the arm action according to the arm calibration action data so that the arm completes the action corresponding to the arm calibration action data; a first creation module, configured to create a human body coordinate system with the calibration position data as the origin; a second creation module, configured to create a robot coordinate system with the calibration position data of the target robot as the origin.

[0013] As an optional example, the first acquisition module includes: an acquisition unit, configured to acquire the motion posture data and bone data of the target object; a first calculation unit, configured to calculate a joint rotation matrix of the target object according to a data fusion algorithm, a filtering algorithm, and the motion posture data; a second calculation unit, configured to calculate a bone vector of the target object according to the bone data; a third calculation unit, configured to calculate the product of the joint rotation matrix and the bone vector to obtain the displacement data.

[0014] As an optional example, the execution module includes: a first control unit, configured to control the head action according to the head action data so that the head completes the action corresponding to the head action data; a second control unit, configured to control the arm action according to the arm action data so that the arm completes the action corresponding to the arm action data.

[0015] As an optional example, the displacement module includes: a mapping unit, configured to map the displacement data into the human body coordinate system to obtain mapped displacement data, and map the mapped displacement data into the

[0016] robot coordinate system to obtain the target centroid trajectory; a movement unit, configured to move to the target position corresponding to the displacement data according to the target centroid 5 trajectory.

[0017] As an alternative example, the above-mentioned robot further includes: a second control module for controlling the head camera to take pictures to obtain field of view data; a sending module for sending the above-mentioned field of view data to a target device.

[0018] According to another aspect of the embodiments of the present invention, there is also provided a storage medium in which a computer program is stored. When the computer program is run by a processor, the above-mentioned robot teleoperation control method is executed.

[0019] According to another aspect of the embodiments of the present invention, there is also provided an electronic device including a memory and a processor. A computer program is stored in the above-mentioned memory, and the above-mentioned processor is configured to execute the above-mentioned robot teleoperation control method through the above-mentioned computer program.

[0020] 5 In the embodiments of the present invention, the target action data and displacement data of a target object are acquired.

[0021] Wherein the above-mentioned target action data includes head action data and arm action data; controlling the actions of a target robot according to the above-mentioned target action data so that the target robot completes the actions corresponding to the above-mentioned target action data; according to the above-mentioned displacement data, based on the MPC algorithm, performing centroid trajectory

[0022] planning on the above-mentioned target robot to obtain a target centroid trajectory, and establishing a spring-damping system to track the above-mentioned target centroid trajectory so that the above-mentioned target robot moves to the position corresponding to the above-mentioned displacement data. In the above method, by acquiring the whole-body motion data (target action data) of the target object and mapping it to the target robot, controlling the actions of the target robot so that the target robot completes the same actions as the target object, thereby achieving the purpose of remote teleoperation of the target object on the target robot, and further solving the technical problem that the prior art cannot achieve whole-body teleoperation of a robot. Description of the Drawings

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a flowchart of an alternative robot teleoperation control method according to an embodiment of the present invention;

[0025] Figure 2 is an overall technical solution diagram of an alternative robot teleoperation control method according to an embodiment of the present invention;

[0026] Figure 3Schematic diagram of wearing an inertial motion capture device for an optional robot teleoperation control method according to an embodiment of the present invention;

[0027] Figure 4 Flowchart of converting data of an inertial motion capture device for an optional robot teleoperation control method according to an embodiment of the present invention;

[0028] Figure 5 Flowchart of a full - body remote robot teleoperation control strategy for an optional robot teleoperation control method according to an embodiment of the present invention;

[0029] Figure 6 Schematic diagram of the structure of an optional robot teleoperation control device according to an embodiment of the present invention;

[0030] Figure 7 Schematic diagram of the structure of an optional robot according to an embodiment of the present invention;

[0031] Figure 8 Schematic diagram of an optional electronic device according to an embodiment of the present invention. Detailed implementation manners

[0032] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above - mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non - exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] According to the first aspect of the embodiments of the present invention, a robot teleoperation control method is provided. Optionally, as Figure 1 shown, the above - mentioned method includes:

[0035] S102. Obtain the target action data and displacement data of the target object, where the target action data includes head action data and arm action data;

[0036] S104. Control the target robot to act according to the target action data, so that the target robot completes the action corresponding to the target action data;

[0037] S106. Based on the displacement data, perform centroid trajectory planning on the target robot using the MPC algorithm to obtain the target centroid trajectory, and establish a spring-damper system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data.

[0038] Optionally, in this embodiment, the overall technical solution of whole-body remote teleoperation is as Figure 2 shown. By wearing an inertial motion capture device on the target object, the motion data (target action data) of the whole body of the target object is obtained, and the target action data is transmitted to the main controller on the computer in real time. The main controller converts the collected target action data into the working space of the target robot to obtain the expected motion trajectory of the target robot. The main controller on the computer solves the expected motion trajectory of the target robot through the whole-body motion control strategy to obtain the control data of each joint of the target robot, and sends the obtained control data of each joint to the target robot far away through the wireless local area network, so that the target robot completes the action and motion trajectory corresponding to the expected motion trajectory, thereby realizing the real-time control of the target robot.

[0039] Optionally, in this embodiment, the inertial motion capture device uses an inertial attitude sensor to collect data. Each sensor is built-in with a high-dynamic three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer. The operator stands in front of the computer and wears the attitude sensors on the corresponding joints of the target object respectively, which can provide data such as acceleration and angular velocity of each joint. By using the nine-axis data fusion algorithm and the Kalman filter algorithm, high-precision motion data can be calculated.

[0040] Optionally, in this embodiment, the schematic diagram of the target object wearing the inertial motion capture device is as Figure 3 shown. Seventeen inertial attitude sensors are worn on the joints of the target object's head, shoulders, upper arms, forearms, palms of both hands, back, waist, thighs, calves, feet, etc. respectively. The motion attitude data of the target object is captured in real time through the sensors to obtain the head action data, arm action data, and displacement data of the target object.

[0041] Optionally, in this embodiment, by obtaining the full-body motion data and target action data of the target object and mapping them to the target robot, the actions of the target robot are controlled so that the target robot completes the same actions as the target object, improving the flexibility, real-time performance, and accuracy of the target robot in completing tasks. Thus, the purpose of remotely operating the target robot by the target object is achieved, and furthermore, the technical problem that the prior art cannot achieve full-body teleoperation of the robot is solved.

[0042] As an alternative example, before obtaining the target action data of the target object, the above method further includes:

[0043] Obtaining the head calibration action data, arm calibration action data, and calibration position data of the target object;

[0044] Controlling the head action of the target robot according to the head calibration action data so that the head of the target robot completes the action corresponding to the head calibration action data, and controlling the arm action of the target robot according to the arm calibration action data so that the arm of the target robot completes the action corresponding to the arm calibration action data;

[0045] Creating a human body coordinate system with the calibration position data as the origin;

[0046] Creating a robot coordinate system with the calibration position data of the target robot as the origin.

[0047] Optionally, in this embodiment, during program initialization, the target object needs to perform a calibration action to eliminate the wearing error of the sensor on the body, and then record the data of the human head, robotic arm, and waist at this time to obtain the head calibration action data, arm calibration action data, and calibration position data. Then, control the head action of the target robot according to the head calibration action data, and control the arm action of the target robot according to the arm calibration action data to initialize the target robot. Let the position of the waist at this time be the origin to create a human body coordinate system, and let the calibration position data of the target robot at this time be the origin to create a robot coordinate system. When the target object moves later, map the rotation angle of the head to the head joint of the robot to adjust the field of view of the head camera. When the arm moves, map the rotation angle of the arm joint to the robotic arm joint of the target robot according to the rules to control the robotic arm to perform specific operations. When the target object moves, map the displacement of the waist from the human body coordinate system to the robot coordinate system to control the robot to move to the specified position.

[0048] As an alternative example, obtaining the displacement data of the target object includes:

[0049] Obtaining the motion posture data and bone data of the target object;

[0050] Calculate the joint rotation matrix of the target object based on the data fusion algorithm, filtering algorithm, and motion attitude data;

[0051] Calculate the bone vector of the target object based on the bone data;

[0052] Calculate the product of the joint rotation matrix and the bone vector to obtain displacement data.

[0053] Optionally, in this embodiment, after the motion attitude data of the target object is captured in real time by a sensor, the rotation matrix of each joint is obtained through the data fusion algorithm and the filtering algorithm. The bone data of the target object, including data such as body length, head length, neck length, shoulder width, upper arm length, forearm length, waist width, thigh length, calf length, ankle height, and foot length, is measured in advance and entered into the program to obtain the bone vector of the target object. By multiplying the bone vector and the rotation matrix, the displacement data of the target object can be obtained.

[0054] As an optional example, controlling the target robot's actions according to the target action data includes:

[0055] Control the actions of the target robot according to the head action data so that the head of the target robot completes the actions corresponding to the head action data;

[0056] Control the actions of the target robot according to the arm action data so that the arms of the target robot complete the actions corresponding to the arm action data.

[0057] As an optional example, based on the displacement data, perform centroid trajectory planning on the target robot using the MPC algorithm to obtain the target centroid trajectory, and establish a spring-damper system to track the target centroid trajectory so that the target robot moves to the position corresponding to the displacement data, including:

[0058] Map the displacement data to the human body coordinate system to obtain the mapped displacement data, and map the mapped displacement data to the robot coordinate system to obtain the target centroid trajectory;

[0059] Control the actions of the target robot according to the target centroid trajectory so that the target robot moves to the target position corresponding to the displacement data.

[0060] Optionally, in this embodiment, the target action data collected by the inertial motion capture device is in the human body coordinate system and needs to be mapped to the robot coordinate system. The data conversion here is divided into three parts: head action data conversion, arm action data conversion, and displacement data conversion. The specific process of the data conversion is as Figure 4 shown, and the control strategy flowchart is as Figure 5As shown in the figure, control is performed according to the target action data. The mapped head action data is received, and the head of the target robot is controlled through a cubic interpolation curve. At the same time, the mapped arm action data is received, and the arm of the target robot is also controlled through a cubic interpolation curve to ensure the smooth and safe operation of the robotic arm. When the target object moves, the displacement data of the target object in the human body coordinate system is mapped into the robot coordinate system to obtain the expected displacement data in the robot coordinate system. At the same time, the states of each joint of the target robot are read in real time, and the centroid trajectory of the target robot is planned based on the MPC algorithm. Then, a spring-damping system is established to track the planned centroid trajectory. Finally, the calculated hip and leg joint control data is sent to the target robot to make the target robot walk to the expected position.

[0061] As an optional example, the above method further includes:

[0062] Controlling the head camera of the target robot to take pictures to obtain visual field data;

[0063] Sending the visual field data to the target device.

[0064] Optionally, in this embodiment, the target device may be a mobile phone, a computer, etc. The picture taken by the head camera of the target robot is transmitted back to the computer in real time through a wireless local area network and displayed on the monitor of the computer, so that the operator standing in front of the computer can obtain the environmental information around the target robot and make adjustments according to the environment where the target robot is located.

[0065] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0066] According to another aspect of the embodiments of the present application, a robot teleoperation control device is further provided, as Figure 6 shown, including:

[0067] A first acquisition module 602, configured to acquire the target action data and displacement data of the target object, where the above target action data includes head action data and arm action data;

[0068] A first control module 604, configured to control the actions of the target robot according to the target action data, so that the target robot completes the actions corresponding to the target action data;

[0069] A mobile module 606 is configured to plan the centroid trajectory of a target robot based on displacement data according to the MPC algorithm, obtain a target centroid trajectory, and establish a spring-damping system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data.

[0070] Optionally, in this embodiment, by wearing an inertial motion capture device on the target object, the motion data of the whole body of the target object, i.e., target action data, is obtained and transmitted to the main controller on the computer in real time. The main controller converts the collected target action data into the workspace of the target robot to obtain the desired motion trajectory of the target robot. The main controller on the computer solves the desired motion trajectory of the target robot through a whole-body motion control strategy to obtain the control data of each joint of the target robot, and sends the obtained control data of each joint to the target robot at a distance through a wireless local area network, so that the target robot completes the actions and motion trajectories corresponding to the desired motion trajectory, thereby realizing the real-time control of the target robot.

[0071] Optionally, in this embodiment, the inertial motion capture device uses inertial attitude sensors to collect data. Each sensor is built-in with a high-dynamic three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. The operator stands in front of the computer and wears the attitude sensors on the corresponding joints of the target object respectively, which can provide data such as acceleration and angular velocity of each joint. By using a nine-axis data fusion algorithm and a Kalman filtering algorithm, high-precision motion data can be calculated.

[0072] Optionally, in this embodiment, 17 inertial attitude sensors are worn on the joints of the target object such as the head, both shoulders, both upper arms, both forearms, both palms, the back, the waist, both thighs, both calves, and both feet respectively. The motion attitude data of the target object is captured in real time through the sensors to obtain the head action data, arm action data, and displacement data of the target object.

[0073] Optionally, in this embodiment, by obtaining the whole-body motion data of the target object, i.e., target action data, and mapping it to the target robot to control the actions of the target robot, so that the target robot completes the same actions as the target object, the flexibility, real-time performance, and accuracy of the target robot to complete tasks are improved. Thus, the purpose of remote teleoperation of the target object for the target robot is realized, and furthermore, the technical problem that the prior art cannot realize the whole-body teleoperation of the robot is solved.

[0074] As an optional example, the above device further includes:

[0075] A second acquisition module, configured to acquire the head calibration action data, arm calibration action data, and calibration position data of the target object before acquiring the target action data of the target object;

[0076] A second control module, configured to control the head movement of the target robot according to the head calibration movement data, so that the head of the target robot completes the movement corresponding to the head calibration movement data, and control the arm movement of the target robot according to the arm calibration movement data, so that the arm of the target robot completes the movement corresponding to the arm calibration movement data;

[0077] A first creation module, configured to create a human body coordinate system with the calibration position data as the origin;

[0078] A second creation module, configured to create a robot coordinate system with the calibration position data of the target robot as the origin.

[0079] Optionally, in this embodiment, during program initialization, the target object needs to perform a calibration movement to eliminate the wearing error of the sensor on the body, and then record the data of the human head, robotic arm, and waist at this time to obtain the head calibration movement data, arm calibration movement data, and calibration position data, and control the head movement of the target robot according to the head calibration movement data, control the arm movement of the target robot according to the arm calibration movement data, and initialize the target robot. Let the position of the waist at this time be the origin to create a human body coordinate system, and let the calibration position data of the target machine at this time be the origin to create a robot coordinate system. When the target object moves later, map the rotation angle of the head to the head joint of the robot to adjust the field of view of the head camera. When the arm moves, map the rotation angle of the arm joint to the robotic arm joint of the target robot according to rules to control the robotic arm to perform specific operations. When the target object moves, map the displacement of the waist from the human body coordinate system to the robot coordinate system to control the robot to move to a specified position.

[0080] As an optional example, the first acquisition module includes:

[0081] An acquisition unit, configured to acquire the motion posture data and bone data of the target object;

[0082] A first calculation unit, configured to calculate the joint rotation matrix of the target object according to the data fusion algorithm, filtering algorithm, and motion posture data;

[0083] A second calculation unit, configured to calculate the bone vector of the target object according to the bone data;

[0084] A third calculation unit, configured to calculate the product of the joint rotation matrix and the bone vector to obtain the displacement data.

[0085] Optionally, in this embodiment, after the motion attitude data of the target object is captured in real time by a sensor, the rotation matrix of each joint is obtained through a data fusion algorithm and a filtering algorithm. The bone data of the target object, including data such as body length, head length, neck length, shoulder width, upper arm length, forearm length, waist width, thigh length, calf length, ankle height, and foot length, is measured in advance and entered into the program to obtain the bone vector of the target object. By multiplying the bone vector by the rotation matrix, the displacement data of the target object can be obtained.

[0086] As an optional example, the first control module includes:

[0087] The first control unit is used to control the actions of the target robot according to the head motion data, so that the head of the target robot completes the actions corresponding to the head motion data;

[0088] The second control unit is used to control the actions of the target robot according to the arm motion data, so that the arms of the target robot complete the actions corresponding to the arm motion data.

[0089] As an optional example, the displacement module includes:

[0090] The mapping unit is used to map the displacement data into the human body coordinate system to obtain the mapped displacement data, and map the mapped displacement data into the robot coordinate system to obtain the target centroid trajectory;

[0091] The third control unit is used to control the actions of the target robot according to the target centroid trajectory, so that the target robot moves to the target position corresponding to the displacement data.

[0092] Optionally, in this embodiment, the target action data collected by the inertial motion capture device is in the human body coordinate system and needs to be mapped to the robot coordinate system. The data conversion here is divided into three parts: head motion data conversion, arm motion data conversion, and displacement data conversion, and the control is performed according to the target action data. Receive the mapped head motion data and control the head of the target robot through a cubic interpolation curve. At the same time, receive the mapped arm motion data and also control the arms of the target robot through a cubic interpolation curve to ensure the smooth and safe operation of the robotic arm. When the target object moves, map the displacement data of the target object in the human body coordinate system to the robot coordinate system to obtain the expected displacement data in the robot coordinate system. At the same time, read the states of each joint of the target robot in real time, perform centroid trajectory planning of the target robot based on the MPC algorithm, then establish a spring-damper system to track the planned centroid trajectory, and finally send the calculated hip and leg joint control data to the target robot to make the target robot walk to the expected position.

[0093] As an optional example, the above device further includes:

[0094] A third control module, configured to control the head camera of the target robot to take pictures and obtain field of view data;

[0095] A sending module, configured to send the field of view data to the target device.

[0096] Optionally, in this embodiment, the target device may be a mobile phone, a computer, etc. The picture taken by the head camera of the target robot is transmitted back to the computer in real time through a wireless local area network and displayed on the monitor of the computer, so that the operator standing in front of the computer can obtain the environmental information around the target robot and make adjustments according to the environment where the target robot is located.

[0097] According to another aspect of the embodiments of the present invention, a robot is provided. Optionally, as Figure 7 shown, it includes:

[0098] A first acquisition module 702, configured to acquire target action data and displacement data of a target object, where the target action data includes head action data and arm action data;

[0099] An execution module 704, configured to execute the target action according to the target action data;

[0100] A movement module 706, configured to perform centroid trajectory planning on the target robot based on the MPC algorithm according to the displacement data to obtain a target centroid trajectory, and establish a spring-damping system, and move to the position corresponding to the displacement data according to the target centroid trajectory.

[0101] Optionally, in this embodiment, by wearing an inertial motion capture device on the target object to acquire the motion data of the whole body of the target object, the target action data is converted into the workspace of the target robot to obtain the expected motion trajectory of the target robot. The expected motion trajectory is solved through a whole body motion control strategy to obtain the control data of each joint of the target robot, and the target action is executed according to the control data of each joint, so that the target robot completes the actions and motion trajectories corresponding to the expected motion trajectory.

[0102] Optionally, in this embodiment, the inertial motion capture device uses an inertial attitude sensor to collect data. Each sensor is built-in with a high-dynamic three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer. The operator stands in front of the computer and wears the attitude sensors on the corresponding joints of the target object respectively, which can provide data such as acceleration and angular velocity of each joint. By using a nine-axis data fusion algorithm and a Kalman filter algorithm, high-precision motion data can be calculated.

[0103] Optionally, in this embodiment, 17 inertial attitude sensors are respectively worn on joints of a target object, such as the head, both shoulders, both upper arms, both forearms, both palm centers, the back, the waist, both thighs, both calves, and both feet. The motion attitude data of the target object is captured in real time through the sensors, and the head motion data, arm motion data, and displacement data of the target object are obtained.

[0104] Optionally, in this embodiment, by obtaining the full-body motion data target action data of the target object and mapping it to the target robot, the target robot executes the action target according to the target action data, so that the target robot completes the same action as the target object, improving the flexibility, real-time performance, and accuracy of the target robot in completing tasks. Thus, the purpose of remote teleoperation of the target object on the target robot is achieved, and furthermore, the technical problem that the prior art cannot achieve full-body teleoperation of a robot is solved.

[0105] As an optional example, the above-mentioned robot further includes:

[0106] A second acquisition module, configured to acquire the head calibration action data, arm calibration action data, and calibration position data of the target object before acquiring the target action data of the target object;

[0107] A first control module, configured to control the head motion according to the head calibration action data so that the head completes the action corresponding to the head calibration action data, and control the arm motion according to the arm calibration action data so that the arm completes the action corresponding to the arm calibration action data;

[0108] A first creation module, configured to create a human body coordinate system with the calibration position data as the origin;

[0109] A second creation module, configured to create a robot coordinate system with the calibration position data of the target robot as the origin.

[0110] Optionally, in this embodiment, during program initialization, the target object needs to perform a calibration action to eliminate the wearing error of the sensor on the body. Then, the data of the human head, robotic arm, and waist are recorded at this time to obtain the head calibration action data, arm calibration action data, and calibration position data. The head action of the target robot is controlled according to the head calibration action data, and the arm action of the target robot is controlled according to the arm calibration action data to initialize the target robot. Set the position of the waist at this time as the origin to create a human body coordinate system, and set the target machine calibration position data at this time as the origin to create a robot coordinate system. When the target object moves later, the rotation angle of the head is mapped to the head joint of the robot to adjust the field of view of the head camera. When the arm moves, the rotation angle of the arm joint is mapped to the robotic arm joint of the target robot according to the rules to control the robotic arm to perform specific operations. When the target object moves, the displacement of the waist is mapped from the human body coordinate system to the robot coordinate system to control the robot to move to the specified position.

[0111] As an optional example, the first acquisition module includes:

[0112] An acquisition unit for acquiring the motion posture data and bone data of the target object;

[0113] A first calculation unit for calculating the joint rotation matrix of the target object according to the data fusion algorithm, filtering algorithm, and motion posture data;

[0114] A second calculation unit for calculating the bone vector of the target object according to the bone data;

[0115] A third calculation unit for calculating the product of the joint rotation matrix and the bone vector to obtain the displacement data.

[0116] Optionally, in this embodiment, after the motion posture data of the target object is captured in real time by the sensor, the rotation matrix of each joint is obtained through the data fusion algorithm and the filtering algorithm. The bone data of the target object, including data such as body length, head length, neck length, shoulder width, upper arm length, forearm length, waist width, thigh length, calf length, ankle height, and foot length, is measured in advance and entered into the program to obtain the bone vector of the target object. By multiplying the bone vector and the rotation matrix, the displacement data of the target object can be obtained.

[0117] As an optional example, the execution module includes:

[0118] A first control unit for controlling the head action according to the head action data so that the head completes the action corresponding to the head action data;

[0119] A second control unit for controlling the arm action according to the arm action data so that the arm completes the action corresponding to the arm action data.

[0120] As an alternative example, the displacement module includes:

[0121] A mapping unit, configured to map displacement data to a human body coordinate system to obtain mapped displacement data, and map the mapped displacement data to a robot coordinate system to obtain a target centroid trajectory;

[0122] A moving unit, configured to move to a target position corresponding to the displacement data according to the target centroid trajectory.

[0123] Optionally, in this embodiment, the target action data collected by the inertial motion capture device is in the human body coordinate system and needs to be mapped to the robot coordinate system. The data conversion here is divided into three parts: head action data conversion, arm action data conversion, and displacement data conversion, and control is performed according to the target action data. Receive the mapped head action data and control the head of the target robot through a cubic interpolation curve. At the same time, receive the mapped arm action data and also control the arm of the target robot through a cubic interpolation curve to ensure the smooth and safe operation of the robotic arm. When the target object moves, map the displacement data of the target object in the human body coordinate system to the robot coordinate system to obtain the expected displacement data in the robot coordinate system. At the same time, read the joint states of the target robot in real time, perform centroid trajectory planning for the target robot based on the MPC algorithm, then establish a spring-damping system to track the planned centroid trajectory, and finally the target robot executes actions according to the calculated hip and leg joint control data so that the target robot walks to the expected position.

[0124] As an alternative example, the above-mentioned robot further includes:

[0125] A second control module, configured to control the head camera to take pictures to obtain field of view data;

[0126] A sending module, configured to send the field of view data to the target device.

[0127] Optionally, in this embodiment, the target device may be a mobile phone, a computer, etc. Transmit the picture taken by the head camera of the target robot back to the computer in real time through a wireless local area network and display it on the monitor of the computer, so that the operator standing in front of the computer can obtain the environmental information around the target robot and make adjustments according to the environment where the target robot is located.

[0128] For other examples of this embodiment, please refer to the above examples and will not be elaborated here.

[0129] Figure 8 is a schematic diagram of an alternative electronic device according to an embodiment of the present application, as Figure 8As shown in the figure, it includes a processor 802, a communication interface 804, a memory 806, and a communication bus 808. Among them, the processor 802, the communication interface 804, and the memory 806 communicate with each other through the communication bus 808. Among them,

[0130] The memory 806 is used to store computer programs;

[0131] When the processor 802 is used to execute the computer program stored on the memory 806, the following steps are implemented:

[0132] Obtain the target action data and displacement data of the target object, where the target action data includes head action data and arm action data;

[0133] Control the target robot to act according to the target action data, so that the target robot completes the action corresponding to the target action data;

[0134] According to the displacement data, based on the MPC algorithm, perform centroid trajectory planning on the target robot to obtain the target centroid trajectory, and establish a spring-damping system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data.

[0135] Optionally, in this embodiment, the above communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 8 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic device and other devices.

[0136] The memory may include RAM, or may include non-volatile memory, for example, at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0137] As an example, the above memory 806 may but is not limited to include the first acquisition module 602, the first control module 604, and the movement module 606 in the above robot teleoperation control device. In addition, it may also include but is not limited to other module units in the above robot teleoperation control device, which will not be elaborated in this example.

[0138] The above-mentioned processor can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit, central processing unit), NP (Network Processor, network processor), etc.; it can also be a DSP (Digital Signal Processing, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), FPGA (Field-Programmable Gate Array, field-programmable gate array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0139] Optionally, the specific examples in this embodiment may refer to the examples described in the above-mentioned embodiment, and will not be elaborated here.

[0140] Those of ordinary skill in the art can understand that Figure 8 The structure shown is only schematic. The device for implementing the robot teleoperation control method can be a terminal device, which can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 8 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 8 or have a different configuration from that shown in Figure 8 Those shown.

[0141] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a ROM, a RAM, a magnetic disk or an optical disc, etc.

[0142] According to another aspect of the embodiment of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is run by a processor, it executes the steps in the above-mentioned robot teleoperation control method.

[0143] Optionally, in this embodiment, those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing the relevant hardware of the terminal device. The program can be stored in a computer-readable storage medium, and the storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0144] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0145] If the integrated unit in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0146] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the units or modules can be in an electrical or other form.

[0148] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, in each embodiment of the present invention, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0150] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A robot teleoperation control method, characterized in that Including: Obtain the target action data and displacement data of the target object, where the target action data includes head action data and arm action data, and the target action data and the displacement data are obtained by an inertial motion capture device; Control the target robot to act according to the target action data, so that the target robot completes the action corresponding to the target action data; Based on the displacement data, perform centroid trajectory planning on the target robot using the MPC algorithm to obtain a target centroid trajectory, and establish a spring-damping system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data; wherein, before obtaining the target action data of the target object, the method further includes: Obtain the head calibration action data, arm calibration action data, and calibration position data of the target object; Control the head action of the target robot according to the head calibration action data, so that the head of the target robot completes the action corresponding to the head calibration action data, and control the arm action of the target robot according to the arm calibration action data, so that the arm of the target robot completes the action corresponding to the arm calibration action data; Create a human body coordinate system with the calibration position data as the origin; Create a robot coordinate system with the calibration position data of the target robot as the origin; Wherein, the obtaining of the displacement data of the target object includes: Obtain the motion posture data and bone data of the target object; Calculate the joint rotation matrix of the target object according to the data fusion algorithm, filtering algorithm, and the motion posture data; Calculate the bone vector of the target object according to the bone data; Calculate the product of the joint rotation matrix and the bone vector to obtain the displacement data; Wherein, the performing centroid trajectory planning on the target robot based on the displacement data using the MPC algorithm to obtain a target centroid trajectory, and establishing a spring-damping system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data includes: Map the displacement data into the human body coordinate system to obtain the mapped displacement data, and map the mapped displacement data into the robot coordinate system to obtain the target centroid trajectory; Control the target robot to act according to the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data.

2. The method according to claim 1, characterized in that, The controlling the target robot to act according to the target action data includes: Control the target robot to act according to the head action data, so that the head of the target robot completes the action corresponding to the head action data; Control the target robot to act according to the arm action data, so that the arm of the target robot completes the action corresponding to the arm action data.

3. The method according to claim 1, wherein The method further includes: Control the head camera of the target robot to take pictures to obtain vision data; Send the vision data to the target device.

4. A robot teleoperation control device, characterized in that, Including: A first acquisition module, configured to acquire target action data and displacement data of a target object, where the target action data includes head action data and arm action data, and the target action data and the displacement data are acquired by an inertial motion capture device; A first control module, configured to control a target robot to act according to the target action data, so that the target robot completes the action corresponding to the target action data; A movement module, configured to perform centroid trajectory planning on the target robot based on the MPC algorithm according to the displacement data to obtain a target centroid trajectory, and establish a spring-damping system to track the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data; A second acquisition module, configured to acquire the head calibration action data, arm calibration action data, and calibration position data of the target object before acquiring the target action data of the target object; A second control module, configured to control the head action of the target robot according to the head calibration action data, so that the head of the target robot completes the action corresponding to the head calibration action data, and control the arm action of the target robot according to the arm calibration action data, so that the arm of the target robot completes the action corresponding to the arm calibration action data; A first creation module, configured to create a human body coordinate system with the calibration position data as the origin; A second creation module, configured to create a robot coordinate system with the calibration position data of the target robot as the origin; The first acquisition module includes: an acquisition unit, configured to acquire the motion posture data and bone data of the target object; a first calculation unit, configured to calculate a joint rotation matrix of the target object according to a data fusion algorithm, a filtering algorithm, and the motion posture data; a second calculation unit, configured to calculate a bone vector of the target object according to the bone data; a third calculation unit, configured to calculate the product of the joint rotation matrix and the bone vector to obtain the displacement data; The displacement module includes: a mapping unit, configured to map the displacement data into the human body coordinate system to obtain mapped displacement data, and map the mapped displacement data into the robot coordinate system to obtain the target centroid trajectory; a third control unit, configured to control the target robot to act according to the target centroid trajectory, so that the target robot moves to the position corresponding to the displacement data.

5. A robot, characterized in that, Including: A first acquisition module, configured to acquire target action data and displacement data of a target object, where the target action data includes head action data and arm action data, and the target action data and the displacement data are acquired by an inertial motion capture device; An execution module, configured to execute a target action according to the target action data; A movement module, configured to perform centroid trajectory planning on a target robot based on the MPC algorithm according to the displacement data to obtain a target centroid trajectory, and establish a spring-damping system to move to the position corresponding to the displacement data according to the target centroid trajectory; A second acquisition module, configured to acquire the head calibration action data, arm calibration action data, and calibration position data of the target object before acquiring the target action data of the target object; a first control module, configured to control the head action according to the head calibration action data, so that the head completes the action corresponding to the head calibration action data, and control the arm action according to the arm calibration action data, so that the arm completes the action corresponding to the arm calibration action data; A first creation module, configured to create a human body coordinate system with the calibration position data as the origin; A second creation module, configured to create a robot coordinate system with the calibration position data of the target robot as the origin; The first acquisition module includes: an acquisition unit, configured to acquire the motion posture data and bone data of the target object; a first calculation unit, configured to calculate the joint rotation matrix of the target object according to the data fusion algorithm, filtering algorithm, and the motion posture data; a second calculation unit, configured to calculate the bone vector of the target object according to the bone data; a third calculation unit, configured to calculate the product of the joint rotation matrix and the bone vector to obtain the displacement data; The displacement module includes: a mapping unit, configured to map the displacement data into the human body coordinate system to obtain the mapped displacement data, and map the mapped displacement data into the robot coordinate system to obtain the target centroid trajectory; a moving unit, configured to move to the target position corresponding to the displacement data according to the target centroid trajectory.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run by a processor, it executes the method described in any one of claims 1 to 3.

7. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 3 through the computer program.

Images