Teleoperation method of robot, robot, and storage medium

Abstract

The application relates to a teleoperation method of a robot, a robot and a computer readable storage medium. The method comprises the steps of: acquiring an image of a target object, and generating a point cloud of the target object according to the image; establishing a virtual clamp based on a geometric feature corresponding to the point cloud; the virtual clamp comprises a forbidden area virtual clamp and / or a guide virtual clamp; determining a reference point of the virtual clamp and a control point of the robot, determining a virtual force of the virtual clamp acting on the control point according to a distance between the reference point and the control point; and determining a control force applied to the control point based on the virtual force. The method can timely adapt to a changing or unstructured environment, and improve the safety and operation performance of teleoperation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot teleoperation, and in particular to a robot teleoperation method, a robot, and a storage medium. BACKGROUND

[0002] Robot teleoperation is widely used in the fields of remote medical treatment, remote service, remote grabbing, space exploration, etc. Compared with robot control, a teleoperation system builds a real and reliable human-machine interaction scene, which enables an operator to remotely control a robot in a natural and controllable manner to complete various tasks. However, when the working space environment is relatively complex and variable, the existing robot teleoperation technology still needs to be improved in terms of the accuracy, stability, and safety of robot control. SUMMARY

[0003] A first aspect of the embodiments of the present application provides a robot teleoperation method, including the steps of: acquiring an image of a target object, and generating a point cloud of the target object according to the image; establishing a virtual clamp based on a geometric feature corresponding to the point cloud; the virtual clamp includes a forbidden area virtual clamp and / or a guide virtual clamp; determining a reference point of the virtual clamp and a control point of the robot, determining a virtual force of the virtual clamp acting on the control point according to a distance between the reference point and the control point; and determining a control force applied to the control point based on the virtual force.

[0004] In some embodiments, the determination of the virtual force of the virtual clamp acting on the control point according to the distance between the reference point and the control point includes: when the distance is greater than or equal to a preset value, the virtual force is zero; and when the distance is less than the preset value, the smaller the distance between the reference point and the control point, the greater the virtual force.

[0005] In some embodiments, the method further includes: determining a spherical potential field with the reference point as the center of a sphere, and setting a radius of the spherical potential field to be greater than or equal to half of a distance between centers of adjacent spherical potential fields; and when the control point is located outside an edge of the spherical potential field, the virtual force is zero.

[0006] In some embodiments, the reference point is selected from a point cloud point representing a surface of an object in the target object.

[0007] In some embodiments, the method further includes: constructing the reference point based on the geometric feature, and a distribution of the reference point corresponds to an expected motion path or an expected obstacle avoidance path of the control point.

[0008] In some embodiments, the geometric feature includes an axis of a cylinder, a surface of a workbench, an edge of a workpiece, or a center of a sphere.

[0009] In some embodiments, the method further comprises: when the virtual clamp is a forbidden area virtual clamp, setting the virtual force as a repulsive force, and setting the direction of the repulsive force as away from the reference point; or when the virtual clamp is a guide virtual clamp, setting the virtual force as an attractive force, and setting the direction of the attractive force as pointing to the reference point.

[0010] In some embodiments, the method further comprises: when the virtual clamp is a guide virtual clamp, presetting a force vacuum range centered at the reference point; when the distance is less than the preset value and the control point is located outside the force vacuum range, the smaller the distance between the reference point and the control point, the greater the virtual force; when the control point is located within the force vacuum range, the virtual force is zero.

[0011] In some embodiments, the method further comprises: when the virtual clamp is a guide virtual clamp, presetting a force vacuum range centered at the reference point; when the distance is less than the preset value and the control point is located outside the force vacuum range, the smaller the distance between the reference point and the control point, the greater the virtual force; when the control point is located within the force vacuum range, the virtual force is zero.

[0012] In some embodiments, the method further comprises: when the virtual clamp includes a plurality of reference points, setting the virtual force as the virtual resultant force of the plurality of reference points on the control point.

[0013] In some embodiments, the method further comprises: when the robot includes a plurality of control points, setting the virtual force as the virtual resultant force of the reference point on each control point.

[0014] In some embodiments, before the virtual force of the virtual clamp acting on the control point is determined according to the distance between the reference point and the control point, the method further comprises: determining a preset range including all control points and moving with the robot; when the preset range enters the virtual force action range of any reference point, determining the control force applied to the control point in the preset range based on the virtual force of the any reference point acting on the control point in the preset range; when the virtual force action range of any reference point is located outside the preset range, not calculating the virtual force of the any reference point acting on the control point in the preset range.

[0015] In some embodiments, the preset range is a spherical range, and the method further comprises: adjusting the radius of the spherical range according to the speed of the movement of the robot, the faster the movement of the robot, the greater the radius of the spherical range.

[0016] In some embodiments, the radius of the spherical range is the product of the movement speed of the spherical range and the cycle interval of adjusting the radius of the spherical range.

[0017] In some embodiments, the method further comprises: providing a master robot and a slave robot; establishing the virtual fixture based on a target object in an environment of the slave robot; adjusting a force output of a drive device of the slave robot based on the virtual force; and providing haptic feedback to the master robot based on the virtual force.

[0018] A second aspect of embodiments of the present application provides a robot, comprising a memory and a processor, the memory storing a computer program, the processor implementing the steps of any of the above methods when executing the computer program.

[0019] A third aspect of embodiments of the present application provides a computer readable storage medium, having stored thereon a computer program, the computer program being executed by a processor to implement the steps of any of the above methods.

[0020] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows, the drawings in the following description are only some embodiments of the present application, and do not constitute a limitation on the disclosure content and protection scope of the present application.

[0022] Figure 1 is a flowchart of a teleoperation method of a robot provided by some embodiments of the present application;

[0023] Figure 2 is a schematic diagram of a virtual fixture of an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a virtual fixture of another embodiment of the present application;

[0025] Figure 4 is a schematic diagram of a fusion potential field formed by two reference points of an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of a wall-shaped potential field formed by multiple reference points of another embodiment of the present application;

[0027] Figure 6 is an application scenario diagram of selecting reference points to calculate virtual forces of some embodiments of the present application;

[0028] Figure 7 is a flowchart of a teleoperation method of a robot provided by some embodiments of the present application;

[0029] Figure 8is a structural schematic diagram of a robot provided by some embodiments of the present application;

[0030] Figure 9 is a structural schematic diagram of a computer readable storage medium provided by some embodiments of the present application. DETAILED DESCRIPTION

[0031] To make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application are described in detail below with reference to the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a number of ways other than those described herein without departing from the spirit of the present application, and it is to be understood that similar implementations can be employed as structural and / or functional equivalents of the devices and methods described herein. Therefore, the present application is not limited to the specific embodiments disclosed below, but only by the claims.

[0032] The flowcharts in this specification show the operations of system implementations according to some embodiments of the present specification. It should be clearly understood that the operations of the flowcharts can not be implemented in sequence. Instead, the operations can be implemented in reverse order or simultaneously. In addition, one or more other operations can be added to the flowcharts. One or more operations can be removed from the flowcharts.

[0033] For the convenience of description, the present specification will explain the terms that will appear in the following description as follows:

[0034] Virtual fixture: also known as virtual guidance or virtual constraint, is an algorithm implemented by a robot controller to simulate a physical fixture in the actual working environment of the robot. For a forbidden area virtual fixture, when the end effector of the robot tries to cross the virtual fixture, a reaction force will be generated by the virtual fixture and transmitted to the robot controller, which has the effect that the end effector of the robot encounters a real obstacle, and the reaction force can limit the end effector from entering the virtual fixture. For a guidance virtual fixture, when the end effector of the robot tries to deviate from a specific trajectory, a reaction force will be generated by the virtual fixture and transmitted to the robot controller, which has the effect that the end effector of the robot deviates from the trajectory due to inertia or other reasons, and the reaction force can constrain and guide the robot to continue moving along the specific trajectory. Here, the virtual fixture can be of any shape, usually designed based on the actual environment to better adapt to actual applications.

[0035] Point cloud: a collection of data points of a three-dimensional plane, each data point including three-dimensional coordinate information of X, Y, Z axes. When the data points converge together, a three-dimensional map or model can be created. For example, if the compiled point cloud data is taken from a neighborhood, the created point cloud can show the locations of buildings, trees and power lines, and their heights relative to the ground. The collected point cloud data and the density of the point cloud can reflect the details of the environment and specific types of terrain and structures. The data points of the point cloud can also include color information or reflectance intensity information, etc.

[0036] Robot teleoperation technology can include several implementations:

[0037] First, robot teleoperation without force (haptic) feedback. The corresponding robot teleoperation system includes a master device and a slave device, which can be a robotic arm, and the master device can be another robotic arm or a dedicated device. In this technology, the slave device will follow the position and speed of the master device, but the slave device will not provide force feedback from the slave device to the master device.

[0038] Second, robot teleoperation with force (haptic) feedback. This technology is based on the above technology and additionally includes the force / torque sensing capability of the slave device, and can transmit the force / torque sensed on the slave device to the master device as feedback information. This feedback information can be presented through the actuation system on the master device side, so that the operator can feel this force feedback.

[0039] Third, robot teleoperation based on virtual fixtures. This technology is based on the above technology and additionally includes a virtual fixture with a fixed shape to assist the operator in performing remote operation tasks.

[0040] The inventors found that the above-mentioned first and second teleoperation technologies mainly rely on the operator to perform tasks under full manual control, while the operation tasks on the robot side can be autonomously and efficiently completed by the robot without human intervention, so the overall task efficiency of this robot teleoperation technology has not been optimized. Secondly, when teleoperation is performed under poor network connection or external interference, network delay and external interference can cause the operator's intention on the master device side to mismatch the actual execution of the slave device, thereby affecting the safety of robot control. For example, when the slave device operates fragile workpieces or the workspace of the slave device is very limited, the safety of the slave device during operation and the reliability of the entire system can be difficult to guarantee. In addition, although the operator has an intelligent advantage, he lacks the high precision of the robot, so full manual teleoperation may not meet the high precision and high quality requirements of certain tasks.

[0041] The third robot teleoperation technology described above realizes human-machine interaction based on a virtual clamp. Although it can overcome some limitations in the full manual teleoperation described above, it requires the position of the virtual clamp to be established in advance, and cannot be updated in real time. Therefore, it is usually applied to a few application scenarios that do not require updating of the virtual clamp, such as repetitive operation tasks. In addition, since the shape of the virtual clamp is pre-configured, it cannot be applied to and adapt in real time to tasks that require the shape of the virtual clamp to change according to the current state of the environment, such as tasks related to processing soft surfaces, and thus is not conducive to improving safety and operation performance.

[0042] In other words, the virtual clamp in the technical solution described above is typically dependent on the task and is pre-set by a person in a known working environment. It cannot reflect and update the current state of the environment in real time according to changes in the environment, and it is also difficult to adapt to changing or unstructured environments in a timely manner.

[0043] Therefore, the embodiments of the present application provide a robot teleoperation method, which can obtain an image of a target object in real time, generate a point cloud of the target object according to the image, and establish a virtual clamp in real time based on the geometric features corresponding to the point cloud, so that the robot teleoperation can adapt to changing or unstructured environments in a timely manner, overcoming the technical defect that the existing pre-set virtual clamp can only be applied to a fixed working environment that has been constructed. In addition, the virtual clamp established in real time by the embodiments of the present application can process and adapt to environments with irregular or constantly changing geometric shapes, thereby overcoming the technical limitation that the existing virtual clamp can only be constructed as a fixed shape.

[0044] Figure 1 A flowchart of a robot teleoperation method provided for some embodiments of the present application is shown in the figure. The method includes the following steps:

[0045] Step S101: Obtain an image of a target object and generate a point cloud of the target object according to the image.

[0046] In some embodiments, the image data and / or depth data of the target object can be collected in real time by a vision device according to a pre-set sampling frequency, so as to generate a point cloud of the target object according to the image data and / or depth data. The method of generating the point cloud of the target object can use related methods in existing image recognition technologies, which will not be described here.

[0047] The vision device can be installed on the end effector of the robot to follow the movement of the end effector and collect images of the target object in real time. The vision device can also be a fixed device relative to the target object, which is used to collect images of the target object in real time.

[0048] The vision device can be a three-dimensional camera or a depth camera for capturing image data and depth data of the target object. The vision device can also be a three-dimensional laser scanner for capturing point cloud data of the target object for generating a point cloud of the target object. The process of three-dimensional image capturing by the vision device can be referred to as scanning, and can be applied to any type of object and surface in the environment. The vision device can scan the target object according to a specific trajectory and at a specific frequency to obtain point cloud data. The point cloud data can then be analyzed and processed by a dedicated software tool to generate a point cloud of the target object. During operation, the vision device can capture images of the target object in real time to reflect the updated shape and / or position state of the target object, and the entire process of capturing images-processing data-generating point clouds can be continuously operated at a certain cycle period.

[0049] Step S102: establishing a virtual fixture of the robot based on the geometric feature corresponding to the point cloud; the virtual fixture includes a forbidden area virtual fixture and / or a guide virtual fixture. The forbidden area virtual fixture refers to a forbidden area for limiting the movement of the robot, and the guide virtual fixture refers to an active area for guiding the movement of the robot along a specific trajectory.

[0050] For a discrete point cloud of a specific environment and object, the point cloud of the object and surface in the environment has a specific geometric shape, and therefore the geometric shape corresponding to the point cloud can be determined based on the point cloud or point cloud data by performing feature value extraction of the point cloud through, for example, point cloud segmentation technology. Then, the geometric feature of the virtual fixture can be obtained based on the geometric shape and the actual scene, for example, the axis of the cylinder, the workbench surface, the workpiece edge, the center of the sphere, etc., to establish the virtual fixture of the robot based on the geometric feature. See Figure 2 In one embodiment, the geometric shape corresponding to the point cloud 201 of the target object can be determined to be a plane based on the point cloud 201 of the target object, and then the geometric feature of the virtual fixture can be determined to be a workbench surface 202 based on the geometric shape and the actual scene, for establishing the corresponding virtual fixture. See Figure 3 In another embodiment, the geometric shape corresponding to the point cloud of the target object can be determined to be a circle based on the point cloud of the target object, and then the geometric feature of the virtual fixture can be determined to be a hole 301 based on the geometric shape and the actual scene, for establishing the corresponding virtual fixture.

[0051] The types of virtual fixtures can include forbidden region virtual fixtures and guidance virtual fixtures. The types of virtual fixtures can be specified by an operator based on the geometric features corresponding to the point cloud and actual application, or can be calibrated based on a machine vision model or a deep learning model, and the application is not limited thereto. For example, an operator of the robot can determine in advance whether a forbidden region virtual fixture or a guidance virtual fixture is expected to be used during operation, and input a confirmation instruction into the control system of the robot, and the robot can define the virtual fixture as the type specified by the operator during the process of establishing the virtual fixture after receiving the instruction.

[0052] According to the actual application of the robot teleoperation, in one example, only forbidden region virtual fixtures can be established, for example, the robot performs obstacle avoidance tasks in the working environment. In one example, only guidance virtual fixtures can be established, for example, the robot is guided to move along a spatial trajectory line or a spatial trajectory surface to a target position. In one example, both forbidden region virtual fixtures and guidance virtual fixtures can be established, for example, the robot is guided to move along a spatial trajectory line or a spatial trajectory surface to a target position, and also performs obstacle avoidance tasks during the movement. The types and number of virtual fixtures can be set according to actual application and demand, and the application is not limited thereto.

[0053] Step 103: determining the reference point of the virtual fixture and the control point of the robot, and determining the virtual force of the virtual fixture acting on the control point according to the distance between the reference point and the control point.

[0054] The control point of the robot can be selected from the end of the actuator of the robot and / or the surface position point of the mechanical arm.

[0055] In some embodiments, the reference point of the virtual fixture can be selected from the point cloud points representing the object surface of the target object. In one example, referring to Figure 2 When the operation task of the robot is to control the end of the actuator to move from the starting position A to the target position B along the dotted trajectory on the surface of the target object, the point cloud points corresponding to the dotted trajectory representing the object surface in the target object are selected as the reference points of the guidance virtual fixture. In another example, when the operation task of the robot is to control the end of the actuator to avoid obstacles in the working environment, the point cloud points representing the object surface of the obstacles are selected as the reference points of the forbidden region virtual fixture.

[0056] In some embodiments, the reference points can be constructed based on geometric features, and the distribution of the reference points corresponds to the expected motion path or the expected obstacle avoidance path of the control point. In one example, referring to Figure 3When the operation task of the robot is to control the end of the manipulator to clamp the workpiece from the starting position C to the target position D through the hole 301 (for example, an insertion operation), the reference point 302 of the guide virtual clamp is constructed based on the geometric features corresponding to the point cloud representing the hole, that is, the central axis of the hole, and the distribution of the reference points corresponds to the central axis of the hole, so as to control the end of the manipulator to operate the workpiece along the central axis of the hole from the starting position C to the target position D. In another example, when the operation task of the robot is to control the end of the manipulator to work within the range of the workpiece and cannot exceed the edge of the workpiece, the reference points of the forbidden area virtual clamp are constructed based on the geometric features corresponding to the point cloud representing the workpiece, that is, the edge of the workpiece, so as to control the end of the manipulator not to exceed the edge of the workpiece, so as to avoid damaging or affecting other objects or operators outside the range of the workpiece.

[0057] When the reference points and the control points are selected, the virtual force of the virtual clamp acting on the control points can be determined according to the distance between the reference points and the control points. The virtual force refers to the force acting on the robot by the virtual clamp according to the rules set by humans. Since the virtual clamp does not actually exist, it will not generate actual contact force on the robot, and therefore, the virtual force is realized in the form of additional control force on the robot. When the distance between the reference points and the control points is greater than or equal to a preset value, the virtual force can be set to zero; when the distance between the reference points and the control points is less than the preset value, the smaller the distance between the reference points and the control points, the greater the virtual force. The size of the virtual force can be adjusted according to the actual application. It can also be understood that the preset value can be set in advance, and first, the distance between the reference points and the control points is compared with the preset value. If the above distance is greater than or equal to the preset value, no virtual force calculation is performed; if the above distance is less than the preset value, the size of the virtual force is calculated according to the set manner. For example, in some embodiments, the virtual force can be determined according to the following manner:

[0058]

[0059] When

[0060] When

[0061] wherein, is the distance vector of the control point to the i-th reference point, C is a constant set by humans (representing the maximum value of the virtual force of a single reference point on the control point), k is a coefficient set according to experience and test results, d ref is the range of the reference point generating the virtual force, is the virtual force vector of the i-th reference point on the control point, n is the number of all reference points affecting the control point, is the resultant force vector of the virtual force of all reference points on the control point.

[0062] In some embodiments, the spherical potential field can be determined with the reference point as the center of the sphere, and the radius of the spherical potential field is set to be greater than or equal to half the distance between the centers of adjacent spherical potential fields. When the control point is located at or outside the edge of the spherical potential field, the virtual force is zero. Here, assuming the radius of the spherical potential field is R, then C = k * R. Since the reference points are selected from the point cloud points representing the surface of the object, and the visual device usually performs scanning operations at a certain frequency, it can be considered that the distance between adjacent point cloud points is basically fixed. See Figure 4 For example, taking two adjacent first reference point 401 and second reference point 402, the first spherical potential field 411 and the second spherical potential field 412 respectively established by them are fused with each other, thereby forming a larger and partially overlapping potential field. See Figure 5 For example, taking a plurality of reference points 501, the spherical potential fields respectively established by them are fused into a continuous wall-shaped potential field 511. For example, when the virtual clamp is a forbidden area virtual clamp, the wall-shaped potential field 511 is used to operate the robot to prevent the control point from passing through the wall-shaped potential field 511 to enter the forbidden area inside the wall-shaped potential field 511. When the virtual clamp is a guide virtual clamp, the wall-shaped potential field is used to operate the robot to prevent the control point from passing through the wall-shaped potential field 511 to constrain the control point in the guide area inside the wall-shaped potential field 511.

[0063] In another way, in order to make the virtual force of the reference point on the control point continuous in the spatial range, the size of the virtual force generated by the forbidden area virtual clamp and the guide virtual clamp on the control point can be determined according to the following way respectively:

[0064] 1. Forbidden area virtual clamp

[0065]

[0066]

[0067] Wherein, K FR is a constant set by man (representing the maximum value of the virtual force of a single reference point on the control point); γ is a coefficient set by man, which affects the range of action of the virtual force of the reference point and the speed of change of the virtual force in the range; and are the positions of the control point and the reference point respectively; is the repulsive virtual force vector generated by the i-th reference point on the control point; is the resultant force vector of the repulsive virtual forces of all reference points on the control point.

[0068] By calculating the virtual force in this way, the change of the virtual force can be continuous in the spatial range, without jump, which ensures the stability of the robot control.

[0069] 2. Guide virtual clamp

[0070]

[0071]

[0072] wherein, is the virtual attractive force vector generated by the ith reference point pair to the control point; is the resultant virtual attractive force vector of all reference point pairs to the control point; other parameters are similar to the aforementioned forbidden virtual clamp.

[0073] It should be noted that in the determination of the virtual force of the guiding virtual clamp, the parameter r s is introduced, which represents a small force vacuum range set by a person with reference to the reference point. If this parameter r s is not set, the closer the control point is to the reference point, the greater the virtual attractive force it receives, and when the control point is very close to the reference point, the control point that no longer needs to be guided to the reference point still receives the greatest virtual attractive force, which may cause a greater impact on the system. Therefore, by introducing the parameter r s , when the control point enters the corresponding force vacuum range r s , the reference point no longer generates a virtual attractive force to the control point, i.e. the reference point can move freely within this range. After the control point reaches the desired position according to the guiding virtual clamp, it no longer receives the virtual guiding force of the guiding virtual clamp; and if the control point deviates from the force vacuum range for some reason, it will again receive the virtual guiding force of the virtual clamp. In this way, the reference point can be guided to the range of the guiding virtual clamp, and the system can be stable and not cause impact.

[0074] In the calculation method of the virtual force of the aforementioned forbidden virtual clamp and guiding virtual clamp, it can be seen that the virtual force is continuous in the spatial range (except the force vacuum range), which is different from the method of using d ref to determine whether the virtual force is 0. However, in the process of robot control, it can also be considered to select part of the reference points for calculating the virtual force. Referring to Figure 6 , a pre-evaluation range r e can be determined with the control point 601 as the center, and it is compared whether each reference point 602 in the point cloud falls within the pre-evaluation range r e , i.e. it is compared whether the distance between each reference point and the control point is less than r e , and only the reference points falling within the pre-evaluation range are calculated according to the aforementioned method, so as to save the calculation resources.

[0075] In some embodiments, the pre-evaluation range r e may be determined in the following way:

[0076] re = r0 + ||v e ||·k v

[0077] wherein r0 is a minimum value of a pre-evaluation range set artificially; v e is a current motion speed of the control point, k v is a coefficient set according to experience and test results. In this way, the faster the current motion speed of the control point, the larger the pre-evaluation range, so that more reference points can fall into the pre-evaluation range to participate in the calculation of the virtual force, thereby avoiding the situation that appropriate reference points cannot be introduced into the calculation in time in the case of fast movement of the robot; in addition, the smaller the current motion speed of the control point, the smaller the pre-evaluation range, so that only a few reference points can be introduced to participate in the calculation in the case of slow movement of the robot, thereby saving computing resources.

[0078] Step S104: determining a control force applied to the control point based on the virtual force.

[0079] In some embodiments, when the virtual force is zero, the virtual clamp does not need to provide additional force feedback, and the robot can output a given driving force to control the control point of the robot to move along a desired motion path or a desired obstacle avoidance path with a given control force. When the virtual force is not zero, the virtual clamp provides additional force feedback, and the robot outputs a driving force based on the force feedback and the given control force to overcome the virtual force and drive the control point to move along the desired motion path or the desired obstacle avoidance path.

[0080] Taking the virtual clamp as a forbidden area virtual clamp as an example, when the distance between the reference point and the control point is too close, the forbidden area virtual clamp will generate a repulsive force, and accordingly, a corresponding control force is applied to the control point, so as to control the robot to perform obstacle avoidance to make the control point leave the area. Taking the virtual clamp as a guide virtual clamp as an example, when the control point drifts in the vicinity of the reference point, the guide virtual clamp will generate a corresponding attractive force according to the offset distance, and accordingly, a corresponding control force is applied to the control point, so as to constrain the control point in the area and make it move along the desired trajectory line as much as possible.

[0081] When the virtual clamp is a forbidden area virtual clamp, the virtual force can be set as a repulsive force, and the direction of the repulsive force can be set as away from the reference point. When the virtual clamp is a guide virtual clamp, the virtual force can be set as an attractive force, and the direction of the attractive force can be set as pointing to the reference point. Based on this, the force direction of the virtual force presented on the human-computer interface can be used to provide intuitive visual feedback to the operator, so that the operator can control the robot to make the control point move along the desired obstacle avoidance path or the desired motion path.

[0082] In some embodiments, the virtual clamp can include a plurality of reference points, and the virtual force can be set as a virtual resultant force generated by the plurality of reference points on the control point. In one example, when there is an obstacle with a certain volume in the motion trajectory of the control point of the robot, the virtual clamp established based on the point cloud of the surface of the obstacle can include a plurality of reference points, and thus it is necessary to calculate the virtual resultant force generated by the plurality of reference points on the control point in real time, and determine the control force applied to the control point based on the virtual resultant force, so as to drive the control point to achieve trajectory deviation obstacle avoidance, and then approach the target position after avoiding the obstacle.

[0083] In some embodiments, the robot can include a plurality of control points, and the virtual force can be set as a virtual resultant force generated by the reference point on each control point. In one example, the end effector of the robot is irregularly shaped, and thus the robot can be set to have a plurality of control points. When there is a point obstacle in the motion trajectory of the end effector of the robot, the virtual clamp established based on the point cloud of the surface of the obstacle can include a reference point, and thus it is necessary to calculate the virtual resultant force generated by the reference point on each control point in real time, and determine the control force applied to each control point based on the virtual resultant force, so as to drive the control point to achieve trajectory deviation obstacle avoidance, and then approach the target position after avoiding the obstacle.

[0084] In some embodiments, before determining the virtual force of the virtual clamp acting on the control points, a preset range including all the control points and moving with the robot is determined, taking into account that the control points move along a predetermined trajectory in the working environment; a reference point is determined, at which the virtual force acting range of the virtual force enters the preset range, and the control force applied to any control point in the preset range is determined based on the virtual force of the virtual force acting on the control point at the reference point. In other words, when the virtual force acting range of any reference point enters the preset range, the control force applied to the control point in the preset range is determined based on the virtual force of the virtual force acting on the control point at the reference point; when the virtual force acting range of any reference point is located outside the preset range, the virtual force of the virtual force acting on the control point in the preset range at the reference point is not calculated. Here, the preset range can be a rectangular range, an ellipsoidal range, or a spherical range, etc. The preset range can include one or more control points. For example, a spherical range can include 3 control points, and the spherical range moves with the control points. There is an obstacle with a certain volume in the trajectory of the control points, and the virtual clamp established based on the point cloud of the surface of the obstacle includes 100 reference points. When the spherical range moves, at a certain time, the virtual force acting range of 10 reference points enters the spherical range, at which time the virtual force acting on the 3 control points in the spherical range is calculated based on the 10 reference points; in addition, the virtual force acting range of the other 90 reference points is located outside the spherical range at this time, so the virtual force acting on the 3 control points in the spherical range at the 90 reference points can not be calculated. That is, only the reference points in a certain range are activated for calculating the virtual force, and the reference points outside the certain range are not considered in the calculation process because they are too far away from the control points to interact with the control points, thereby reducing the number of virtual forces that need to be calculated simultaneously.

[0085] The spherical range can be set according to actual application. In some embodiments, the radius of the spherical range can be adjusted according to the speed of the movement of the robot, the faster the movement of the robot, the larger the radius of the spherical range. In some embodiments, the radius of the spherical range is the product of the movement speed of the spherical range and the cycle interval for adjusting the radius of the spherical range, thereby ensuring that the control points in the working environment move to a position where the reference points are not activated along the predetermined route before the next calculation program cycle arrives. Based on this, the present application can significantly reduce the number of virtual forces that need to be calculated simultaneously, thereby reducing the calculation burden, improving the calculation frequency, and providing better operation performance and safety.

[0086] The method of each of the above embodiments can be applied to the controller of the robot to operate the end effector of the robot to safely and efficiently perform tasks in a complex and variable working environment.

[0087] The method of each of the above embodiments can also be applied to a teleoperated robot that provides haptic feedback. When the teleoperated robot performs remote operation, the master robot can send the operator's control instructions to the slave robot through a communication link, and then the slave robot acts on the environment. The relevant information of the slave robot and the working environment can be fed back to the operator, so that the operator has a sense of being on the scene, thereby effectively completing the operation task. In some embodiments, the teleoperation method of the robot further includes the following steps:

[0088] Step S201: providing a master robot and a slave robot.

[0089] Step S202: establishing a virtual clamp based on a target object in the environment of the slave robot.

[0090] Step S203: adjusting the force output of the driving device of the slave robot based on the virtual force.

[0091] Step S204: providing haptic feedback to the master robot based on the virtual force.

[0092] In an exemplary robot teleoperation process, the operation task of the robot is that the operator controls the movement of the mechanical arm of the slave robot through the hand controller of the master robot, so that the mechanical arm moves from the starting position A to the target position B. When there is an obstacle in the travel trajectory of the slave robot, a forbidden area virtual clamp is established based on the geometric features of the point cloud of the obstacle and a reference point is determined. When the control point of the mechanical arm enters the virtual force action range of the reference point, the forbidden area virtual clamp generates a virtual force acting on the control point. On the one hand, based on the virtual force, the controller of the slave robot can determine the control force applied to the control point to timely adjust the travel trajectory of the control point by adjusting the force output of the driving device of the slave robot. That is, since the slave robot interacts with the working environment, the slave robot can realize timely adjustment of the control force of the control point in the actual working environment based on the virtual force, avoid the influence of communication delay on the movement of the slave robot, and improve the safety of the movement of the slave robot. On the other hand, when the forbidden area virtual clamp generates a virtual force acting on the control point, haptic feedback can be provided to the master robot based on the virtual force, which improves the performance of the operator's on-site experience and assists the operator in controlling the mechanical arm to dynamically avoid obstacles or guide the path in the working environment, thereby improving the operation performance of the master robot.

[0093] Some embodiments of the present application also provide a robot including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any of the above embodiments. The memory and the processor can be connected by a bus or other means. For example, the processor can execute the following steps:

[0094] Step S101: acquiring an image of a target object, and generating a point cloud of the target object according to the image;

[0095] Step S102: establishing a virtual clamp based on a geometric feature corresponding to the point cloud; the virtual clamp includes a forbidden area virtual clamp and / or a guide virtual clamp;

[0096] Step S103: determining a reference point of the virtual clamp and a control point of the robot, and determining a virtual force of the virtual clamp acting on the control point according to a distance between the reference point and the control point; and

[0097] Step S104: determining a control force applied to the control point based on the virtual force.

[0098] Some embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the above-mentioned embodiments. For example, the computer program can implement the following steps:

[0099] Step S101: acquiring an image of a target object, and generating a point cloud of the target object according to the image;

[0100] Step S102: establishing a virtual clamp based on a geometric feature corresponding to the point cloud; the virtual clamp includes a forbidden area virtual clamp and / or a guide virtual clamp;

[0101] Step S103: determining a reference point of the virtual clamp and a control point of the robot, and determining a virtual force of the virtual clamp acting on the control point according to a distance between the reference point and the control point; and

[0102] Step S104: determining a control force applied to the control point based on the virtual force. Other implementation details of the robot and the computer readable storage medium provided by some embodiments of the present application can be referred to the related description of the above-mentioned embodiments of the teleoperation method of the robot, which will not be described here.

[0103] In summary, the teleoperation method of the robot, the robot and the computer readable storage medium provided by the embodiments of the present application can acquire an image of a target object in real time, generate a point cloud of the target object according to the image, and establish a virtual clamp of the robot in real time based on a geometric feature corresponding to the point cloud, so as to adapt to a changing or unstructured environment in time, and overcome the technical defect that the existing preset virtual clamp can only be applied to a fixed working environment. In addition, the virtual clamp established in real time by the embodiments of the present application can process and adapt to an environment with irregular or constantly changing geometric shapes, thereby overcoming the technical limitation that the existing virtual clamp can only be constructed as a fixed shape.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in a variety of forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0105] The above embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for teleoperating a robot, characterized in that, Includes the following steps: Acquire an image of the target object, and generate a point cloud of the target object based on the image; A virtual fixture is established based on the geometric features corresponding to the point cloud; the virtual fixture includes a prohibited area virtual fixture and / or a guiding virtual fixture. Determine the reference point of the virtual gripper and the control point of the robot, and determine the virtual force exerted by the virtual gripper on the control point based on the distance between the reference point and the control point; and The control force applied to the control point is determined based on the virtual force. The step of determining the virtual force exerted by the virtual fixture on the control point based on the distance between the reference point and the control point includes: When the distance is greater than or equal to a preset value, the virtual force is zero; When the distance is less than a preset value, the smaller the distance between the reference point and the control point, the greater the virtual force. The method further includes: When the virtual fixture is a guiding virtual fixture, a force vacuum range centered on the reference point is set. When the distance is less than a preset value and the control point is outside the force vacuum range, the smaller the distance between the reference point and the control point, the greater the virtual force. When the control point is within the force vacuum range, the virtual force is zero.

2. The method according to claim 1, wherein, The method further includes: determining a spherical shape field with a reference point as the center of the sphere, wherein the radius of the spherical shape field is set to be greater than or equal to half the distance between the centers of adjacent spherical shape fields; wherein, when the control point is located outside the edge of the spherical shape field, the virtual force is zero.

3. The method according to claim 1, wherein, The reference points are selected from point cloud points that characterize the surface of the target object.

4. The method according to claim 1, wherein, The method further includes: The reference points are constructed based on the geometric features, and the distribution of the reference points corresponds to the desired motion path or desired obstacle avoidance path of the control points.

5. The method according to claim 4, wherein, The geometric features include the central axis of the cylinder, the surface of the worktable, the edge of the workpiece, or the center of the sphere.

6. The method according to claim 1, wherein, The method further includes: When the virtual fixture is a restricted area virtual fixture, the virtual force is set to a repulsive force, and the direction of the repulsive force is set to move away from the reference point; or When the virtual fixture is a guiding virtual fixture, the virtual force is set to attraction, and the direction of attraction is set to point towards the reference point.

7. The method according to claim 1, wherein, The acquisition of the image of the target object includes: using a vision device to collect image data and / or depth data of the target object in real time at a preset sampling frequency; The vision device is installed on the end effector of the robot or on a fixing device relative to the target object.

8. The method according to claim 1, wherein, The method further includes: when the virtual fixture includes multiple reference points, setting the virtual force to be the virtual resultant force generated by the multiple reference points on the control point.

9. The method according to claim 1, wherein, The method further includes: when the robot includes multiple control points, setting the virtual force as the virtual resultant force generated by the reference point on each control point.

10. The method according to claim 1, wherein, Before determining the virtual force exerted by the virtual fixture on the control point based on the distance between the reference point and the control point, the method further includes: Determine a preset range that includes all control points and moves with the robot; When the preset range enters the virtual force range of any reference point, the control force applied to the control point is determined based on the virtual force exerted by the arbitrary reference point on the control point within the preset range; and When the virtual force of any reference point is outside the preset range, the virtual force of the reference point acting on the control point within the preset range is not calculated.

11. The method according to claim 10, wherein, The preset range is a spherical range, and the method further includes: adjusting the radius of the spherical range according to the speed at which the robot moves; the faster the robot moves, the larger the radius of the spherical range.

12. The method according to claim 11, wherein, The radius of the sphere is the product of the movement speed of the sphere and the cyclic interval for adjusting the radius of the sphere.

13. The method according to claim 1, wherein, The method further includes: Provide master robots and slave robots; The virtual gripper is created based on the target object in the robot's environment; The force output of the robot's drive device is adjusted based on the virtual force; and The virtual force provides tactile feedback to the main robot.

14. A robot comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images