Teleoperation method, electronic equipment and computer readable storage medium

Abstract

The invention is applicable to the field of medical equipment, and provides a teleoperation method, electronic equipment and a computer readable storage medium, the method comprises the following steps: determining a motion pose of an operation master hand, determining an expected pose of a target slave arm according to the motion pose of the operation master hand, the target slave arm being a slave arm mapped by the operation master hand, performing servo execution constraint optimization on the motion of the target slave arm based on the expected pose to obtain an optimized motion instruction; the servo execution constraint comprises at least one of position constraint, speed constraint and acceleration constraint; the motion optimization instruction is used for indicating the motion of the target slave arm, and the motion optimization instruction is sent to the target slave arm. According to the invention, the stability of master-slave mapping during teleoperation can be improved.

Description

Technical Field

[0001] This application belongs to the field of medical devices, and particularly relates to a teleoperation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of robot technology, teleoperated minimally invasive surgical robots have been widely used. Due to the heterogeneity of the master hand and the slave arm and the difference in the working space during teleoperation, a master-slave boundary problem will occur during master-slave operation, that is, the pose within the working space of the master hand is unreachable by the slave arm after master-slave mapping, resulting in a process from having a solution to having no solution and then to having a solution again during master-slave operation. If not handled, instruction jumps will occur.

[0003] Currently, the existing master-slave boundary processing usually ignores the boundary inverse solution and then truncates or modifies the Jacobian for processing. For example, when the slave arm exceeds the joint range, it will be directly modified to the upper limit or lower limit of joint movement. These two methods will make the instructions of master-slave operation discontinuous, resulting in poor stability of master-slave movement. Summary of the Invention

[0004] The embodiments of this application provide a teleoperation method, an electronic device, and a computer-readable storage medium, which can improve the stability of master-slave mapping during teleoperation.

[0005] In a first aspect, the embodiments of this application provide a teleoperation method, including:

[0006] Determine the motion pose of the operating master hand;

[0007] Determine the desired pose of the target slave arm according to the motion pose of the operating master hand, where the target slave arm is the slave arm mapped by the operating master hand;

[0008] Optimize the servo execution constraint of the motion of the target slave arm based on the desired pose to obtain an optimized motion instruction; the servo execution constraint includes at least one of a position constraint, a speed constraint, and an acceleration constraint; the optimized motion instruction is used to indicate the motion of the target slave arm;

[0009] Send the optimized motion instruction to the target slave arm.

[0010] Optionally, the determining the motion pose of the operating master hand includes:

[0011] Obtain the master hand state information of the operating master hand at the current moment; the master hand state information includes the master hand joint angle;

[0012] Determine the end pose of the operating master hand according to the master hand joint angle;

[0013] Determine the motion pose based on the end pose and the master hand state information of the master hand at the previous moment.

[0014] Optionally, when the motion of the target slave arm at the previous moment was not optimized for servo execution constraints, the optimization of the servo execution constraints for the motion of the target slave arm based on the desired pose to obtain an optimized motion command includes:

[0015] Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a first slave arm reference angle, and the first slave arm reference angle is the slave arm joint angle of the target slave arm at the current moment;

[0016] Determine an optimization target based on the slave arm pose information and the desired pose; the optimization target includes a desired velocity determined based on the slave arm pose information and the desired pose;

[0017] Determine the servo execution constraints using the first slave arm reference angle;

[0018] Optimize the motion angle of the target slave arm according to the servo execution constraints and the optimization target to obtain a primary optimized angle;

[0019] Perform a secondary optimization based on the primary optimized angle to obtain the optimized motion command; the optimized motion command includes optimized joint angles.

[0020] Optionally, when the motion of the target slave arm at the previous moment was optimized for servo execution constraints, the optimization of the servo execution constraints for the motion of the target slave arm based on the desired pose to obtain an optimized motion command includes:

[0021] Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a second slave arm reference angle, and the second slave arm reference angle is the optimized joint angle of the target slave arm at the previous moment;

[0022] Determine an optimization target based on the slave arm pose information and the desired pose; the optimization target includes a desired velocity determined based on the slave arm pose information and the desired pose;

[0023] Determine the servo execution constraints using the second slave arm reference angle;

[0024] Optimize the motion angle of the target slave arm according to the servo execution constraints and the optimization target to obtain a primary optimized angle;

[0025] Perform a secondary optimization based on the primary optimized angle to obtain the optimized motion command; the optimized motion command includes optimized joint angles.

[0026] Optionally, the re-optimization based on the initial optimization angle includes:

[0027] Updating the optimization objective according to the initial optimization angle;

[0028] Re-optimizing the initial optimization angle by using the updated optimization objective and the servo execution constraint.

[0029] Optionally, the optimization of the movement angle of the target slave arm according to the servo execution constraint and the optimization objective includes:

[0030] Obtaining a task constraint, where the task constraint includes at least one of an angular velocity constraint and a linear velocity constraint;

[0031] Optimizing the movement angle of the target slave arm by using the servo execution constraint, the task constraint, and the optimization objective.

[0032] Optionally, the optimizing the movement angle of the target slave arm by using the servo execution constraint, the task constraint, and the optimization objective includes:

[0033] When the servo execution constraint includes the velocity constraint and the task constraint includes the angular velocity constraint;

[0034] Adjusting the desired velocity;

[0035] Optimizing the movement angle of the target slave arm by using the adjusted desired velocity.

[0036] Optionally, the adjusting the desired velocity includes:

[0037] Adjusting the desired velocity according to the configuration of the target slave arm;

[0038] Or

[0039] Adjusting the desired velocity according to the velocity constraint.

[0040] In a second aspect, an embodiment of the present application provides a teleoperation device, including:

[0041] A master hand movement pose determination module, configured to determine the movement pose of the operating master hand;

[0042] A slave arm desired pose determination module, configured to determine the desired pose of a target slave arm according to the movement pose of the operating master hand, where the target slave arm is the slave arm mapped by the operating master hand;

[0043] A servo execution optimization module, configured to optimize the servo execution constraint of the movement of the target slave arm based on the expected pose to obtain an optimized movement instruction; the servo execution constraint includes at least one of a position constraint, a speed constraint, and an acceleration constraint; the optimized movement instruction is used to indicate the movement of the target slave arm;

[0044] An instruction sending module, configured to send the optimized movement instruction to the target slave arm.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the teleoperation method described in the first aspect above are implemented.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the teleoperation method described in the first aspect above are implemented.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the teleoperation method described in any item of the first aspect above.

[0048] The beneficial effects of the embodiments of the present application compared with the prior art are as follows:

[0049] By optimizing the servo execution constraint of the movement of the target slave arm to obtain an optimized movement instruction, the present application can improve the stability of the master-slave mapping. Specifically, the expected pose of the target slave arm is determined by the movement pose of the master hand in the above operation. When determining the movement instruction of the target slave arm using this expected pose, the movement of the target slave arm is optimized according to the above servo execution constraint, and the solution process of the master-slave mapping can be changed into a constrained optimization problem. Since the above constrained optimization problem is a process of continuously approaching the optimal solution, the problem of instruction jump caused by no solution can be avoided, and the continuity of the movement instruction is also ensured. At the same time, the above servo execution constraint includes at least one of a position constraint, a speed constraint, and an acceleration constraint, which means that the above optimization considers the constraints of the real physical space, can improve the accuracy of the movement instruction optimization, and obtain a more accurate optimized movement instruction. Therefore, the present application can improve the stability of the master-slave mapping while ensuring the accuracy of the movement instruction. Description of the Drawings

[0050] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 is a schematic flowchart of a teleoperation method provided by an embodiment of the present application;

[0052] Figure 2 is a schematic structural diagram of a teleoperation device provided by an embodiment of the present application;

[0053] Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0054] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0055] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0056] It should also be understood that the term "and / or" used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0058] In addition, in the description of the specification and the appended claims of this application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0059] The reference to "one embodiment" or "some embodiments" etc. described in the specification of this application means that in one or more embodiments of this application, the specific features, structures or characteristics described in connection with that embodiment are included. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0060] When performing minimally invasive surgery using a teleoperated robot, the user inputs a motion control intention (command) by manipulating the master hand, and maps the displacement, posture, orientation, etc. information of the master hand movement to the slave arm, so that the slave arm realizes the corresponding movement. During master-slave mapping, since the configurations of the master hand and the slave arm are different and the joint ranges of motion are also different, there may be no solution when calculating the mapping solution, resulting in the master-slave boundary problem, that is, the pose within the working space of the master hand cannot be reached by the slave arm after master-slave mapping. When dealing with the master-slave boundary problem by means of ignoring the boundary inverse solution and then truncating or modifying the Jacobian, the following problems will occur: The method of ignoring the boundary inverse solution can handle the case where the over-boundary is small, but it does not retain the maximum intuitiveness. And when the over-boundary is large, the expected pose of the inverse solution may be in a singularity, and further loss of intuitiveness may occur due to reasons such as the need to limit the joint speed, resulting in a large difference between the movement of the slave end and the movement of the master hand; The method of modifying the Jacobian requires multiple iterations. When the master-slave pose difference is large, the difference in joint angles output may be large, and at the same time, the problem of non-convergence may occur, resulting in discontinuity of the commands for master-slave operation, that is, the stability of master-slave mapping during teleoperation is poor.

[0061] To improve the stability of teleoperation, this application provides a teleoperation method based on servo execution constraints.

[0062] Figure 1 The flowchart of a teleoperation method provided by an embodiment of this application is shown and is described in detail as follows:

[0063] S1. Determine the motion pose of the operating master hand.

[0064] In the embodiments of the present application, the motion pose of the master operating hand refers to the spatial state information of the master operating hand during its movement, including the displacement distance during the movement of the master operating hand, the master hand pose (such as coordinates, angle orientation), etc. Among them, the movement of the master operating hand can be operated by the user or by other devices, which is not limited here.

[0065] S2. Determine the desired pose of the target slave arm according to the motion pose of the master operating hand, where the target slave arm is the slave arm mapped by the master operating hand.

[0066] In the embodiments of the present application, when controlling the movement of the slave arm through the master operating hand, it is necessary to map the motion pose of the master operating hand to the slave arm so that the slave arm can achieve the corresponding movement, that is, to realize the master-slave mapping of teleoperation. Since the master operating hand may correspond to multiple slave arms, it is necessary to determine the slave arm mapped by the master operating hand from multiple slave arms, that is, the above-mentioned target slave arm. At the same time, due to the differences in the configurations (such as link lengths) and joint movement ranges of the master operating hand and the target slave arm, the motion pose of the master operating hand is often different from the movement of the target slave arm. Therefore, it is necessary to determine the desired pose of the target slave arm according to the motion pose of the master operating hand to ensure the accuracy of the movement of the target slave arm.

[0067] In an alternative embodiment, taking a teleoperated minimally invasive surgical robot as an example, after determining the motion pose of the master operating hand, the desired pose of the target slave arm can be determined by combining the display interface corresponding to the master operating hand and the coordinate system obtained from the endoscopic imaging of the slave arm.

[0068] S3. Optimize the servo execution constraints on the movement of the target slave arm based on the above-mentioned desired pose to obtain an optimized motion instruction; the above-mentioned servo execution constraints include at least one of position constraint, speed constraint, and acceleration constraint; the above-mentioned optimized motion instruction is used to indicate the movement of the target slave arm.

[0069] In the embodiments of the present application, to avoid a large deviation between the above-mentioned desired pose and the actual movement of the target slave arm, the movement of the target slave arm is optimized by servo execution constraints. Since the servo execution constraints include at least one of position constraint, speed constraint, and acceleration constraint, it means that the movement of the target slave arm can be constrained according to the actual physical space, thereby avoiding the master-slave boundary problem in the movement of the target slave arm and improving the movement stability of the target slave arm. The above-mentioned optimized motion instruction is used to indicate the movement of the target slave arm and may include the optimized pose, such as the moving distance, joint angle, joint speed, etc.

[0070] S4. Send the above-mentioned optimized motion instruction to the above-mentioned target slave arm.

[0071] In the embodiments of the present application, by sending the optimized above-mentioned optimized motion instruction to the target slave arm, the target slave arm can perform motion based on the optimized motion instruction. Optionally, the joint angles, speeds, etc. in the above-mentioned optimized motion instruction can also be filtered to remove high-frequency noise therein and improve the smoothness of the motion of the target slave arm; or, for the driver executing the above-mentioned optimized motion instruction, corresponding gains can be set according to the specific usage scenario, so as to ensure the stability of the system in executing the motion optimization instruction.

[0072] In the present application, by optimizing the servo execution constraint of the motion of the target slave arm to obtain an optimized motion instruction, the stability of the master-slave mapping can be improved. Specifically, the expected pose of the target slave arm is determined by the motion pose of the operating master hand through the above operations. When determining the motion instruction of the target slave arm using the expected pose, the motion of the target slave arm is optimized according to the above servo execution constraint, and the solution process of the master-slave mapping can be changed into a constrained optimization problem. Since the above constrained optimization problem is a process of continuously approaching the optimal solution, the problem of instruction jump caused by no solution can be avoided, and the continuity of the motion instruction is also ensured. At the same time, the above servo execution constraint includes at least one of position constraint, speed constraint, and acceleration constraint, which means that the above optimization considers the constraints of the real physical space, can improve the accuracy of the motion instruction optimization, and obtain a more accurate optimized motion instruction. Therefore, the present application can improve the stability of the master-slave mapping while ensuring the accuracy of the motion instruction.

[0073] In the embodiments of the present application, the above-mentioned determination of the motion pose of the operating master hand includes:

[0074] Obtain the master hand state information of the above-mentioned operating master hand at the current moment; the master hand state information includes master hand joint angles;

[0075] Determine the end pose of the above-mentioned operating master hand according to the above-mentioned master hand joint angles;

[0076] Determine the above-mentioned motion pose according to the above-mentioned end pose and the master hand state information of the above-mentioned operating master hand at the previous moment.

[0077] In some embodiments, in order to ensure the continuity of the motion of the operating master hand, the pose to be mapped by the above-mentioned operating master hand at the current moment (i.e., the above-mentioned motion pose) is determined according to the master hand state information of the operating master hand at the current moment and the previous moment. For example, the end pose of the operating master hand can be determined according to the master hand joint angles of the operating master hand at the current moment. The end pose includes joint coordinates, joint angle orientations, etc. Then, combined with the corresponding master hand state information (or end pose) at the previous moment, the joint coordinates, angle orientations, etc. to be mapped are obtained.

[0078] In the embodiment of the present application, when the movement of the target slave arm at the previous moment was not optimized for servo execution constraints, the optimization of the servo execution constraints for the movement of the target slave arm based on the expected pose to obtain an optimized motion instruction includes:

[0079] Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a first slave arm reference angle, and the first slave arm reference angle is the slave arm joint angle of the target slave arm at the current moment;

[0080] Determine an optimization target according to the slave arm pose information and the expected pose; the optimization target includes an expected velocity determined according to the slave arm pose information and the expected pose;

[0081] Use the first slave arm reference angle to determine the servo execution constraints;

[0082] Optimize the movement angle of the target slave arm according to the servo execution constraints and the optimization target to obtain a primary optimized angle;

[0083] Perform secondary optimization based on the primary optimized angle to obtain the optimized motion instruction; the optimized motion instruction includes optimized joint angles.

[0084] In the embodiment of the present application, the slave arm state information includes slave arm pose information (such as movement direction, movement displacement, pose of the target slave arm, etc.) and slave arm reference angles at different moments. Among them, when the movement of the target slave arm at the previous moment was not optimized for servo execution constraints, for example, when a doctor initially uses the master manipulator to control the movement of the target slave arm, the slave arm reference angle can be the slave arm joint angle of the target slave arm at the current moment, that is, the above-mentioned first reference angle.

[0085] In some embodiments, the optimization target can be determined intuitively. The intuition can be defined as the difference between the expected pose and the actually issued pose in the master-slave mapping, and the corresponding evaluation method can be ‖V d -V c ‖, where V d represents the velocity from the slave arm pose at the previous moment (which can be determined according to the slave arm pose information) to the expected pose, that is, the above-mentioned expected velocity where T d represents the above-mentioned expected pose, T l represents the slave arm pose at the previous moment, and V c represents the velocity from the slave arm pose at the previous moment to the actually issued pose. Assuming that the intuition is maximized according to the joint angle of the target slave arm, the above-mentioned optimization target can be where J represents the Jacobian, Represents the speed corresponding to the angle of the slave arm joint. After obtaining the speed corresponding to the optimized joint angle through the above optimization objective, the optimized joint angle can be obtained by integration.

[0086] In an alternative embodiment, assuming that the above servo execution constraints include position constraints, speed constraints, and acceleration constraints at the same time, the above servo execution constraints include:

[0087]

[0088] Where θ represents the angle of the slave arm joint (i.e., the first slave arm reference angle), θ l 、θ u Represents the upper and lower limits of the movement of the slave arm joint angle; Represents the speed corresponding to the angle of the slave arm joint, Represents the upper and lower limits of the speed corresponding to the angle of the slave arm joint; Represents the acceleration corresponding to the angle of the slave arm joint, Represents the upper and lower limits of the acceleration corresponding to the angle of the slave arm joint.

[0089] In some embodiments, the movement angle of the target slave arm is optimized according to the above servo execution constraints and optimization objective to obtain a primary optimized angle. In order to improve the accuracy of the movement of the target slave arm, it can be re-optimized based on the above primary optimized angle and servo execution constraints to obtain an optimized joint angle.

[0090] In another embodiment of the present application, in the case where the movement of the above target slave arm is optimized for servo execution constraints at the previous moment, the optimization of the movement of the above target slave arm based on the above expected pose for servo execution constraints to obtain an optimized movement instruction includes:

[0091] Determine the slave arm state information of the above target slave arm at the current moment; the above slave arm state information includes slave arm pose information and a second slave arm reference angle, and the above second slave arm reference angle is the optimized joint angle of the above target slave arm at the previous moment;

[0092] Determine an optimization objective according to the above slave arm pose information and the above expected pose; the above optimization objective includes an expected speed determined according to the above slave arm pose information and the above expected pose;

[0093] Use the above second slave arm reference angle to determine the above servo execution constraints;

[0094] Optimize the movement angle of the above target slave arm according to the above servo execution constraints and the above optimization objective to obtain a primary optimized angle;

[0095] Perform re-optimization based on the above primary optimized angle to obtain the above optimized movement instruction; the above optimized movement instruction includes an optimized joint angle.

[0096] In some embodiments, in the case where the motion of the target slave arm at the previous moment is optimized by servo execution constraint, for example, when the doctor continuously uses the operating master hand to control the motion of the target slave arm, the above-mentioned slave arm reference angle may be the optimized joint angle of the above-mentioned target slave arm at the previous moment, that is, the above-mentioned second reference angle. The rest of the optimization process is similar to that in the above embodiments and will not be elaborated here.

[0097] In the embodiments of the present application, the re-optimization based on the above-mentioned initially optimized angle includes:

[0098] Updating the above-mentioned optimization target according to the above-mentioned initially optimized angle;

[0099] Re-optimizing the above-mentioned initially optimized angle by using the updated above-mentioned optimization target and the above-mentioned servo execution constraint.

[0100] In some embodiments, after obtaining the initially optimized angle, the optimization target can be updated according to the above-mentioned initially optimized angle, that is, the Jacobian in the optimization target (such as dividing the initially optimized angle by the link length of the target slave arm) and the desired speed are updated according to the initially optimized angle, so as to obtain a new optimization target. Re-optimizing the initially optimized angle by using the new optimization target and the servo execution constraint can improve the accuracy of the joint angle optimization of the target slave arm motion. At the same time, in order to avoid too many iterations, some common convex optimization tools can also be used to improve the optimization efficiency.

[0101] In another alternative embodiment of the present application, when re-optimizing by using the servo execution constraint again, the constraint conditions of the above-mentioned servo execution constraint can also be relaxed to improve the efficiency of the angle optimization of the target slave arm. For example, the speed constraint in the servo execution constraint can be simplified to:

[0102]

[0103] where θ represents the slave arm joint angle (which can be the initially optimized angle during re-optimization), θ l 、θ u represent the upper and lower limits of the movement of the slave arm joint angle; represents the speed corresponding to the slave arm joint angle, represents the upper and lower limits of the speed corresponding to the slave arm joint angle; represents the acceleration corresponding to the slave arm joint angle, represents the upper and lower limits of the acceleration corresponding to the slave arm joint angle, and T represents the period of the actually issued command.

[0104] It should be noted that when obtaining the initially optimized angle, that is, during the first optimization, the above-mentioned relaxed servo execution constraint (that is, the servo execution constraint with reduced constraint conditions) can also be used to improve the efficiency of the angle optimization of the target slave arm.

[0105] In the embodiments of the present application, optimizing the motion angle of the target slave arm according to the above servo execution constraints and the above optimization objectives includes:

[0106] Obtain task constraints, where the task constraints include at least one of angular velocity constraints and linear velocity constraints;

[0107] Optimize the motion angle of the target slave arm by using the above servo execution constraints, the above task constraints, and the above optimization objectives.

[0108] In some embodiments, to improve the safety and stability of teleoperation, task constraints can be added according to actual needs, including at least one of angular velocity constraints and linear velocity constraints. For example, when a doctor performs teleoperation, in order to avoid inconvenience or safety problems caused by excessive speed of the end of the slave arm, angular velocity constraints and / or linear velocity constraints of the motion of the slave arm can be set.

[0109] In an alternative embodiment, assuming that the task constraints include both angular velocity constraints and linear velocity constraints at the same time, the total constraint conditions can be:

[0110]

[0111] Among them, the first three represent servo execution constraints, which respectively represent position constraints, velocity constraints, and acceleration constraints, corresponding to the capabilities of servo execution, that is, actual physical constraints; the latter two are task constraints, which are actually Cartesian space constraints, represents the Cartesian angular velocity constraint, J w represents the angular velocity Jacobian, w m represents the maximum angular velocity; represents the Cartesian linear velocity constraint, J v represents the linear velocity Jacobian, v m represents the maximum linear velocity. It should be noted that the above servo execution constraints can also relax the limiting conditions, which will not be elaborated here.

[0112] In another alternative embodiment, the desired velocity can also be directly scaled to meet the Cartesian space constraints. For example, it can be scaled by the method of desired velocity × scaling ratio, and the above scaling ratio is min{the maximum angular velocity divided by the first three norms of the desired velocity, 1, the maximum linear velocity divided by the last three norms of the desired velocity}.

[0113] In another alternative embodiment of the present application, the above optimizing the motion angle of the target slave arm by using the above servo execution constraints, the above task constraints, and the above optimization objectives includes:

[0114] In the case where the above servo execution constraint includes the above speed constraint and the above task constraint includes the above angular velocity constraint;

[0115] Adjust the above desired speed;

[0116] Optimize the motion angle of the above target slave arm using the adjusted above desired speed.

[0117] Specifically, when optimizing the motion angle of the target slave arm using the servo execution constraint, task constraint, and optimization objective, there are too many constraints on the target slave arm. To improve the efficiency of optimizing the target slave arm, the desired speed can be directly adjusted to meet the speed constraint and angular velocity constraint at the end of the slave arm, so that the optimization of the master-slave mapping can be directly completed in the speed space.

[0118] In the embodiments of the present application, the above adjustment of the above desired speed includes:

[0119] Adjust the above desired speed according to the configuration of the above target slave arm;

[0120] Or

[0121] Adjust the above desired speed according to the above speed constraint.

[0122] In some embodiments, since the association between the above desired speed and the optimization objective is relatively close, adjusting the desired speed can achieve the adjustment of the corresponding optimization objective, and then optimize the motion of the target slave arm through the servo execution constraint. Therefore, the desired speed can be scaled according to the configuration of the target slave arm, including information such as the length of the link in the target slave arm, to meet the speed constraint and angular velocity constraint, and at the same time, the desired speed is further scaled in combination with the mapping relationship of the Jacobian to meet the maximum joint speed constraint. At this time, the acceleration constraint can be not considered, thereby reducing the conditions of the servo execution constraint and improving the efficiency of optimizing the motion joint angle of the target slave arm. Or, after calculating the joint angle (speed), it can be directly corrected to meet the acceleration constraint. The position constraint for the joints of the target slave arm can be obtained by truncating the desired speed according to the speed constraint corresponding to the joint position and boundary position of the target slave arm, thereby also reducing the conditions of the servo execution constraint and improving the optimization efficiency of the target slave arm.

[0123] It should be understood that the magnitudes of the sequence numbers of the above steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0124] Corresponding to the above remote operation method in the above embodiments, Figure 2 The structural schematic diagram of the remote operation device provided by the embodiments of the present application is shown. For the sake of convenience of description, only the parts related to the embodiments of the present application are shown.

[0125] Reference Figure 2 , the device may be a teleoperation device 21, and the teleoperation device 21 may include a master hand motion pose determination module 211, a slave arm desired pose determination module 212, a servo execution optimization module 213, and an instruction sending module 214.

[0126] Reference Figure 2 , the teleoperation device 21 includes:

[0127] The master hand motion pose determination module 211 is used to determine the motion pose of the operating master hand;

[0128] The slave arm desired pose determination module 212 is used to determine the desired pose of the target slave arm according to the motion pose of the operating master hand, and the target slave arm is the slave arm mapped by the operating master hand;

[0129] The servo execution optimization module 213 is used to optimize the servo execution constraint of the motion of the target slave arm based on the desired pose to obtain an optimized motion instruction; the servo execution constraint includes at least one of a position constraint, a speed constraint, and an acceleration constraint; the optimized motion instruction is used to instruct the motion of the target slave arm;

[0130] The instruction sending module 214 is used to send the optimized motion instruction to the target slave arm.

[0131] In some embodiments, the slave arm desired pose determination module 212 determines the motion pose of the operating master hand through the following steps, including:

[0132] Obtain the master hand state information of the operating master hand at the current moment; the master hand state information includes master hand joint angles;

[0133] Determine the end pose of the operating master hand according to the master hand joint angles;

[0134] Determine the motion pose according to the end pose and the master hand state information of the operating master hand at the previous moment.

[0135] In some embodiments, when the servo execution constraint of the motion of the target slave arm has not been optimized at the previous moment, the servo execution optimization module 213 optimizes the servo execution constraint of the motion of the target slave arm based on the desired pose through the following steps to obtain an optimized motion instruction, including:

[0136] Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a first slave arm reference angle, and the first slave arm reference angle is the slave arm joint angle of the target slave arm at the current moment;

[0137] Determine an optimization objective based on the slave arm pose information and the desired pose; the optimization objective includes a desired velocity determined based on the slave arm pose information and the desired pose.

[0138] Use the first slave arm reference angle to determine the servo execution constraint.

[0139] Optimize the motion angle of the target slave arm according to the servo execution constraint and the optimization objective to obtain a preliminary optimized angle.

[0140] Perform a secondary optimization based on the preliminary optimized angle to obtain the optimized motion instruction; the optimized motion instruction includes optimized joint angles.

[0141] In some embodiments, when optimizing the servo execution constraint for the motion of the target slave arm at the previous moment, the servo execution optimization module 213 optimizes the servo execution constraint for the motion of the target slave arm based on the desired pose through the following steps to obtain an optimized motion instruction, including:

[0142] Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a second slave arm reference angle, and the second slave arm reference angle is the optimized joint angle of the target slave arm at the previous moment.

[0143] Determine an optimization objective based on the slave arm pose information and the desired pose; the optimization objective includes a desired velocity determined based on the slave arm pose information and the desired pose.

[0144] Use the second slave arm reference angle to determine the servo execution constraint.

[0145] Optimize the motion angle of the target slave arm according to the servo execution constraint and the optimization objective to obtain a preliminary optimized angle.

[0146] Perform a secondary optimization based on the preliminary optimized angle to obtain the optimized motion instruction; the optimized motion instruction includes optimized joint angles.

[0147] In some embodiments, the servo execution optimization module 213 performs a secondary optimization based on the preliminary optimized angle through the following steps, including:

[0148] Update the optimization objective according to the preliminary optimized angle.

[0149] Perform a secondary optimization on the preliminary optimized angle using the updated optimization objective and the servo execution constraint.

[0150] In some embodiments, the servo execution optimization module 213 optimizes the movement angle of the target slave arm according to the servo execution constraint and the optimization objective through the following steps, including:

[0151] Obtain task constraints, where the task constraints include at least one of angular velocity constraint and linear velocity constraint;

[0152] Optimize the movement angle of the target slave arm by using the servo execution constraint, the task constraint and the optimization objective.

[0153] In some embodiments, the servo execution optimization module 213 optimizes the movement angle of the target slave arm by using the servo execution constraint, the task constraint and the optimization objective through the following steps, including:

[0154] In the case where the servo execution constraint includes the velocity constraint and the task constraint includes the angular velocity constraint;

[0155] Adjust the desired velocity;

[0156] Optimize the movement angle of the target slave arm by using the adjusted desired velocity.

[0157] In some embodiments, the servo execution optimization module 213 adjusts the desired velocity through the following steps, including:

[0158] Adjust the desired velocity according to the configuration of the target slave arm;

[0159] Or

[0160] Adjust the desired velocity according to the velocity constraint.

[0161] It should be noted that for the information interaction, execution process, etc. between the devices / units, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought, please refer to the method embodiment part specifically, and details will not be repeated here.

[0162] Figure 3 This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 3 shown, the electronic device 3 in this embodiment includes: at least one processor 30 ( Figure 3 only one is shown here), a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the method embodiments are implemented.

[0163] The electronic device 3 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art can understand that Figure 3 merely examples of the electronic device 3, which do not constitute a limitation on the electronic device 3, may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device may further include an input and sending device, a network access device, a bus, etc.

[0164] The so-called processor 30 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] The memory 31 may be an internal storage unit of the electronic device 3 in some embodiments, such as the hard disk or memory of the electronic device 3. The memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 3. Further, the memory 31 may also include both the internal storage unit and the external storage device of the electronic device 3. The memory 31 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The memory 31 may also be used to temporarily store data that has been sent or will be sent.

[0166] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the various functional units and modules is used as an example. In actual applications, the functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and do not limit the protection scope of this application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments and will not be elaborated here.

[0167] An embodiment of this application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the foregoing method embodiments are implemented.

[0168] An embodiment of this application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in each of the foregoing method embodiments can be implemented.

[0169] An embodiment of this application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to implement the steps in each of the foregoing method embodiments.

[0170] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the method of the above embodiments in this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, USB flash drive, mobile hard disk, magnetic disk or optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be electrical carrier signal and telecommunication signal.

[0171] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0172] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0173] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0174] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A teleoperation method, characterized in that, Including: Determine the motion pose of the master operating hand; Determine the desired pose of the target slave arm according to the motion pose of the master operating hand, where the target slave arm is the slave arm mapped by the master operating hand; Optimize the motion of the target slave arm based on the desired pose by servo execution constraints to obtain an optimized motion instruction; the servo execution constraints include at least one of position constraint, speed constraint, and acceleration constraint; the optimized motion instruction is used to indicate the motion of the target slave arm; Send the optimized motion instruction to the target slave arm.

2. The teleoperation method according to claim 1, wherein The determining the motion pose of the master operating hand includes: Obtain the master hand state information of the master operating hand at the current moment; the master hand state information includes master hand joint angles; Determine the end pose of the master operating hand according to the master hand joint angles; Determine the motion pose according to the end pose and the master hand state information of the master operating hand at the previous moment.

3. The teleoperation method according to claim 1, characterized in that, In the case where the motion of the target slave arm was not optimized by servo execution constraints at the previous moment, the optimizing the motion of the target slave arm based on the desired pose by servo execution constraints to obtain an optimized motion instruction includes: Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a first slave arm reference angle, and the first slave arm reference angle is the slave arm joint angle of the target slave arm at the current moment; Determine an optimization target according to the slave arm pose information and the desired pose; the optimization target includes a desired speed determined according to the slave arm pose information and the desired pose; Determine the servo execution constraints using the first slave arm reference angle; Optimize the motion angle of the target slave arm according to the servo execution constraints and the optimization target to obtain a primary optimized angle; Perform secondary optimization based on the primary optimized angle to obtain the optimized motion instruction; the optimized motion instruction includes optimized joint angles.

4. The teleoperation method according to claim 3, wherein In the case where the motion of the target slave arm was optimized by servo execution constraints at the previous moment, the optimizing the motion of the target slave arm based on the desired pose by servo execution constraints to obtain an optimized motion instruction includes: Determine the slave arm state information of the target slave arm at the current moment; the slave arm state information includes slave arm pose information and a second slave arm reference angle, and the second slave arm reference angle is the optimized joint angle of the target slave arm at the previous moment; Determine an optimization target according to the slave arm pose information and the desired pose; the optimization target includes a desired speed determined according to the slave arm pose information and the desired pose; Determine the servo execution constraints using the second slave arm reference angle; Optimize the motion angle of the target slave arm according to the servo execution constraints and the optimization target to obtain a primary optimized angle; Perform secondary optimization based on the primary optimized angle to obtain the optimized motion instruction; the optimized motion instruction includes optimized joint angles.

5. The teleoperation method according to claim 3, wherein The performing secondary optimization based on the primary optimized angle includes: Update the optimization target according to the primary optimized angle; Re-optimize the initial optimized angle by using the updated optimization objective and the servo execution constraint.

6. The teleoperation method according to claim 3, wherein Optimizing the motion angle of the target slave arm according to the servo execution constraint and the optimization objective includes: Obtain a task constraint, where the task constraint includes at least one of an angular velocity constraint and a linear velocity constraint; Optimize the motion angle of the target slave arm by using the servo execution constraint, the task constraint, and the optimization objective.

7. The teleoperation method according to claim 6, wherein The optimizing the motion angle of the target slave arm by using the servo execution constraint, the task constraint, and the optimization objective includes: In the case where the servo execution constraint includes the velocity constraint and the task constraint includes the angular velocity constraint; Adjust the desired velocity; Optimize the motion angle of the target slave arm by using the adjusted desired velocity.

8. The teleoperation method according to claim 7, characterized in that The adjusting the desired velocity includes: Adjust the desired velocity according to the configuration of the target slave arm; Or Adjust the desired velocity according to the velocity constraint.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images