Teleoperation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a teleoperation method and device, electronic equipment and storage medium, and relates to the technical field of teleoperation. The method comprises: obtaining a teleoperation message, the teleoperation message carrying control information and a controller identifier; wherein the controller identifier is used to indicate a controller generating the control information; in the case that the controller identifier is a target controller identifier, controlling a target control object based on the control information to realize teleoperation on the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object. The present disclosure can distinguish the source of the control information through the controller identifier, so that the control object is only controlled by a specific controller corresponding thereto, and confusion of the control information is avoided, thereby enabling multiple controllers in the same teleoperation system to operate multiple control objects in parallel.

Description

Technical Field

[0001] This disclosure relates to the field of teleoperation technology, and in particular to a teleoperation method, apparatus, electronic device and storage medium. Background Technology

[0002] Teleoperation is a technology that allows an operator to control a machine or equipment remotely. The operator can send control messages by manipulating a controller; these messages are parsed and forwarded through communication nodes and ultimately transmitted to the specific machine or equipment, thus enabling remote operation of that machine or equipment.

[0003] In related technologies, the same remote operating system can only control one controlled object through one controller at a time, and cannot operate multiple controlled objects in parallel through multiple controllers. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a remote operation method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of the present disclosure, a teleoperation method is provided, comprising:

[0006] Obtain a remote operation message, the remote operation message carrying control information and a controller identifier; wherein, the controller identifier is used to identify the controller that generated the control information;

[0007] When the controller identifier is the target controller identifier, the target control object is controlled based on the control information to realize teleoperation of the target control object; wherein, there is a corresponding relationship between the target controller identifier and the target control object.

[0008] In some embodiments, the remote operation message carries multiple sets of control information, and each set of control information corresponds to a different controller identifier;

[0009] When the controller identifier is the target controller identifier, controlling the target controlled object based on the control information includes:

[0010] For each set of control information in multiple sets of control information, if the controller identifier corresponding to that set of control information is the target controller identifier, the target control object is controlled based on that set of control information.

[0011] In some embodiments, the control information includes: angle information of each reference component in the target control object;

[0012] The control of the target object based on the control information includes:

[0013] For each pair of adjacent reference components in the target control object, the following operations are performed respectively:

[0014] Based on the angle information of the two adjacent reference components, the rotational relationship between the two adjacent reference components is determined;

[0015] Identify the movable part in the target control object that corresponds to the two adjacent reference components; wherein the movable part includes at least the two adjacent reference components;

[0016] Based on the rotational relationship, the rotation angle of each component in the movable part is determined in order to control each component in the movable part.

[0017] In some embodiments, determining the rotational relationship between the two adjacent reference components based on the angle information of the two adjacent reference components includes:

[0018] Using one of two adjacent reference components as a reference component, the rotational relationship of the other reference component relative to the reference component is determined based on the angle information of the two adjacent reference components.

[0019] In some embodiments, determining the rotation angle of each component in the movable part based on the rotation relationship includes:

[0020] Starting from the reference component, and following the connection order of each component in the movable part, the rotation amounts of each component in its rotatable direction are sequentially superimposed to obtain the attitude information of the movable part; the attitude information is used to show the relationship between the rotation angle of each component in the movable part and the rotation relationship.

[0021] Based on the mapping relationship between the rotation relationship and the posture information, the rotation angle of each component in the movable part is determined.

[0022] In some embodiments, determining the movable part in the target control object corresponding to the two adjacent reference components includes:

[0023] In the target control object, a transition component is identified between two adjacent reference components;

[0024] The two adjacent reference components and the transition component are used as the movable part.

[0025] In some embodiments, after determining the rotation angle of each component in the movable part based on the rotation relationship, the method further includes:

[0026] For each component in the movable part, the following operations are performed:

[0027] Obtain the rotation angle of the component at the previous moment;

[0028] If the relationship between the current rotation angle of the component and the rotation angle at the previous moment meets an abnormal condition, the rotation angle of the component at the current moment is re-determined based on the rotation angle of the component at a historical moment.

[0029] In some embodiments, determining that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment satisfies an abnormal condition includes:

[0030] If the ratio between the current rotation angle of the component and the previous rotation angle is greater than a preset threshold, it is determined that the relationship between the current rotation angle and the previous rotation angle of the component meets an abnormal condition.

[0031] In some embodiments, the step of redetermining the rotation angle of the component at the current moment based on the rotation angle at a historical moment includes:

[0032] Based on the rotation angles of the component at any two historical moments, determine the reference rotation angle of the component per unit time.

[0033] Based on the reference rotation angle and the time interval between the current moment and the previous moment, the rotation angle of the component at the current moment is determined.

[0034] In some embodiments, the controller is a motion capture suit and / or motion capture gloves.

[0035] According to a second aspect of the present disclosure, a remote operation device is provided, comprising:

[0036] An acquisition module is used to acquire a teleoperation message, the teleoperation message carrying control information and a controller identifier; wherein, the controller identifier is used to identify the controller that generated the control information;

[0037] The control module is used to control the target control object based on the control information when the controller identifier is the target controller identifier, so as to realize teleoperation of the target control object; wherein, there is a corresponding relationship between the target controller identifier and the target control object.

[0038] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0039] Processor; and

[0040] Memory for storing the executable instructions of the processor;

[0041] The processor is configured to execute the method described in the first aspect by executing the executable instructions.

[0042] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0043] The solution provided in this disclosure can acquire a teleoperation message carrying control information and a controller identifier. If the controller identifier is a target controller identifier, the target control object is controlled based on the control information to achieve teleoperation of the target control object. The controller identifier indicates the controller that generated the control information, and a corresponding relationship is configured between the target controller identifier and the target control object. Based on this, this disclosure can distinguish the source of the control information through the controller identifier, ensuring that the control object is controlled only by its corresponding specific controller, avoiding confusion of control information, and enabling multiple controllers in the same teleoperation system to operate multiple control objects in parallel. Attached Figure Description

[0044] Figure 1 A schematic diagram of a teleoperation system architecture is shown in an embodiment of this disclosure.

[0045] Figure 2 A flowchart illustrating a teleoperation method according to an embodiment of this disclosure is shown.

[0046] Figure 3 A schematic diagram of the configuration interface of a remote operating system according to an embodiment of the present disclosure is shown.

[0047] Figure 4 A flowchart illustrating another teleoperation method in an embodiment of this disclosure is shown.

[0048] Figure 5 A schematic diagram of the structure of the robotic arm in an embodiment of this disclosure is shown.

[0049] Figure 6 A schematic diagram of sampling points for the motion capture suit and motion capture gloves in an embodiment of this disclosure is shown.

[0050] Figure 7 A schematic diagram of the structure of a remote operation device according to an embodiment of the present disclosure is shown.

[0051] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0053] Figure 1 A schematic diagram of a system architecture for remote operation is shown in an embodiment of this disclosure.

[0054] like Figure 1 As shown, the system architecture may include: a controller, a communication node, and a controlled object. Both the controller and the controlled object are connected to the communication node via a network, for example, via wired or wireless means.

[0055] For example, the controller can be a device with a similar shape to the controlled object, and the operator can change the controller's posture to cause the controller to generate control information. The controlled object can then mimic the controller's posture based on this control information.

[0056] For example, in the scenario of teleoperated robots, the controller can be a motion capture suit, and the controlled object can be a humanoid robot. The operator can wear the motion capture suit to perform specific actions to form control information, and the humanoid robot can imitate the operator's actions based on this control information.

[0057] In the context of autonomous driving, the controller can be a steering wheel housed in a vehicle simulation device, and the controlled object can be the steering wheel of an autonomous vehicle. The operator can generate control information by turning the steering wheel in the vehicle simulation device, and the autonomous vehicle can then rotate the steering wheel in the vehicle by the same angle based on this control information.

[0058] Of course, the controller can also be an electronic device such as a smartphone or computer. Such devices can have a client application installed, which can be an application client or a browser client. Users can use the client to adjust the posture of the controlled object in a virtual scene, thereby generating control information.

[0059] For example, the communication node can be an electronic device such as a smartphone or computer, on which a remote operation program is installed. The remote operation program can receive control information sent by the controller through the network, convert the control information into a message type that the controlled object can recognize, and then send it to the controlled object.

[0060] For example, the controlled object can be any machine or device, such as a robot, robotic arm, drive system, gripping device, etc. The controlled object can receive control information generated by the controller and perform corresponding actions.

[0061] In related technologies, since communication nodes do not have the ability to distinguish controllers, when forwarding control information, communication nodes will send each set of control information to all control objects connected to them, and the control objects will also be unable to distinguish the source of the control information.

[0062] Therefore, in the same teleoperation system, if multiple controllers are used to control multiple controlled objects at the same time, control information will be confused. Each controlled object will be used to execute control information from all controllers. Therefore, parallel teleoperation of multiple controllers cannot be achieved in related technologies.

[0063] The solution provided in this disclosure allows remote operation messages to simultaneously carry control information and a controller identifier. Even if the communication node lacks the ability to distinguish the controller, the controlled object, upon receiving the remote operation message, can determine the source of the control message based on the controller identifier carried within it. Once it is determined that the control message originates from its corresponding controller, the control information is then processed.

[0064] Based on this, the embodiments of this disclosure can ensure that the controlled object is controlled only by its corresponding specific controller in scenarios where multiple controllers perform remote operations in parallel, thus avoiding confusion of control information.

[0065] Next, the exemplary embodiments of this disclosure will be described in detail with reference to the above application scenarios.

[0066] First, this disclosure provides a teleoperation method that can be executed by a controlled object in a teleoperation scenario. For example, the controlled object can be a robot, a robotic arm, a gripping device, an autonomous vehicle, etc.

[0067] Figure 2 This diagram illustrates a flowchart of a teleoperation method according to an embodiment of the present disclosure, as shown below. Figure 2 As shown, the teleoperation method provided in this embodiment includes the following steps.

[0068] S201, Obtain remote operation message. The remote operation message carries control information and controller identifier.

[0069] The controller identifier is used to identify the controller that generates the control information.

[0070] In some embodiments, the controller identifier is unique and may be a serial number of the controller or a custom identification code. Each controller identifier is used to identify a specific controller.

[0071] In some embodiments, teleoperation messages can be sent from a communication node to a controlled object.

[0072] For example, after an operator generates control information through the controller, the controller can send the control information, along with its own controller identifier, to the communication node. Upon receiving the control information and controller identifier from the controller, the communication node can parse and integrate the control information and controller identifier, converting them into a teleoperation message that the controlled objects can recognize, and broadcasting it to each controlled object connected to the communication node.

[0073] S202, when the controller identifier is the target controller identifier, control the target controlled object based on the control information to realize teleoperation of the target controlled object.

[0074] In some embodiments, a pre-configured correspondence exists between controller identifiers and controlled objects. Based on this correspondence, when there are multiple controllers, each controlled object can only be teleoperated through the controller indicated by its corresponding controller identifier.

[0075] The correspondence between controller identifiers and controlled objects can be one-to-one or one-to-many. That is, each controller can be used to teleoperate on a specific controlled object, or it can be used to teleoperate on multiple controlled objects simultaneously.

[0076] For example, Figure 3 A schematic diagram of the configuration interface of a remote operating system according to an embodiment of this disclosure is shown. Figure 3 As shown, when the controller is a motion capture suit and the controlled object is a robot, the corresponding motion capture suit serial number (i.e., controller identifier) ​​can be selected sequentially for each robot to be operated to complete the configuration process of the correspondence between the controller identifier and the controlled object.

[0077] After the mapping relationship is configured, it can be triggered Figure 3 The various function buttons shown, such as "Enable motion capture data parsing and forwarding", "Enable joint mapping", and "Record joint data and image data", can enable the teleoperation functions of the currently configured robot.

[0078] It is worth noting that the target control object in this embodiment is the execution entity used to perform the above steps S201 and S202. The target control object can control itself based on the control information carried in the teleoperation message.

[0079] For example, when the controller identifier is a target controller identifier, the target controlled object can parse the control information corresponding to that controller identifier into control instructions. These control instructions may include operation instructions for various controllable components within the target controlled object, such as translation or rotation instructions for the controllable components. By executing these instructions, the posture of the target controlled object can be changed, enabling the target controlled object to control itself to perform specific actions.

[0080] In some embodiments, the teleoperation message carries multiple sets of control information, each set corresponding to a different controller identifier. In this case, for each set of control information, if it is determined that the controller identifier corresponding to that set of control information is the target controller identifier, then the target controlled object can be controlled based on that set of control information.

[0081] In other words, regardless of how many sets of control information are contained in the acquired teleoperation message, the target control object only focuses on the control information generated by the target controller. Control information generated by controllers other than the target controller is invalid for the target control object.

[0082] Therefore, the solution provided by this disclosure can distinguish the source of control information by carrying a controller identifier in the teleoperation message, so that the controlled object is only controlled by its corresponding specific controller, avoiding confusion of control information in parallel teleoperation scenarios, thereby enabling multiple controllers in the same teleoperation system to operate multiple controlled objects in parallel.

[0083] Next, we will combine Figure 4 This section details the process of remotely operating a target control object based on the control information carried in the teleoperation information.

[0084] In some embodiments, in order to enable teleoperation of the target control object, the control information includes at least the angle information of each reference component in the target control object.

[0085] It is understandable that the principle of teleoperation is to position a reference component within the controlled object at a specific angle, thereby causing the controlled object as a whole to exhibit the desired posture. The reference component can be understood as a component whose orientation angle is directly specified by the controller, or as a reference used to determine the rotation angle of each component within the controlled object.

[0086] For example, when the controller is a motion capture suit and the controlled object is a robot, the reference component can be a joint in the robot corresponding to the sampling point of the motion capture suit.

[0087] Specifically, taking the arm as an example, assuming the motion capture suit has three sampling points at the shoulder, elbow, and wrist, then the robot's shoulder, elbow, and wrist joints are the reference components for the robot's arm position. By telemanipulating the robot, the angles of these joints in the robot can be made consistent with the joint angles captured by the motion capture suit, thus enabling the robot to perform the same movements as the operator wearing the motion capture suit.

[0088] Figure 4 A flowchart illustrating yet another teleoperation method according to an embodiment of this disclosure is shown, with reference to... Figure 4 The teleoperation method provided in this embodiment includes the following steps.

[0089] S401, for each pair of adjacent reference components in the target control object, execute S402 to S404 respectively.

[0090] In this context, adjacent reference components can be understood as: among the various reference components of the target control object, reference components that are structurally adjacent.

[0091] For example, taking the position of a robot's arm as an example, assuming that there are three reference components on the robot's arm: the shoulder joint, the elbow joint, and the wrist joint, then the shoulder joint and the elbow joint are two adjacent reference joints, and the elbow joint and the wrist joint are adjacent reference joints.

[0092] S402, based on the angle information of two adjacent reference components, determines the rotational relationship between the two adjacent reference components.

[0093] In some embodiments, the angle information may indicate the orientation angle of a reference component in three-dimensional space. Rotational relationships are used to describe, with one reference component as a reference component, what rotations allow it to achieve the same orientation angle as another reference component.

[0094] In other words, when determining rotational relationships, one of two adjacent reference components can be used as a reference component. Then, based on the angle information of the two adjacent reference components, the rotational relationship of the other reference component relative to the reference component is determined.

[0095] For example, both angle information and rotation relationship can be represented by a matrix, and thus the following equation (1) can be obtained.

[0096] R relative = R a -1 ·R b (1)

[0097] Among them, R relative R is the rotation relation matrix. a R is the angle information matrix of the reference component in two adjacent reference components.b R is the angle information matrix of the non-reference component in two adjacent reference components. a and R b This can be obtained by parsing the teleoperation message.

[0098] S403, determine the movable part in the target control object that corresponds to two adjacent reference parts; wherein the movable part includes at least two adjacent reference parts.

[0099] It should be noted that the movable part in the embodiments of this disclosure can be understood as a minimal structure comprising two adjacent reference components. In this case, the two adjacent reference components are often located at the ends of the movable part.

[0100] Since two adjacent reference components may not be directly connected in the overall structure of the target control object, but are indirectly connected through other transition components, when determining the movable part, the transition component between the two adjacent reference components can be identified first in the target control object, and the two adjacent reference components and the transition component can be considered together as the movable part.

[0101] It is understood that the term "component" (e.g., reference component, transition component, etc.) used in the embodiments of this disclosure can be understood as a controllable rotational mechanism in the controlled object. For example, when the target controlled object is a robot, each rotatable joint of the robot can be called a "component". When the target controlled object is an autonomous vehicle, the rotational mechanism that enables the steering wheel to rotate can also be called a "component" as used in this disclosure, and this disclosure will not elaborate further on this aspect.

[0102] S404, based on the rotation relationship, determines the rotation angle of each component in the movable part in order to control each component in the movable part.

[0103] In some embodiments, starting from a reference component, the rotation amounts of each component in its rotatable direction can be sequentially superimposed according to the connection order of the components in the movable part to obtain the attitude information of the movable part. Based on the mapping relationship between the rotation relationship and the attitude information, the rotation angle of each component in the movable part is determined.

[0104] The attitude information shows the relationship between the rotation angles of each component in the movable part and the rotational relationships determined in S402. By mapping the rotational relationships determined in S402 to the representation of the attitude information, the rotation angles of each component can be solved.

[0105] For example, the amount of rotation of each component in its rotatable direction can be characterized by its rotation matrix. By multiplying the rotation matrices of each component in turn, the amount of rotation of each component in its rotatable direction can be superimposed to obtain the attitude information of the movable part.

[0106] In some embodiments, after completing S402 to S404 for every two adjacent reference components in the target control object, each component in the target control object can be controlled to rotate by a corresponding rotation angle, thereby realizing teleoperation of the target control object.

[0107] Therefore, the embodiments of this disclosure can determine the rotation angle of each component based on the physical rotational relationship between the components, thereby obtaining a rotation angle with high accuracy.

[0108] The above combination Figure 4 The teleoperation method provided in this disclosure is further explained. For ease of understanding, it will be described using... Figure 5 Taking the illustrated robotic arm structure as an example, a detailed explanation is provided below. Figure 4 The specific application process of the method shown.

[0109] For example, it can be Figure 5 The illustrated robotic arm structure serves as the target control object, and is remotely operated via a motion capture suit. The motion capture suit may have four sampling points: shoulder, upper arm, lower arm, and hand. These sampling points correspond sequentially to joints ①, ③, ⑤, and ⑦ in the robotic arm structure.

[0110] In other words, in the structure of the robotic arm, joints ①, ③, ⑤ and ⑦ are all reference components. Based on the adjacency relationship, joints ① and ③ are two adjacent reference components, joints ③ and ⑤ are two adjacent reference components, and joints ⑤ and ⑦ are two adjacent reference components.

[0111] Meanwhile, the robotic arm is also equipped with joints ②, ④ and ⑥. These joints do not have corresponding motion capture suit sampling points, and can be used as transition joints to improve the rotational freedom of the robotic arm.

[0112] To enable remote operation of the robotic arm structure, the control information carried in the remote operation message includes: shoulder angle information, upper arm angle information, lower arm angle information, and hand angle information. This information can be collected by the sampling points of the motion capture suit.

[0113] After obtaining the angle information of each reference component, firstly, regarding the two adjacent reference components, joint ① and joint ③, the rotational relationship between these two adjacent reference components can be expressed as:

[0114] R relative1 = R shoulder -1 ·R upper.arm (2)

[0115] Among them, R relative1R is the rotation relationship matrix between two adjacent reference components. shoulder R is the shoulder angle information matrix. upper.arm This is the upper arm angle information matrix.

[0116] Through observation Figure 5 It can be observed that the movable parts corresponding to joints ① and ③ include not only joints ① and ③, but also joint ② located between joints ① and ③.

[0117] Since equation (2) is obtained using the shoulder joint ① as a reference component, the posture information of the movable part should be determined starting from joint ①. Figure 5 It can be seen that, starting from the joint ① corresponding to the shoulder, the movable part will rotate around the Y-axis, around the X-axis and around the Z-axis in sequence to reach the joint ③. Therefore, the posture information of the movable part can be expressed as the following formulas (3) to (5).

[0118] R relative1 =R(Y)R(X)R(Z) (3)

[0119]

[0120] In equations (3) to (5) above, R relative1 Given the rotation relationship, R(Y), R(X), and R(Z) are the rotation matrices of joint ①, joint ②, and joint ③, respectively, and θ1, θ2, and θ3 are the rotation angles of joint ①, joint ②, and joint ③, respectively.

[0121] The rotation relation matrix R obtained by equation (2) relative1 Substituting into equation (5) and based on the trigonometric function relationship, we can obtain equations (6) to (8) to complete the solution of the rotation angles of joint ①, joint ② and joint ③.

[0122] θ1 = arctan(R) relative1 [0,2] / R relative1 [2,2]) (6)

[0123] θ2= -arcsin(R relative1 [1,2]) (7)

[0124] θ3= -arctan(R relative1 [1,0] / R relative1 [1,1]) (8)

[0125] Next, regarding the two adjacent reference components, joint ③ and joint ⑤, the rotational relationship between these two adjacent reference components can be expressed as follows:

[0126] R relative2 = Rupper.arm -1 ·R forearm (9)

[0127] Among them, R relative2 R is the rotation relationship matrix between two adjacent reference components. upper.arm R is the upper arm angle information matrix. forearm This is the lower arm angle information matrix.

[0128] Through observation Figure 5 It can be observed that the movable parts corresponding to joints ③ and ⑤ include not only joints ③ and ⑤, but also joint ④ located between joints ③ and ⑤.

[0129] Since equation (9) is obtained using joint ③ of the upper arm as a reference component, the attitude information of the movable part should be determined starting from joint ③. Figure 5 As can be seen, starting from joint ③ corresponding to the upper arm, the movable part will successively rotate around the Z-axis, rotate around the Y-axis and rotate around the Z-axis to reach joint ③. Therefore, the posture information of the movable part can be expressed as the following formulas (10) to (12).

[0130] R relative2 =R(Z)R(Y)R(Z) (10)

[0131]

[0132]

[0133] In equations (10) to (12) above, R relative2 Given the rotation relationship, R(Z), R(X), and R(Z) are the rotation matrices of joints ③, ④, and ⑤, respectively, and θ3, θ4, and θ5 are the rotation angles of joints ③, ④, and ⑤, respectively.

[0134] The rotation relation matrix R obtained by equation (9) relative2 Substituting into equation (12) and based on the trigonometric function relationship, we can obtain equations (13) to (15) to complete the solution of the rotation angles of joints ③, ④ and ⑤.

[0135] θ3 = arctan(R) relative2 [1,2] / R relative2 [0,2]) (13)

[0136] θ4= arcsin(R relative2 [2,2]) (14)

[0137] θ5= arctan(-R relative2 [2,1] / R relative2[2,0]) (15)

[0138] Finally, regarding the two adjacent reference components, joint ⑤ and joint ⑦, the rotational relationship between these two adjacent reference components can be expressed as:

[0139] R relative3 = R forearm -1 ·R hand (16)

[0140] Among them, R relative3 R is the rotation relationship matrix between two adjacent reference components. forearm R is the lower arm angle information matrix. hand This is a matrix containing hand angle information.

[0141] Through observation Figure 5 It can be observed that the movable parts corresponding to joints ⑤ and ⑦ include not only joints ⑤ and ⑦, but also joint ⑥ located between joints ⑤ and ⑦.

[0142] Since equation (16) is obtained using joint ⑤ corresponding to the lower arm as a reference component, the attitude information of the movable part should be determined starting from joint ⑤. Figure 5 As can be seen, starting from joint ⑤ corresponding to the lower arm, the movable part will successively rotate around the Z-axis, around the X-axis and around the Y-axis to reach joint ⑦. Therefore, the attitude information of the movable part can be expressed as the following equations (17) to (19).

[0143] R relative3 =R(Z)R(X)R(Y) (17)

[0144]

[0145] In equations (17) to (19) above, R relative3 Given the rotation relationship, R(Z), R(X), and R(Y) are the rotation matrices of joints ⑤, ⑥, and ⑦, respectively, and θ5, θ6, and θ7 are the rotation angles of joints ⑤, ⑥, and ⑦, respectively.

[0146] The rotation relation matrix R obtained by equation (16) relative2 Substituting into equation (19) and based on the trigonometric function relationship, we can obtain equations (20) to (22), thereby completing the solution of the rotation angles of joints ⑤, ⑥ and ⑦.

[0147] θ5 = arctan(-R) relative3 [0,1] / R relative3 [1,1]) (20)

[0148] θ6= arcsin(Rrelative3 [2,1]) (21)

[0149] θ7= arctan(-R relative3 [2,0] / R relative3 [2,2]) (22)

[0150] It is worth noting that in this embodiment, since joint ③ and joint ① are adjacent reference components, and joint ③ and joint ⑤ are also adjacent reference components, joint ③ participates in two calculations. Therefore, the final rotation angle of joint ③ is the superposition of the two calculation results, that is, the superposition of the calculation results of equation (8) and equation (13).

[0151] Based on the same principle, the final rotation angle of joint ⑤ is the superposition of equation (15) and equation (20), which will not be elaborated in this embodiment.

[0152] In some embodiments, after determining the rotation angle of each component, outlier detection can be performed on the rotation angle of each component, and the abnormal rotation angle can be corrected.

[0153] Specifically, for each component, the following operations can be performed: Obtain the rotation angle of the component at the previous moment. Determine whether the relationship between the current rotation angle of the component and the rotation angle at the previous moment meets any abnormal conditions. If so, redetermine the current rotation angle of the component based on its historical rotation angles.

[0154] For example, if the rotation angle at the current moment changes abruptly compared to the rotation angle at the previous moment, the rotation angle at the current moment can be determined to be an abnormal rotation angle.

[0155] To accurately measure this anomaly, the anomaly condition can be set as follows: the ratio between the rotation angle at the current moment and the rotation angle at the previous moment must be greater than a preset threshold. That is, the ratio between the rotation angle at the current moment and the rotation angle at the previous moment is used to measure whether a sudden change in the rotation angle has occurred.

[0156] Of course, the abnormal condition can also be set as follows: the difference between the rotation angle at the current moment and the rotation angle at the previous moment is greater than a preset threshold. That is, the difference between the two can also be used to measure whether the rotation angle has changed abruptly, and this embodiment of the present disclosure does not limit this.

[0157] For example, if the rotation angle is determined to be an abnormal rotation angle, the rotation angle can be re-determined as follows: Based on the rotation angles of the component at any two historical moments, determine the reference rotation angle of the component per unit time. Based on the reference rotation angle, and combined with the time interval between the current moment and the previous moment, determine the rotation angle of the component at the current moment.

[0158] In other words, the rotation angle at the current moment can be calculated using interpolation in the time dimension, thereby correcting abnormal rotation angles.

[0159] It is worth noting that in this example, the reference rotation angle per unit time is calculated using the rotation angles at any two historical moments, and this is used as the basis for interpolation. To make the interpolation results more accurate, the reference rotation angle per unit time can be calculated based on the rotation angles at the two historical moments closest to the current moment, so that the rotation angle obtained by the difference more closely matches the motion trend of the target controlled object.

[0160] Of course, the reference rotation angle per unit time can also be preset. For example, if the rotation of the component is approximately uniform, the rotation angle that corresponds to its rotation speed per unit time can be directly set as the reference rotation angle per unit time.

[0161] For example, assuming t1, t2, t3, and t4 are consecutive moments, and the rotation angle corresponding to each moment is θ respectively. 1 θ 2 θ 3 and θ 4 Therefore, θ can be calculated. 4 With θ 3 The ratio e between them determines the rotation angle θ at time t4. 4 Is it an abnormal rotation angle?

[0162] When e is greater than a preset threshold (for example, the preset threshold could be 2), determine the rotation angle θ at time t4. 4 This is an abnormal rotation angle. At this point, the rotation angle θ at time t4 can be recalculated using the following formula (23). 4 .

[0163]

[0164] The teleoperation method and corresponding abnormal angle judgment method provided by the embodiments of this disclosure have been described in detail above. For example, in the solution provided by this disclosure, a motion capture suit or motion capture glove can be used as a controller, or both can be used as controllers simultaneously, thereby realizing the teleoperation of the robot.

[0165] Please refer to Figure 6 , Figure 6 A schematic diagram of sampling points for the motion capture suit and motion capture gloves in this disclosure example is shown. Figure 6 As shown, when the controller includes a motion capture glove, the controller's sampling points can cover all the joints of the operator's hand, thereby enabling the robot's hand to perform fine movements (e.g., grasping small objects).

[0166] Based on the same inventive concept, this disclosure also provides a remote operation device, as shown in the following embodiment. Since the principle of this remote operation device embodiment in solving the problem is the same as that described above... Figure 2 The method embodiments shown are similar, therefore the implementation of this teleoperation device embodiment can refer to the above. Figure 2 The implementation of the method embodiments shown will not be repeated here.

[0167] Figure 7 A schematic diagram of a remote operation device according to an embodiment of this disclosure is shown. Figure 7 As shown, the remote operation device 700 includes an acquisition module 701 and a control module 702.

[0168] The acquisition module 701 is used to acquire remote operation messages, which carry control information and controller identifiers; wherein, the controller identifier is used to identify the controller that generated the control information.

[0169] The control module 702 is used to control the target control object based on control information when the controller identifier is the target controller identifier, so as to realize the teleoperation of the target control object; wherein, there is a corresponding relationship between the target controller identifier and the target control object.

[0170] In some embodiments, the remote operation message carries multiple sets of control information, each set of control information corresponding to a different controller identifier. Specifically, the control module 702 is used to control the target control object based on each set of control information, provided that the controller identifier corresponding to that set of control information is the target controller identifier.

[0171] In some embodiments, the control information includes angle information of each reference component in the target control object. Specifically, the control module 702 is configured to perform the following operations for each pair of adjacent reference components in the target control object: determining the rotational relationship between the two adjacent reference components based on their angle information; determining a movable part in the target control object corresponding to the two adjacent reference components; wherein the movable part includes at least the two adjacent reference components; and determining the rotation angle of each component in the movable part based on the rotational relationship, so as to control each component in the movable part.

[0172] In some embodiments, the control module 702 is specifically used to determine the rotational relationship of the other reference component relative to the reference component based on the angle information of the two adjacent reference components, using one of the two adjacent reference components as a reference component.

[0173] In some embodiments, the control module 702 is specifically used to, starting from the reference component, sequentially superimpose the rotation amounts of each component in its rotatable direction according to the connection order of each component in the movable part to obtain the attitude information of the movable part; the attitude information is used to show the relationship between the rotation angle and the rotation relationship of each component in the movable part; and based on the mapping relationship between the rotation relationship and the attitude information, determine the rotation angle of each component in the movable part.

[0174] In some embodiments, the control module 702 is specifically used to determine a transition component between two adjacent reference components in the target control object; and to designate the two adjacent reference components and the transition component as movable parts.

[0175] In some embodiments, the control module 702 is further configured to perform the following operations for each component in the movable part: obtain the rotation angle of the component at the previous moment; determine that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets an abnormal condition; and redetermine the rotation angle of the component at the current moment based on the rotation angle of the component at a historical moment.

[0176] In some embodiments, the control module 702 is further configured to determine that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets an abnormal condition if the ratio between the rotation angle of the component at the current moment and the rotation angle at the previous moment is greater than a preset threshold.

[0177] In some embodiments, the control module 702 is further configured to: determine a reference rotation angle of the component per unit time based on the rotation angles of the component at any two historical moments; and determine the rotation angle of the component at the current moment based on the reference rotation angle and the time interval between the current moment and the previous moment.

[0178] In some embodiments, the controller is a motion capture suit and / or motion capture gloves.

[0179] The following reference Figure 8 This describes an electronic device 800 capable of implementing embodiments of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of the embodiments disclosed herein.

[0180] like Figure 8As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processor 810, at least one memory 820, and a bus 830 connecting different system components (including memory 820 and processor 810).

[0181] The memory stores program code that can be executed by the processor 810, causing the processor 810 to perform the steps described in the "Exemplary Methods" section of this disclosure according to various exemplary embodiments of this disclosure.

[0182] In some embodiments, the processor 810 may also perform the following steps of the above method embodiments:

[0183] Obtain a remote operation message, the remote operation message carrying control information and a controller identifier; wherein, the controller identifier is used to identify the controller that generated the control information;

[0184] When the controller identifier is the target controller identifier, the target control object is controlled based on the control information to realize teleoperation of the target control object; wherein, there is a corresponding relationship between the target controller identifier and the target control object.

[0185] The memory 820 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include read-only memory (ROM) 8203.

[0186] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0187] Bus 830 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0188] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with the electronic device 800, and / or with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. Figure 8 As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0189] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0190] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this disclosure according to various exemplary embodiments of this disclosure.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A teleoperation method, characterized in that, include: Obtain a remote operation message, the remote operation message carrying control information and a controller identifier; wherein, the controller identifier is used to identify the controller that generated the control information; When the controller identifier is the target controller identifier, the target control object is controlled based on the control information to realize teleoperation of the target control object; wherein, there is a corresponding relationship between the target controller identifier and the target control object; The control information includes: angle information of each reference component in the target control object; two adjacent reference components in the target control object correspond to movable parts, and the movable parts include at least the two adjacent reference components; controlling the target control object based on the control information includes: For each component in the movable part, the following operations are performed: Obtain the rotation angle of the component at the previous moment; If the relationship between the current rotation angle of the component and the rotation angle at the previous moment meets an abnormal condition, the rotation angle of the component at the current moment is re-determined based on the rotation angle of the component at historical moments. This includes: determining a reference rotation angle of the component per unit time based on the rotation angle of the component at any two historical moments; determining the rotation angle of the component at the current moment based on the reference rotation angle and the time interval between the current moment and the previous moment; and correcting abnormal rotation angles by calculating the rotation angle at the current moment through interpolation in the time dimension.

2. The method according to claim 1, characterized in that, The remote operation message carries multiple sets of control information, and each set of control information corresponds to a different controller identifier. When the controller identifier is the target controller identifier, controlling the target controlled object based on the control information includes: For each set of control information in multiple sets of control information, if the controller identifier corresponding to that set of control information is the target controller identifier, the target control object is controlled based on that set of control information.

3. The method according to claim 1, characterized in that, The control information includes: the angle information of each reference component in the target control object; The control of the target object based on the control information includes: For each pair of adjacent reference components in the target control object, the following operations are performed respectively: Based on the angle information of the two adjacent reference components, the rotational relationship between the two adjacent reference components is determined; Identify the movable part in the target control object that corresponds to the two adjacent reference components; wherein the movable part includes at least the two adjacent reference components; Based on the rotational relationship, the rotation angle of each component in the movable part is determined in order to control each component in the movable part.

4. The method according to claim 3, characterized in that, Determining the rotational relationship between the two adjacent reference components based on their angle information includes: Using one of two adjacent reference components as a reference component, the rotational relationship of the other reference component relative to the reference component is determined based on the angle information of the two adjacent reference components.

5. The method according to claim 4, characterized in that, Determining the rotation angle of each component in the movable part based on the rotation relationship includes: Starting from the reference component, and following the connection order of each component in the movable part, the rotation amounts of each component in its rotatable direction are sequentially superimposed to obtain the attitude information of the movable part; the attitude information is used to show the relationship between the rotation angle of each component in the movable part and the rotation relationship. Based on the mapping relationship between the rotation relationship and the posture information, the rotation angle of each component in the movable part is determined.

6. The method according to claim 3, characterized in that, The step of determining the movable part in the target control object corresponding to the two adjacent reference components includes: In the target control object, a transition component is identified between two adjacent reference components; The two adjacent reference components and the transition component are used as the movable part.

7. The method according to claim 1, characterized in that, The condition that the relationship between the current rotation angle of the component and the previous rotation angle satisfies an abnormal condition includes: If the ratio between the current rotation angle of the component and the previous rotation angle is greater than a preset threshold, it is determined that the relationship between the current rotation angle and the previous rotation angle of the component meets an abnormal condition.

8. The method according to claim 1, characterized in that, The controller is a motion capture suit and / or motion capture gloves.

9. A remote operation device, characterized in that, include: An acquisition module is used to acquire a teleoperation message, the teleoperation message carrying control information and a controller identifier; wherein, the controller identifier is used to identify the controller that generated the control information; The control module is configured to control a target control object based on the control information when the controller identifier is a target controller identifier, thereby enabling teleoperation of the target control object; wherein, a correspondence is configured between the target controller identifier and the target control object; the control information includes: angle information of each reference component in the target control object; two adjacent reference components in the target control object correspond to movable parts, and the movable parts include at least the two adjacent reference components; controlling the target control object based on the control information includes: performing the following operations for each component in the movable parts. The process involves: obtaining the rotation angle of the component at the previous moment; determining that the relationship between the current rotation angle of the component and the rotation angle at the previous moment meets an abnormal condition; and re-determining the current rotation angle of the component based on its historical rotation angles, including: determining a reference rotation angle of the component per unit time based on the rotation angles of any two historical moments; determining the current rotation angle of the component based on the reference rotation angle and the time interval between the current moment and the previous moment; and correcting abnormal rotation angles by calculating the current rotation angle using interpolation in the time dimension.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images