Teleoperation method and device, electronic equipment and storage medium

Abstract

The invention provides a teleoperation method and device, electronic equipment and a storage medium, and relates to the technical field of teleoperation. The method comprises the following steps: acquiring a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used for showing the controller generating the control information; and under the condition that the controller identifier is the target controller identifier, controlling the target control object based on the control information so as to realize teleoperation for the target control object, wherein a corresponding relationship is configured between the target controller identifier and the target control object. According to the embodiment of the invention, the source of the control information can be distinguished through the controller identifier, so that the control object is only controlled by the corresponding specific controller, the confusion of the control information is avoided, and a plurality of controllers in the same teleoperation system can operate a plurality of control objects in parallel.

Description

Technical Field

[0001] The present disclosure relates to the field of remote operation technology, and in particular to a remote operation method, device, electronic equipment and storage medium. Background Art

[0002] Teleoperation is a technology that allows operators to remotely control machines or equipment. The operator can send control information by operating the controller, and the control message is parsed and forwarded by the communication node, and finally transmitted to a specific machine or equipment to achieve remote operation of the machine or equipment.

[0003] In the related art, the same teleoperation system can only control one control object through one controller at the same time, and cannot operate multiple control objects in parallel through multiple controllers. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a remote operation method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a remote operation method, comprising:

[0006] Acquire a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used to indicate the controller that generates the control information;

[0007] In the case where the controller identifier is a target controller identifier, the target control object is controlled based on the control information to implement remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

[0008] In some embodiments, the control information carried by the teleoperation message is multiple groups, and each group of control information corresponds to a different controller identifier;

[0009] When the controller identifier is a target controller identifier, controlling the target control object based on the control information includes:

[0010] For each set of control information in the multiple sets of control information, when the controller identifier corresponding to the set of control information is the target controller identifier, the target control object is controlled based on the set of control information.

[0011] In some embodiments, the control information includes: angle information of each reference component in the target control object;

[0012] The controlling the target control object based on the control information includes:

[0013] For every two adjacent reference components in the target control object, the following operations are performed respectively:

[0014] Determining a rotational relationship between the two adjacent reference components based on the angle information of the two adjacent reference components;

[0015] Determining a movable part in the target control object corresponding to the two adjacent reference parts; wherein the movable part at least includes the two adjacent reference parts;

[0016] Based on the rotational relationship, the rotation angle of each component in the movable part is determined so as to control each component in the movable part.

[0017] In some embodiments, determining the rotational relationship between the two adjacent reference components based on the angle information of the two adjacent reference components includes:

[0018] One of the two adjacent reference components is used as a reference component, and the rotation relationship of the other reference component relative to the reference component is determined based on the angle information of the two adjacent reference components.

[0019] In some embodiments, determining the rotation angle of each component in the movable part based on the rotation relationship includes:

[0020] Taking the reference component as the starting point, according to the connection order of the components in the movable part, the rotation amounts of the components in their rotatable directions are sequentially superimposed to obtain the posture information of the movable part; the posture information is used to show the relationship between the rotation angles of the components in the movable part and the rotation relationship;

[0021] Based on the mapping relationship between the rotation relationship and the posture information, the rotation angle of each component in the movable part is determined.

[0022] In some embodiments, the determining of the movable parts in the target controlled object corresponding to the two adjacent reference parts includes:

[0023] determining a transition component between two adjacent reference components in the target control object;

[0024] Two adjacent reference components and the transition component are used as the movable part.

[0025] In some embodiments, after determining the rotation angle of each component in the movable part based on the rotation relationship, the method further includes:

[0026] For each component in the movable part, perform the following operations respectively:

[0027] Get the rotation angle of the component at the last moment;

[0028] It is determined that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets an abnormal condition, and based on the rotation angle of the component at the historical moment, the rotation angle of the component at the current moment is re-determined.

[0029] In some embodiments, determining that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment satisfies an abnormal condition includes:

[0030] When the ratio between the rotation angle of the component at the current moment and the rotation angle at the previous moment is greater than a preset threshold, it is determined that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets the abnormal condition.

[0031] In some embodiments, re-determining the rotation angle of the component at the current moment based on the rotation angle of the component at the historical moment includes:

[0032] Based on the rotation angles of the component at any two historical moments, determining a reference rotation angle of the component per unit time;

[0033] Based on the reference rotation angle and in combination with the time interval between the current moment and the previous moment, the rotation angle of the component at the current moment is determined.

[0034] In some embodiments, the controller is a motion capture suit and / or motion capture gloves.

[0035] According to a second aspect of an embodiment of the present disclosure, there is provided a remote operation device, comprising:

[0036] An acquisition module, used for acquiring a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used for indicating the controller that generates the control information;

[0037] A control module is used to control a target control object based on the control information when the controller identifier is a target controller identifier, so as to realize remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

[0038] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0039] Processor; and

[0040] A memory, configured to store executable instructions of the processor;

[0041] The processor is configured to execute the method described in the first aspect by executing the executable instructions.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The solution provided by the embodiment of the present disclosure can obtain a teleoperation message carrying control information and a controller identifier, and when the controller identifier is a target controller identifier, the target control object is controlled based on the control information to achieve teleoperation for the target control object. Among them, the controller identifier is used to indicate the controller that generates the control information, and a corresponding relationship is configured between the target controller identifier and the target control object. Based on this, the embodiment of the present disclosure can distinguish the source of the control information through the controller identifier, so that the control object is only controlled by the specific controller corresponding to it, avoiding confusion of the control information, so that multiple controllers in the same teleoperation system can operate multiple control objects in parallel. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a remote operation system architecture in an embodiment of the present disclosure is shown.

[0045] Figure 2 A flowchart of a remote operation method in an embodiment of the present disclosure is shown.

[0046] Figure 3 A schematic diagram of a configuration interface of a remote operation system in an embodiment of the present disclosure is shown.

[0047] Figure 4 A flowchart of another remote operation method in an embodiment of the present disclosure is shown.

[0048] Figure 5 A schematic diagram of the structure of the robotic arm in an embodiment of the present disclosure is shown.

[0049] Figure 6 A schematic diagram of sampling points of a motion capture suit and motion capture gloves in an embodiment of the present disclosure is shown.

[0050] Figure 7 A schematic structural diagram of a remote operation device in an embodiment of the present disclosure is shown.

[0051] Figure 8 A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0053] Figure 1 A schematic diagram of a system architecture applied to remote operation in an embodiment of the present disclosure is shown.

[0054] like Figure 1 As shown, the system architecture may include: a controller, a communication node, and a control object. The controller and the control object are both connected to the communication node via a network, for example, in a wired or wireless manner.

[0055] For example, the controller may be a device having a similar shape to the controlled object, and the operator may change the posture of the controller to make the controller generate control information, and the controlled object may imitate the posture of the controller according to the control information.

[0056] For example, in the scenario of remotely controlling a robot, the controller may be a motion capture suit, and the controlled object may be a humanoid robot. The operator can wear the motion capture suit and make specific movements to form control information, and the humanoid robot can imitate the operator's movements based on the control information.

[0057] For the unmanned driving scenario, the controller may be a steering wheel in the vehicle simulation device, and the controlled object may be the steering wheel of the unmanned vehicle. The operator may generate control information by turning the steering wheel in the vehicle simulation device, and the unmanned vehicle may rotate the steering wheel in the vehicle by the same angle according to the control information.

[0058] Of course, the controller may also be an electronic device such as a smart phone or a computer, in which a client may be provided, and the client may be an application client or a browser client, etc. The user may adjust the posture of the control object in a virtual scene through the client, thereby generating control information.

[0059] For example, the communication node may be an electronic device such as a smart phone or a computer, on which a teleoperation program is installed. The teleoperation program may receive control information sent by the controller through the network, convert the control information into a message type that can be recognized by the controlled object, and send it to the controlled object.

[0060] Exemplarily, the control object may be any machine or device, such as a robot, a mechanical arm, a drive system, a gripping device, etc. The control object may receive control information generated by the controller and perform corresponding actions.

[0061] In the related art, since the communication node does not have the ability to distinguish the controller, when forwarding control information, the communication node will send each set of control information to all control objects connected to it, and the control object side is also unable to distinguish the source of the control information.

[0062] Therefore, in the same teleoperation system, if multiple controllers are used to control multiple control objects respectively, control information confusion will occur, and each control object will be used to execute control information from all controllers. Therefore, parallel teleoperation of multiple controllers cannot be achieved in related technologies.

[0063] The solution provided by the embodiment of the present disclosure can make the teleoperation message carry both control information and controller identification. Even if the communication node does not have the ability to distinguish controllers, after the control object obtains the teleoperation message, it can determine the source of the control message according to the controller identification carried therein. When it is determined that the control message comes from the controller corresponding to it, the control information is processed.

[0064] Based on this, in the scenario where multiple controllers are remotely operated in parallel, the embodiment of the present disclosure can ensure that the controlled object is only controlled by the specific controller corresponding to it, thereby avoiding confusion of control information.

[0065] Next, exemplary embodiments of the present disclosure will be described in detail in conjunction with the above application scenarios.

[0066] First, a teleoperation method is provided in an embodiment of the present disclosure, and the method can be executed by a control object in a teleoperation scene. For example, the control object can be a robot, a mechanical arm, a gripping device, an unmanned vehicle, etc.

[0067] Figure 2 A schematic diagram of a remote operation method in an embodiment of the present disclosure is shown. Figure 2 As shown, the remote operation method provided in the embodiment of the present disclosure includes the following steps.

[0068] S201, obtaining a remote operation message, where the remote operation message carries control information and a controller identifier.

[0069] The controller identifier is used to indicate the controller that generates the control information.

[0070] In some embodiments, the controller identifier is unique and may be a serial number of the controller or a custom identification code. Each controller identifier is used to identify a specific controller.

[0071] In some embodiments, teleoperation messages may be sent by a communication node to a controlled object.

[0072] For example, after the operator generates control information by operating the controller, the controller can send the control information together with its own controller identification to the communication node. After receiving the control information and controller identification sent by the controller, the communication node can parse and integrate the control information and controller identification, convert them into remote operation messages that can be recognized by the control object, and broadcast them to each control object connected to the communication node.

[0073] S202: When the controller identifier is a target controller identifier, control the target control object based on the control information to implement remote operation on the target control object.

[0074] In some embodiments, a corresponding relationship is preconfigured between the controller identifier and the control object. Based on the corresponding relationship, when there are multiple controllers, each control object can be remotely operated only by the controller indicated by the controller identifier corresponding to it.

[0075] The corresponding relationship between the controller identifier and the control object can be one-to-one or one-to-many. That is, each controller can be used to remotely operate a specific control object or to remotely operate multiple control objects at the same time.

[0076] For example, Figure 3 FIG. 2 shows a schematic diagram of a configuration interface of a remote operation system in an embodiment of the present disclosure. Figure 3 As shown, when the controller is a motion capture suit and the controlled object is a robot, the corresponding motion capture suit serial number (i.e., controller identifier) ​​can be selected in sequence for each robot to be operated to complete the configuration process of the corresponding relationship between the controller identifier and the controlled object.

[0077] After the corresponding relationship is configured, you can trigger Figure 3 The various function buttons shown in, for example, "Start capturing data parsing and forwarding", "Start joint mapping", "Joint data and image data recording", etc., can turn on the teleoperation function of the currently configured robot.

[0078] It is worth noting that the target control object in the embodiment of the present disclosure is the execution subject for executing the above steps S201 and S202. The target control object can control itself based on the control information carried in the teleoperation message.

[0079] Exemplarily, when the controller identifier is a target controller identifier, the target control object may parse the control information corresponding to the controller identifier into a control instruction, which may include an operation instruction for each controllable component in the target control object, such as a translation instruction or a rotation instruction for the controllable component. By executing these instructions, the posture of the target control object may be changed so that the target control object controls itself to perform a specific action.

[0080] In some embodiments, the control information carried by the teleoperation message is multiple groups, and each group of control information corresponds to a different controller identifier. At this time, for each group of control information, when the controller identifier corresponding to the group of control information is determined to be the target controller identifier, the target control object can be controlled based on the group of control information.

[0081] That is to say, no matter how many groups of control information are contained in the acquired teleoperation message, the target control object only focuses on the control information generated by the target controller. As for the control information generated by other controllers except the target controller, they are all invalid control information for the target control object.

[0082] Therefore, the solution provided by the embodiment of the present disclosure can distinguish the source of control information by carrying a controller identifier in the remote operation message, so that the control object is only controlled by the specific controller corresponding to it, avoiding confusion of control information in parallel remote operation scenarios, thereby enabling multiple controllers in the same remote operation system to operate multiple control objects in parallel.

[0083] Next, we will combine Figure 4 , which details the process of teleoperating the target control object based on the control information carried in the teleoperation information.

[0084] In some embodiments, in order to implement remote operation of a target control object, the control information at least includes: angle information of each reference component in the target control object.

[0085] It is understood that the principle of teleoperation is to make the control object as a whole present a desired posture by making the reference component in the control object be at a specific angle. The reference component can be understood as a component whose orientation angle is directly specified by the controller, or as a reference for determining the rotation angle of each component in the control object.

[0086] For example, when the controller is a motion capture suit and the controlled object is a robot, the reference component may be a joint of the robot corresponding to the sampling point of the motion capture suit.

[0087] Specifically, taking the arm as an example, assuming that the arm position of the motion capture suit has three sampling points: shoulder, elbow and wrist, then the shoulder joint, elbow joint and wrist joint of the robot are the reference parts of the robot arm position. By teleoperating the robot, the angles of these joints in the robot will be consistent with the joint angles captured by the motion capture suit, so that the robot can make the same movements as the operator wearing the motion capture suit.

[0088] Figure 4 A schematic diagram of a remote operation method according to an embodiment of the present disclosure is shown. Figure 4 The remote operation method provided in the embodiment of the present disclosure includes the following steps.

[0089] S401 , for every two adjacent reference components in the target control object, respectively execute the following S402 to S404 .

[0090] The adjacent reference components may be understood as reference components that are structurally adjacent to each other among the reference components of the target control object.

[0091] For example, still taking the position of the robot's arm as an example, assuming that there are three reference components on the robot's arm, namely the shoulder joint, elbow joint and wrist joint, then the shoulder joint and the elbow joint are two adjacent reference joints, and the elbow joint and the wrist joint are adjacent reference joints.

[0092] S402: Determine a rotational relationship between two adjacent reference components based on the angle information of the two adjacent reference components.

[0093] In some embodiments, the angle information may show the orientation angle of the reference component in three-dimensional space. The rotation relationship is used to describe how one of the reference components is rotated so that the reference component can have the same orientation angle as another reference component.

[0094] That is, when determining the rotational relationship, one of the two adjacent reference components can be used as a reference component, and then the rotational relationship of the other reference component relative to the reference component is determined based on the angle information of the two adjacent reference components.

[0095] Exemplarily, both the angle information and the rotation relationship can be represented by a matrix, and thus the following formula (1) can be obtained.

[0096] R relative = R a -1 ·R b (1)

[0097] Among them, R relative is the rotation relationship matrix. R a is the angle information matrix of the reference component in two adjacent reference components, Rb is the angle information matrix of the non-reference component in two adjacent reference components, R a and R b It can be obtained by parsing the teleoperation message.

[0098] S403, determining a movable part in the target controlled object corresponding to two adjacent reference parts; wherein the movable part includes at least two adjacent reference parts.

[0099] It should be noted that the movable part in the embodiment of the present disclosure can be understood as a minimum structure including two adjacent reference components. In this case, the two adjacent reference components are often located at the end positions of the movable part.

[0100] Since in the overall structure of the target control object, two adjacent reference components may not be directly connected but indirectly connected through other transition components, therefore, when determining the movable part, the transition component between the two adjacent reference components can be first determined in the target control object, and the two adjacent reference components and the transition component are collectively regarded as the movable part.

[0101] It is understandable that the "components" (e.g., reference components, transition components, etc.) referred to in the embodiments of the present disclosure can be understood as controllable rotating mechanisms in the control object. For example, when the target control object is a robot, each rotatable joint of the robot can be referred to as a "component". When the target control object is an unmanned vehicle, the rotating mechanism that enables the steering wheel to rotate can also be referred to as a "component" in the present disclosure, and the embodiments of the present disclosure will not elaborate on this.

[0102] S404, based on the rotation relationship, determining the rotation angle of each component in the movable part, so as to control each component in the movable part.

[0103] In some embodiments, the reference component can be used as a starting point, and the rotation amounts of each component in its rotatable direction can be sequentially superimposed according to the connection order of each component in the movable part to obtain the posture information of the movable part. Based on the mapping relationship between the rotation relationship and the posture information, the rotation angle of each component in the movable part is determined.

[0104] The posture information may show the relationship between the rotation angle of each component in the movable part and the rotation relationship determined in S402. By mapping the rotation relationship determined in S402 to the representation of the posture information, the rotation angle of each component can be solved.

[0105] Exemplarily, the rotation amount of each component in its rotatable direction can be represented by its rotation matrix. By multiplying the rotation matrices of each component in sequence, the rotation amounts of each component in its rotatable direction can be superimposed in sequence, thereby obtaining the posture information of the movable part.

[0106] In some embodiments, after completing the above S402 to S404 for every two adjacent reference components in the target control object, each component in the target control object may be controlled to rotate at a corresponding rotation angle, thereby achieving remote operation of the target control object.

[0107] Therefore, the embodiment of the present disclosure can determine the rotation angle of each component based on the physical rotation relationship between the components, so that the obtained rotation angle has a higher accuracy.

[0108] Combination of the above Figure 4 , the remote operation method provided by the present disclosure is further explained. Figure 5 Taking the structure of the mechanical arm shown in the figure as an example, the following is described in detail. Figure 4 The specific application process of the method is shown.

[0109] For example, Figure 5 The robotic arm structure shown is used as the target control object, and is remotely operated through the motion capture suit. The motion capture suit can be provided with four sampling points, namely, the shoulder, upper arm, lower arm, and hand, which correspond to joints ①, ③, ⑤, and ⑦ in the robotic arm structure in sequence.

[0110] That is to say, in the robotic arm structure, joint ①, joint ③, joint ⑤ and joint ⑦ are all reference components. At the same time, based on the adjacent relationship, it can be seen that joint ① and joint ③ are two adjacent reference components, joint ③ and joint ⑤ are two adjacent reference components, and joint ⑤ and joint ⑦ are two adjacent reference components.

[0111] At the same time, the robotic arm is also provided with joints ②, ④ and ⑥. These joints do not have corresponding motion capture suit sampling points, and can be used as transition joints for improving the rotational freedom of the robotic arm.

[0112] In order to realize remote operation of the robotic arm structure, the control information carried in the remote operation message includes: shoulder angle information, upper arm angle information, lower arm angle information and hand angle information, which can be collected by the sampling points of the motion capture suit.

[0113] After obtaining the angle information of each reference component, firstly for the two adjacent reference components of joint ① and joint ③, the rotation relationship between the two adjacent reference components can be expressed as:

[0114] R relative1 = R shoulder -1 ·R upper.arm (2)

[0115] Among them, R relative1is the rotation relationship matrix of two adjacent reference parts, R shoulder is the shoulder angle information matrix, R upper.arm is the upper arm angle information matrix.

[0116] By observation Figure 5 It can be found that the movable parts corresponding to joints ① and ③ include not only joints ① and ③, but also joints ② located between joints ① and ③.

[0117] Since equation (2) is obtained by taking joint ① corresponding to the shoulder as the reference part, the posture information of the movable part should be determined with joint ① as the starting point. Figure 5 It can be seen that, with joint ① corresponding to the shoulder as the starting point, the movable part will successively rotate around the Y axis, around the X axis, and around the Z axis to reach joint ③. Therefore, the posture information of the movable part can be expressed as the following equations (3) to (5).

[0118] R relative1 =R(Y)R(X)R(Z) (3)

[0119]

[0120] In the above formulas (3) to (5), R relative1 is a rotation relationship, R(Y), R(X), R(Z) are the rotation matrices of joint ①, joint ② and joint ③ respectively, and θ1, θ2, θ3 are the rotation angles of joint ①, joint ② and joint ③ respectively.

[0121] The rotation relationship matrix R obtained by formula (2) relative1 Substituting into equation (5) and based on the relationship of trigonometric functions, we can obtain equations (6) to (8) below, thereby solving the rotation angles of joints ①, ② and ③.

[0122] θ1 = arctan(R relative1 [0,2] / R relative1 [2,2]) (6)

[0123] θ2= -arcsin(R relative1 [1,2]) (7)

[0124] θ3= -arctan(R relative1 [1,0] / R relative1 [1,1]) (8)

[0125] Next, for the two adjacent reference components, joint ③ and joint ⑤, the rotation relationship between the two adjacent reference components can be expressed as:

[0126] R relative2 = Rupper.arm -1 ·R forearm (9)

[0127] Among them, R relative2 is the rotation relationship matrix of two adjacent reference parts, R upper.arm is the upper arm angle information matrix, R forearm is the lower arm angle information matrix.

[0128] By observation Figure 5 It can be found that the movable parts corresponding to joints ③ and ⑤ include not only joints ③ and ⑤, but also joint ④ located between joints ③ and ⑤.

[0129] Since equation (9) is obtained by taking joint ③ corresponding to the upper arm as the reference part, the posture information of the movable part should be determined with joint ③ as the starting point. Figure 5 It can be seen that, with joint ③ corresponding to the upper arm as the starting point, the movable part will successively rotate around the Z axis, around the Y axis, and around the Z axis to reach joint ③. Therefore, the posture information of the movable part can be expressed as the following equations (10) to (12).

[0130] R relative2 =R(Z)R(Y)R(Z) (10)

[0131]

[0132]

[0133] In the above formulas (10) to (12), R relative2 is a rotation relationship, R(Z), R(X), R(Z) are the rotation matrices of joint ③, joint ④ and joint ⑤ respectively, θ3, θ4, θ5 are the rotation angles of joint ③, joint ④ and joint ⑤ respectively.

[0134] The rotation relationship matrix R obtained by formula (9) is relative2 Substituting into equation (12) and based on the relationship of trigonometric functions, we can obtain equations (13) to (15), thereby solving the rotation angles of joints ③, ④ and ⑤.

[0135] θ3 = arctan(R relative2 [1,2] / R relative2 [0,2]) (13)

[0136] θ4= arcsin(R relative2 [2,2]) (14)

[0137] θ5= arctan(-R relative2 [2,1] / R relative2[2,0]) (15)

[0138] Finally, for the two adjacent reference components, joint ⑤ and joint ⑦, the rotation relationship between the two adjacent reference components can be expressed as:

[0139] R relative3 = R forearm -1 ·R hand (16)

[0140] Among them, R relative3 is the rotation relationship matrix of two adjacent reference parts, R forearm is the lower arm angle information matrix, R hand is the hand angle information matrix.

[0141] By observation Figure 5 It can be found that the movable parts corresponding to joints ⑤ and ⑦ include not only joints ⑤ and ⑦, but also joint ⑥ located between joints ⑤ and ⑦.

[0142] Since equation (16) is obtained by taking joint ⑤ corresponding to the lower arm as the reference part, the posture information of the movable part should be determined with joint ⑤ as the starting point. Figure 5 It can be seen that, with joint ⑤ corresponding to the lower arm as the starting point, the movable part will rotate around the Z axis, around the X axis and around the Y axis in sequence to reach joint ⑦. Therefore, the posture information of the movable part can be expressed as the following equations (17) to (19).

[0143] R relative3 =R(Z)R(X)R(Y) (17)

[0144]

[0145] In the above formulas (17) to (19), R relative3 is a rotation relationship, R(Z), R(X), R(Y) are the rotation matrices of joint ⑤, joint ⑥ and joint ⑦ respectively, and θ5, θ6, θ7 are the rotation angles of joint ⑤, joint ⑥ and joint ⑦ respectively.

[0146] The rotation relationship matrix R obtained by formula (16) relative2 Substituting into equation (19) and based on the relationship of trigonometric functions, we can obtain equations (20) to (22) below, thereby solving the rotation angles of joints ⑤, ⑥ and ⑦.

[0147] θ5 = arctan(-R relative3 [0,1] / R relative3 [1,1]) (20)

[0148] θ6 = arcsin(Rrelative3 [2,1]) (21)

[0149] θ7= arctan(-R relative3 [2,0] / R relative3 [2,2]) (22)

[0150] It is worth noting that in the disclosed embodiment, since joint ③ and joint ① are adjacent reference components to each other, and joint ③ and joint ⑤ are also adjacent reference components to each other, that is, joint ③ participates in two calculations. Therefore, the final rotation angle of joint ③ is the superposition of the two calculation results, that is, the superposition of the calculation results of formula (8) and the calculation results of formula (13).

[0151] Based on the same principle, the final rotation angle of joint ⑤ is the superposition of equation (15) and equation (20), which will not be elaborated in the present embodiment.

[0152] In some embodiments, after the rotation angle of each component is determined, outlier detection may be performed on the rotation angle of each component, and the abnormal rotation angle may be corrected.

[0153] Specifically, the following operations may be performed for each component: Obtain the rotation angle of the component at the previous moment. Determine whether the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets the abnormal condition. If so, re-determine the rotation angle of the component at the current moment based on the rotation angle of the component at the historical moment.

[0154] Exemplarily, when the rotation angle at the current moment suddenly changes compared with the rotation angle at the previous moment, it can be determined that the rotation angle at the current moment is an abnormal rotation angle.

[0155] In order to accurately measure this abnormality, the abnormal condition can be set as: the ratio between the current rotation angle and the previous rotation angle is greater than a preset threshold. That is, the ratio between the current rotation angle and the previous rotation angle is used to measure whether the rotation angle has changed suddenly.

[0156] Of course, the abnormal condition can also be set as: the difference between the rotation angle at the current moment and the rotation angle at the previous moment is greater than the preset threshold. That is, the difference between the two can also be used to measure whether the rotation angle has a sudden change, which is not limited in the embodiments of the present disclosure.

[0157] Exemplarily, when the rotation angle is determined to be an abnormal rotation angle, the rotation angle can be re-determined in the following manner: based on the rotation angles of the component at any two historical moments, a reference rotation angle of the component per unit time is determined. Based on the reference rotation angle, combined with the time interval between the current moment and the previous moment, the rotation angle of the component at the current moment is determined.

[0158] That is to say, the rotation angle at the current moment can be calculated in the time dimension through interpolation, so as to correct the abnormal rotation angle.

[0159] It is worth noting that in this example, the reference rotation angle per unit time is calculated by the rotation angles of any two historical moments, and is used as the basis for interpolation. In order to make the interpolation result more accurate, the reference rotation angle per unit time can be calculated based on the rotation angles of the two historical moments closest to the current moment, so that the rotation angle obtained by the difference is more consistent with the motion trend of the target control object.

[0160] Of course, the reference rotation angle per unit time can also be preset. For example, when the rotation of the component is approximately uniform, the rotation angle per unit time that matches its rotation speed can be directly set as the reference rotation angle per unit time.

[0161] For example, assuming that t1, t2, t3, and t4 are consecutive moments, the rotation angles corresponding to each moment are θ 1 ,θ 2 ,θ 3 and θ 4 Then we can calculate θ 4 With θ 3 The ratio e between them determines the rotation angle θ at time t4 4 Whether it is an abnormal rotation angle.

[0162] When e is greater than a preset threshold (for example, the preset threshold may be 2), the rotation angle θ at time t4 is determined. 4 is the abnormal rotation angle. At this time, the rotation angle θ at time t4 can be recalculated by the following formula (23): 4 .

[0163]

[0164] The above describes in detail the remote operation method and the corresponding abnormal angle judgment method provided by the embodiment of the present disclosure. For example, in the scheme provided by the present disclosure, the motion capture suit or the motion capture gloves can be used as a controller, or the motion capture suit and the motion capture gloves can be used as controllers at the same time, so as to realize remote operation of the robot.

[0165] Please refer to Figure 6 , Figure 6 Schematic diagram of sampling points of motion capture suit and motion capture gloves in the example of the present disclosure is shown. Figure 6 As shown, when the controller includes motion capture gloves, the sampling points of the controller can cover various hand joints of the operator, thereby being able to operate the robot's hand to achieve fine movements (for example, grasping smaller objects).

[0166] Based on the same inventive concept, the present disclosure also provides a remote control device in the following embodiments. Figure 2 The method embodiment shown is similar to that shown in FIG. 1 , so the implementation of the remote operation device embodiment can refer to the above Figure 2 The implementation of the illustrated method embodiment will not be described in detail in detail.

[0167] Figure 7 FIG. 2 is a schematic diagram showing the structure of a remote control device according to an embodiment of the present disclosure. Figure 7 As shown, the remote operation device 700 includes: an acquisition module 701 and a control module 702 .

[0168] The acquisition module 701 is used to acquire a teleoperation message, which carries control information and a controller identifier; wherein the controller identifier is used to indicate the controller that generates the control information.

[0169] The control module 702 is used to control the target control object based on the control information when the controller identifier is the target controller identifier, so as to realize the remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

[0170] In some embodiments, the control information carried by the teleoperation message is multiple groups, and each group of control information corresponds to a different controller identifier. The control module 702 is specifically configured to, for each group of control information in the multiple groups of control information, control the target control object based on the group of control information when the controller identifier corresponding to the group of control information is the target controller identifier.

[0171] In some embodiments, the control information includes: angle information of each reference component in the target control object. The control module 702 is specifically used to perform the following operations for each two adjacent reference components in the target control object: determine the rotation relationship between the two adjacent reference components based on the angle information of the two adjacent reference components; determine the movable part corresponding to the two adjacent reference components in the target control object; wherein the movable part includes at least the two adjacent reference components; determine the rotation angle of each component in the movable part based on the rotation relationship, so as to control each component in the movable part.

[0172] In some embodiments, the control module 702 is specifically configured to use one of the two adjacent reference components as a reference component, and determine a rotational relationship of the other reference component relative to the reference component based on angle information of the two adjacent reference components.

[0173] In some embodiments, the control module 702 is specifically used to, with the reference component as the starting point, sequentially superimpose the rotation amounts of each component in its rotatable direction according to the connection sequence of each component in the movable part, to obtain the posture information of the movable part; the posture information is used to show the relationship between the rotation angle and the rotation relationship of each component in the movable part; based on the mapping relationship between the rotation relationship and the posture information, determine the rotation angle of each component in the movable part.

[0174] In some embodiments, the control module 702 is specifically configured to determine a transition component between two adjacent reference components in the target control object; and use the two adjacent reference components and the transition component as movable parts.

[0175] In some embodiments, the control module 702 is also used to perform the following operations for each component in the movable part: obtaining the rotation angle of the component at the previous moment; determining whether the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets the abnormal condition, and re-determining the rotation angle of the component at the current moment based on the rotation angle of the component at the historical moment.

[0176] In some embodiments, the control module 702 is also used to determine that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets an abnormal condition when the ratio between the rotation angle of the component at the current moment and the rotation angle at the previous moment is greater than a preset threshold.

[0177] In some embodiments, the control module 702 is also used to determine the reference rotation angle of the component per unit time based on the rotation angle of the component at any two historical moments; based on the reference rotation angle, combined with the time interval between the current moment and the previous moment, determine the rotation angle of the component at the current moment.

[0178] In some embodiments, the controller is a motion capture suit and / or motion capture gloves.

[0179] Refer to the following Figure 8 800 capable of implementing the embodiments of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of application of the embodiments of the present disclosure.

[0180] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: at least one processor 810, at least one memory 820, and a bus 830 connecting different system components (including the memory 820 and the processor 810).

[0181] The memory stores program codes, which can be executed by the processor 810, so that the processor 810 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of the present disclosure.

[0182] In some embodiments, the processor 810 may further perform the following steps of the above method embodiment:

[0183] Acquire a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used to indicate the controller that generates the control information;

[0184] In the case where the controller identifier is a target controller identifier, the target control object is controlled based on the control information to implement remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

[0185] The memory 820 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8201 and / or a cache memory 8202 , and may further include a read-only memory (ROM) 8203 .

[0186] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0187] Bus 830 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0188] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 860. Figure 8 As shown, the network adapter 860 communicates with other modules of the electronic device 800 via the bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0189] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0190] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of the present disclosure.

[0191] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A remote operation method, characterized in that: include: Acquire a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used to indicate the controller that generates the control information; In the case where the controller identifier is a target controller identifier, the target control object is controlled based on the control information to implement remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

2. The method according to claim 1, characterized in that The remote operation message carries multiple groups of control information, and each group of control information corresponds to a different controller identifier; When the controller identifier is a target controller identifier, controlling the target control object based on the control information includes: For each group of control information in the multiple groups of control information, when the controller identifier corresponding to the group of control information is the target controller identifier, the target control object is controlled based on the group of control information.

3. The method according to claim 1, characterized in that The control information includes: angle information of each reference component in the target control object; The controlling the target control object based on the control information includes: For every two adjacent reference components in the target control object, the following operations are performed respectively: Determining a rotational relationship between the two adjacent reference components based on the angle information of the two adjacent reference components; Determining a movable part in the target control object corresponding to the two adjacent reference parts; wherein the movable part at least includes the two adjacent reference parts; Based on the rotational relationship, the rotation angle of each component in the movable part is determined so as to control each component in the movable part.

4. The method according to claim 3, characterized in that The step of determining the rotational relationship between the two adjacent reference components based on the angle information of the two adjacent reference components comprises: One of the two adjacent reference components is used as a reference component, and the rotation relationship of the other reference component relative to the reference component is determined based on the angle information of the two adjacent reference components.

5. The method according to claim 4, characterized in that The step of determining the rotation angle of each component in the movable part based on the rotation relationship comprises: Taking the reference component as the starting point, according to the connection order of the components in the movable part, the rotation amounts of the components in their rotatable directions are sequentially superimposed to obtain the posture information of the movable part; the posture information is used to show the relationship between the rotation angles of the components in the movable part and the rotation relationship; Based on the mapping relationship between the rotation relationship and the posture information, the rotation angle of each component in the movable part is determined.

6. The method according to claim 3, characterized in that The step of determining the movable part of the target control object corresponding to the two adjacent reference parts includes: determining a transition component between two adjacent reference components in the target control object; Two adjacent reference components and the transition component are used as the movable part.

7. The method according to claim 3, characterized in that After determining the rotation angle of each component in the movable part based on the rotation relationship, the method further includes: For each component in the movable part, perform the following operations respectively: Get the rotation angle of the component at the last moment; It is determined that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets an abnormal condition, and based on the rotation angle of the component at the historical moment, the rotation angle of the component at the current moment is re-determined.

8. The method according to claim 7, characterized in that The determining that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment satisfies the abnormal condition includes: When the ratio between the rotation angle of the component at the current moment and the rotation angle at the previous moment is greater than a preset threshold, it is determined that the relationship between the rotation angle of the component at the current moment and the rotation angle at the previous moment meets the abnormal condition.

9. The method according to claim 7, characterized in that: The re-determining the rotation angle of the component at the current moment based on the rotation angle of the component at the historical moment includes: Based on the rotation angles of the component at any two historical moments, determining a reference rotation angle of the component per unit time; Based on the reference rotation angle and in combination with the time interval between the current moment and the previous moment, the rotation angle of the component at the current moment is determined.

10. The method according to claim 1, characterized in that The controller is a motion capture suit and / or motion capture gloves.

11. A remote control device, characterized in that: include: An acquisition module, used for acquiring a teleoperation message, wherein the teleoperation message carries control information and a controller identifier; wherein the controller identifier is used for indicating the controller that generates the control information; A control module is used to control a target control object based on the control information when the controller identifier is a target controller identifier, so as to realize remote operation of the target control object; wherein a corresponding relationship is configured between the target controller identifier and the target control object.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images