Armband-based robot remote control method, device, electronic device, storage medium and system

Abstract

The present disclosure provides a robot remote control method, device, electronic device, storage medium and system based on an armband. The robot includes a mechanical arm and a mechanical hand connected to each other, and an electromyographic sensor and an inertial measurement unit are installed on the armband. The method of the embodiment of the present disclosure includes: obtaining a first electromyographic signal from the electromyographic sensor on the armband and a first motion data from the inertial measurement unit on the armband; controlling a first finger of the mechanical hand to perform a first action based on the first electromyographic signal; and controlling the mechanical arm to perform a second action based on the first motion data. The present disclosure is simple to operate and easy to implement, and the mechanical hand control accuracy and mechanical hand operation flexibility are both high, and the hardware cost is low.

Description

Technical Field

[0001] The present disclosure relates to the field of robot technology, and in particular to an armband-based robot remote control method, device, electronic device, storage medium and system. Background Art

[0002] The combination of a robotic arm and a manipulator is usually called a robot. Robots are widely used in various fields such as warehousing and logistics, surgery, medical rehabilitation, processing and manufacturing, energy and mineral collection, etc. Therefore, the control method of the robot is very important.

[0003] At present, the remote control of robots is mainly completed through large equipment such as computers and servers. This remote control method is not only complex and difficult to implement, but also has high costs, and has poor control accuracy and low flexibility. Therefore, a robot remote control solution that is simple to operate, easy to implement and flexible to use is needed. Summary of the invention

[0004] In view of this, the present disclosure provides an armband-based robot remote control method, device, electronic device, storage medium and system.

[0005] According to a first aspect of the present disclosure, a robot remote control method based on an armband is provided, wherein the robot comprises a robotic arm and a robotic hand connected to each other, and an electromyographic sensor and an inertial measurement unit are installed on the armband; the method comprises: when a user wears the armband, obtaining a first electromyographic signal from the electromyographic sensor on the armband and first motion data from the inertial measurement unit on the armband; controlling a first finger of the robotic hand to perform a first action based on the first electromyographic signal; and controlling the robotic arm to perform a second action based on the first motion data.

[0006] In some possible implementations of the first aspect of the present disclosure, controlling the manipulator to perform a first action based on the first electromyographic signal includes: extracting characteristic data of the first electromyographic signal; using an electromyographic model to process the characteristic data of the first electromyographic signal to identify the user's desired finger action and generate corresponding finger action instructions, wherein the finger action instructions include information of the first finger and its action instructions, and the first finger includes one or more of the thumb, index finger, middle finger, ring finger and little finger of the manipulator; controlling the first finger of the manipulator to perform the first action according to the finger action instructions.

[0007] In some possible implementations of the first aspect of the present disclosure, controlling the robotic arm to perform a second action based on the first motion data includes: analyzing the first motion data to obtain first posture data indicating a first posture of the user's arm, the first posture data including a roll angle, a pitch angle, and a heading angle; and controlling the robotic arm to perform a second action based on the first posture data, the second action including one or more of the following: wrist rotation, arm flexion and extension, and arm adduction and abduction.

[0008] In some possible implementations of the first aspect of the present disclosure, controlling the mechanical arm to perform the second action based on the first posture data includes:

[0009] Verifying whether the roll angle in the first posture data is within a predetermined roll angle valid range, whether the pitch angle is within a predetermined pitch angle valid range, and whether the heading angle is within a predetermined heading angle valid range;

[0010] If the roll angle in the first posture data is within the roll angle valid range, the pitch angle is within the pitch angle valid range, and the heading angle is within the heading angle valid range, then controlling the robotic arm to perform the second action based on the first posture data;

[0011] If the roll angle in the first posture data is not within the effective range of the roll angle, the pitch angle is not within the effective range of the pitch angle, or the heading angle is not within the effective range of the heading angle, the robotic arm is controlled not to perform any action.

[0012] In some possible implementations of the first aspect of the present disclosure, the effective range of the roll angle is determined according to the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs a wrist rotation action; and / or, the effective range of the pitch angle is determined according to the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user wears the armband and performs an arm flexion and extension action; and / or, the effective range of the heading angle is determined according to the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs an arm abduction and adduction action.

[0013] In some possible implementations of the first aspect of the present disclosure, controlling the robotic arm to perform a second action based on the first posture data includes one or more of the following:

[0014] Determine whether the roll angle in the first posture data is within a first predetermined range or a second predetermined range, and if the roll angle is within the first predetermined range, control the end of the robot arm to move in the positive direction of the X-axis of the robot arm so that the robot arm performs a left-hand wrist rotation action; if the roll angle is within the second predetermined range, control the end of the robot arm to move in the negative direction of the X-axis of the robot arm so that the robot arm performs a right-hand wrist rotation action;

[0015] Determine whether the pitch angle in the first posture data is within a third predetermined range or a fourth predetermined range, and if the pitch angle is within the third predetermined range, control the end of the robotic arm to move along the positive direction of the Y axis of the robotic arm so that the robotic arm performs an arm bending action, and if the pitch angle is within the fourth predetermined range, control the end of the robotic arm to move along the negative direction of the Y axis of the robotic arm so that the robotic arm performs an arm extending action;

[0016] Determine whether the heading angle in the first posture data is within the fifth predetermined range or the sixth predetermined range. If the heading angle is within the fifth predetermined range, control the end of the robotic arm to move in the positive direction of the Z axis of the robotic arm so that the robotic arm performs an arm retraction action. If the heading angle is within the sixth predetermined range, control the end of the robotic arm to move in the negative direction of the Z axis of the robotic arm so that the robotic arm performs an arm abduction action.

[0017] According to a second aspect of the present disclosure, a robot remote control device based on an armband is provided, wherein the robot comprises a robotic arm and a robotic hand connected to each other, and an electromyographic sensor and an inertial measurement unit are installed on the armband; the robot remote control device based on the armband comprises: an acquisition unit, for acquiring a first electromyographic signal from the electromyographic sensor on the armband and first motion data from the inertial measurement unit on the armband when the user wears the armband; a robotic hand control unit, for controlling a first finger of the robotic hand to perform a first action based on the first electromyographic signal; and a robotic arm control unit, for controlling the robotic arm to perform a second action based on the first motion data.

[0018] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: one or more processors and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned armband-based robot remote control method.

[0019] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the above-mentioned armband-based robot remote control method.

[0020] According to a fifth aspect of the present disclosure, a robot remote control system is provided, wherein the robot comprises a robotic arm and a robotic hand connected to each other, and the robot remote control system comprises: an armband and an electronic device, wherein the electronic device is built into the robot or electrically connected to the robot; the armband comprises an armband body and an electromyographic sensor and an inertial measurement unit installed on the armband body; the electronic device comprises: a controller and a communication module, wherein the communication module is electrically connected to the controller, the communication module can communicate with the electromyographic sensor and the inertial measurement unit respectively, and the controller is used to execute the above-mentioned robot remote control method based on the armband.

[0021] It can be seen from the above technical solutions that through the disclosed embodiments, users only need to wear an armband to control specific fingers of the manipulator to perform specified actions and control the manipulator arm to perform required actions at the same time. The operation is simple and easy to implement. The hardware and software development costs are low. The manipulator can be controlled to perform various actions such as release, grasping, and fisting. The manipulator arm can be operated to perform various actions such as wrist rotation, arm flexion and extension, and arm adduction and abduction. It is suitable for complex robot operations in various fields such as warehousing and logistics, surgical operations, medical rehabilitation, processing and manufacturing, energy and mineral collection. The disclosed embodiments are simple to operate, easy to implement, the manipulator has high control accuracy, has high flexibility of use, and can be implemented using low-cost hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of the structure of a robot remote control system provided by an embodiment of the present disclosure;

[0024] Figure 2 A physical schematic diagram of a robot involved in an embodiment of the present disclosure;

[0025] Figure 3 A flowchart of a robot remote control method based on an armband provided in an embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram of an implementation flow of controlling a first finger of a manipulator to perform a first action based on a first electromyographic signal according to an embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of the structure of an armband-based robot remote control device provided in an embodiment of the present disclosure;

[0028] Figure 6 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0031] As used herein, the words "if," "if," and the like may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0032] To facilitate understanding, the robot remote control system provided in the embodiment of the present disclosure is first exemplarily described below.

[0033] Figure 1 FIG. 1 shows a schematic diagram of the structure of a robot remote control system provided by an embodiment of the present disclosure. Figure 1 The robot remote control system provided by the embodiment of the present disclosure includes: an armband 110, a robot 120 and an electronic device 130. The electronic device 130 can be used to control the robot to perform actions through the robot remote control method based on the armband described below.

[0034] See also Figure 1 As shown, the robot 120 may include a manipulator 121 and a manipulator arm 122, and the manipulator 121 is connected to the manipulator arm 122. Exemplarily, the upper end of the manipulator 121 may be movably connected to the end of the manipulator arm 122, so that the manipulator 121 and the manipulator arm 122 can better simulate the linkage movement of human arms and hands under the control of the electronic device 130.

[0035] See also Figure 1The armband 110 includes an armband body 111 and an electromyographic sensor 112 and an inertial measurement unit 113 (IMU, Inertial Measurement Unit) installed on the armband body 111. The electromyographic sensor 112 and the inertial measurement unit 113 can communicate with a communication module 132 in the electronic device 130 respectively.

[0036] See also Figure 1 The electronic device 130 may include a controller 131 and a communication module 132. The communication module 132 is electrically connected to the controller 131. The communication module 132 may communicate with the electromyographic sensor 112 to obtain the electromyographic signal collected by the electromyographic sensor 112. The communication module 132 may communicate with the inertial measurement unit 113 to obtain the posture data collected by the inertial measurement unit 113. The controller 131 may be used to execute the armband-based robot remote control method described below. In specific applications, the electronic device 130 may be implemented as a microcontroller 131 or other similar control device that can be built into the robot 120. Alternatively, the electronic device 130 may be implemented as an external device such as a computer or a server.

[0037] In some examples, the communication module 132 can be implemented as, but not limited to, a Bluetooth module, a short-range communication module, an Ethernet communication interface, or other similar modules to communicate wirelessly. In other examples, the communication module 132 can also be implemented as a communication data serial port or other electrical connection interface to communicate wiredly.

[0038] The myoelectric sensor 112 can be fixed on the inner surface of the armband body 111 so that after the user wears the armband 110, the myoelectric sensor 112 can contact the skin surface of the user's arm and generate an electromyographic signal by sensing the contraction of the user's arm muscles. In some examples, the myoelectric sensor 112 can be but is not limited to a surface electromyographic sensor, and the myoelectric signal can be but is not limited to a surface electromyography (sEMG).

[0039] The inertial measurement unit 113 can be fixed to any area of ​​the armband body 111 so that the inertial measurement unit 113 can measure the motion data of the user's arm in real time. In specific applications, the inertial measurement unit 113 may include but is not limited to a gyroscope, an accelerometer, a magnetometer, and the like.

[0040] Figure 2 A real picture of the robot remote control system provided by the embodiment of the present disclosure is shown. Figure 2 In the example, the electronic device 130 is implemented as a computer, and the computer is electrically connected to the robot 120.

[0041] The above-mentioned system architecture of the embodiment of the present disclosure is only taken as an example. Those skilled in the art should understand that in specific applications, the robot remote control system including the armband 110 may also adopt any other applicable architecture, and the embodiment of the present disclosure does not limit this.

[0042] The armband-based robot remote control method provided in the embodiments of the present disclosure is described in detail below.

[0043] Figure 3 The flowchart of the armband-based robot remote control method provided by the embodiment of the present disclosure is shown. The armband-based robot remote control method provided by the embodiment of the present disclosure can be executed by an electronic device built into or connected to the robot. Figure 3 , the method of the embodiment of the present disclosure may include:

[0044] Step 301, when the user wears the armband, obtaining a first electromyographic signal from an electromyographic sensor on the armband and first motion data from an inertial measurement unit on the armband;

[0045] The first electromyographic signal is used to indicate the muscle contraction status of the arm of the user wearing the wall ring, and the first motion data is used to indicate the position of the arm of the user wearing the wall ring. It should be noted that the first electromyographic signal generally refers to an electromyographic signal collected by the electromyographic sensor at a certain moment or a sequence of electromyographic signals collected in a certain period of time, and the first motion data generally refers to a set of posture data collected by the inertial measurement unit or a sequence of posture data in a certain period of time. The first electromyographic signal and the first motion data are collected synchronously.

[0046] Step 302, controlling a first finger of the manipulator to perform a first action based on the first electromyographic signal;

[0047] Step 303: Control the robotic arm to perform a second action based on the first motion data.

[0048] In a specific application, step 302 and step 303 may be performed simultaneously or sequentially. The acquisition of the first electromyographic signal and the first motion data in step 301 may be performed synchronously or sequentially.

[0049] Through the embodiments of the present disclosure, users only need to wear an armband to control specific fingers of the robot to perform specified actions and control the robot arm to perform required actions at the same time. The operation is simple and easy to implement, and the hardware and software development costs are low. The robot hand can be controlled to perform various actions such as releasing, grasping, and making a fist, and the robot arm can be operated to perform various actions such as wrist rotation, arm flexion and extension, and arm adduction and abduction. It is suitable for complex operations of robots in various fields such as warehousing and logistics, surgical operations, medical rehabilitation, processing and manufacturing, energy and mineral collection, etc.

[0050] The specific implementation of the armband-based robot remote control method provided in the embodiment of the present disclosure is described in detail below.

[0051] In step 301, when the user wears the armband, the user's brain controls the arm wearing the armband to perform muscle contraction. The electromyographic sensor generates a first electromyographic signal due to sensing the contraction of the user's arm muscles and sends it to the electronic device through its own communication module.

[0052] In step 301, when the user wears the armband, the inertial measurement unit on the armband measures the motion data of the user's arm wearing the armband in real time and sends it to the electronic device through its own communication module. In specific applications, the first motion data may include but is not limited to acceleration, angular velocity and other data. The first motion data can be used to determine the posture of the user's arm wearing the armband, and the posture can be represented by the roll angle, pitch angle and heading angle.

[0053] Step 302 can be implemented by using a pre-trained myoelectric model. Figure 4 FIG. 3 is a flow chart showing a process of controlling the first finger of the manipulator to perform the first action based on the first electromyographic signal in step 302. Figure 4 , step 302 may include the following steps 401 to 403:

[0054] Step 401, extracting characteristic data of a first electromyographic signal;

[0055] Specifically, feature data of the first electromyographic signal can be extracted by, for example, signal processing, and the feature data can be used to indicate the muscle contraction activity of the arm of the user wearing the armband. In specific applications, the feature data of the first electromyographic signal can be features that are helpful for subsequent analysis, such as time domain features such as root mean square value, average absolute value, and integral absolute value, frequency domain features such as power spectral density, average frequency, and median frequency, or time-frequency domain features such as wavelet transform coefficients.

[0056] Considering that the electromyographic signal usually contains noise and artifacts such as electrocardiographic interference and motion artifacts, a preprocessing step of the electromyographic signal can also be performed before step 401. Exemplarily, the preprocessing step may include: filtering, amplification and denoising. For example, a bandpass filter can be used to remove low-frequency noise and high-frequency noise in the electromyographic signal, and retain the frequency range related to muscle activity, which can be set to 20 Hz to 500 Hz.

[0057] Step 402, using an electromyographic model to process the characteristic data of the first electromyographic signal to identify the finger movement desired by the user and generate a corresponding finger movement instruction;

[0058] In some examples, the myoelectric model can be, but is not limited to, a machine learning model such as a support vector machine (SVM), a random forest, a deep neural network, a Transformer model, etc. Specifically, the feature data of the first myoelectric signal is input into the myoelectric model for processing to identify the finger movement that the user expects to make and generate a corresponding finger movement instruction.

[0059] The finger motion instruction may include information about the first finger that the user expects to perform an action and information about the action that the user expects the first finger to perform. The first finger includes one or more of the thumb, index finger, middle finger, ring finger and little finger of the manipulator.

[0060] For example, if the action the user expects to perform is grasping, that is, bending five fingers at the same time, the first finger includes the five fingers of the robot, namely the thumb, index finger, middle finger, ring finger and little finger, and the corresponding finger action instruction includes action instructions corresponding to these five fingers. The action instruction of each finger includes the identification of the finger and the bending angle of the finger. The bending angle of the finger can be the bending angle of the finger relative to the palm.

[0061] For another example, if the action the user wants to perform is pinching, that is, bending the thumb and index finger at the same time so that they can "pinch an object", at this time, the first finger includes the thumb and index finger of the robot, and the corresponding finger action instruction includes action instructions corresponding to these two fingers. The action instruction of each finger includes the identifier of the finger and the bending angle of the finger. The bending angle of the finger can be the bending angle of the finger relative to the palm.

[0062] In specific applications, the myoelectric model can be pre-trained. For example, the myoelectric model can be trained using the myoelectric signal feature data that has been annotated with finger movements as training samples, so that the myoelectric model can learn the mapping relationship between the myoelectric signal feature data and the finger movements, so that the myoelectric model can be used to identify the finger movements corresponding to the first myoelectric signal feature data.

[0063] In some examples, considering the differences in arm muscle activity among different people, a dedicated myoelectric model can be trained for a specific user, thereby improving the recognition accuracy of the myoelectric model for the specific user, and further improving the control accuracy of the finger movements of the robot. For example, the myoelectric model can be trained using training samples of multiple groups of people (i.e., myoelectric signal feature data with finger movements annotated), and then the myoelectric model can be further trained using training samples of a specific user using a small sample learning method to make it more suitable for the specific user.

[0064] In some examples, considering that finger movements in different application scenarios vary greatly, training samples of commonly used finger movements in specific application scenarios (for example, medical rehabilitation) can be collected and used to train an electromyographic model so that the electromyographic model can better identify specific finger movements in the specific scenario, thereby improving the control flexibility of the robot's finger movements.

[0065] In some examples, when the user wears the armband and makes finger movements, the electromyographic signal of the electromyographic sensor on the armband and the corresponding finger movement information (for example, the identification of the finger making the movement and the bending angle of the finger, etc.) can be collected simultaneously, the characteristic data of the electromyographic signal can be extracted, and the electromyographic model can be trained using the characteristic data of the electromyographic signal and the corresponding finger movement information as sample data, so that the electromyographic model has the ability to recognize the finger movement corresponding to the first electromyographic signal characteristic data.

[0066] Step 403: Control the first finger of the robot to perform a first action according to the finger action instruction.

[0067] The first action generally refers to any type of single-finger action or multi-finger action. In some examples, the first action may be a flexion and extension action of a single finger or multiple fingers. When the first action is a flexion and extension action of multiple fingers, the flexion and extension degrees of different fingers may be different.

[0068] It can be seen that through the embodiments of the present disclosure, each finger of the robot can be controlled to make the action desired by the user, so that the user can control the robot to make complex hand actions such as grasping, pinching, and releasing by wearing the armband.

[0069] Step 303 may include: analyzing the first motion data to obtain first posture data indicating a first posture of the user's arm, and controlling the robotic arm to perform a second action based on the first posture data, wherein the first posture data includes a roll angle, a pitch angle, and a heading angle, and the second action may include but is not limited to one or more of the following: wrist rotation, arm flexion and extension, arm adduction and abduction, etc.

[0070] In specific applications, various applicable methods can be used to analyze the first motion data to determine the posture of the user's arm. For example, a motion analysis model capable of analyzing motion data to obtain the arm posture can be pre-trained. The motion analysis model can be, but is not limited to, a support vector machine, a random forest, a convolutional neural network, a recurrent neural network, a Transformer model, and other machine learning models. The first motion data can be processed using the motion analysis model to determine the first posture of the user's arm, and obtain first posture data indicating the first posture of the user's arm, that is, the value of the roll angle, the value of the pitch angle, and the value of the heading angle.

[0071] Furthermore, step 303 may also include: before controlling the robotic arm to perform the second action based on the first posture data, it may be determined whether the first posture data is valid, and if the first posture data is valid, the robotic arm may be controlled to perform the second action based on the first posture data.

[0072] In some examples, whether the first posture data is valid can be judged according to a preset posture valid range. Specifically, the posture valid range may include: a roll angle valid range, a pitch angle valid range, and a heading angle valid range. If the roll angle in the first posture data is within the roll angle valid range, the pitch angle in the first posture data is within the pitch angle valid range, and the heading angle in the first posture data is within the heading angle valid range, it indicates that the first posture data is valid. If the roll angle in the first posture data is not within the roll angle valid range, the pitch angle in the first posture data is not within the pitch angle valid range, or the heading angle in the first posture data is not within the heading angle valid range, the first posture data is invalid.

[0073] Specifically, step 303 may include: verifying whether the roll angle in the first posture data is within a predetermined effective range of the roll angle, whether the pitch angle is within a predetermined effective range of the pitch angle, and whether the heading angle is within a predetermined effective range of the heading angle; if the roll angle in the first posture data is within the effective range of the roll angle, the pitch angle is within the effective range of the pitch angle, and the heading angle is within the effective range of the heading angle, then controlling the robotic arm to perform the second action according to the first posture data; if the roll angle in the first posture data is not within the effective range of the roll angle, the pitch angle is not within the effective range of the pitch angle, or the heading angle is not within the effective range of the heading angle, then controlling the robotic arm not to perform any action.

[0074] In specific applications, the effective range of the gesture can be set by the user himself or according to the amplitude of the user's arm movement.

[0075] In some examples, the effective range of the roll angle can be determined based on the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs a wrist rotation action. Specifically, when the user wears the armband, the maximum roll angle a2 measured by the inertial measurement unit when the user's wrist is rotated to the left to the maximum extent is collected, and the minimum roll angle a1 measured by the inertial measurement unit when the user's wrist is rotated to the right to the maximum extent is collected, and then the effective range of the roll angle can be determined to be the interval [a1, a2].

[0076] In some examples, the effective range of the pitch angle can be determined based on the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user wears the armband and performs the arm flexion and extension action. Specifically, when the user wears the armband, the minimum pitch angle b1 measured by the inertial measurement unit when the user's arm is extended to the maximum extent is collected, and the maximum pitch angle b2 measured by the inertial measurement unit when the user's arm is bent to the maximum extent is collected, and the effective range of the pitch angle can be determined to be the interval [b1, b2].

[0077] In some examples, the effective range of the heading angle is determined based on the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs the arm abduction and adduction action. Specifically, when the user wears the armband, the minimum heading angle c1 measured by the inertial measurement unit when the user's arm is adducted to the maximum extent is collected, and the maximum heading angle c2 measured by the inertial measurement unit when the user's arm is abducted to the maximum extent is collected, and the effective range of the heading angle can be determined to be the interval [c1, c2].

[0078] Further, step 303 may include one or more of the following:

[0079] 1) Determine whether the roll angle in the first posture data is within the first predetermined range or the second predetermined range. If the roll angle is within the first predetermined range, control the end of the robot arm to move in the positive direction of the X-axis of the robot arm so that the robot arm performs a left-hand rotation of the wrist. If the roll angle is within the second predetermined range, control the end of the robot arm to move in the negative direction of the X-axis of the robot arm so that the robot arm performs a right-hand rotation of the wrist.

[0080] 2) determining whether the pitch angle in the first posture data is within a third predetermined range or a fourth predetermined range, and if the pitch angle is within the third predetermined range, controlling the end of the robotic arm to move along the positive direction of the Y axis of the robotic arm so that the robotic arm performs an arm bending action; and if the pitch angle is within the fourth predetermined range, controlling the end of the robotic arm to move along the negative direction of the Y axis of the robotic arm so that the robotic arm performs an arm extending action;

[0081] 3) Determine whether the heading angle in the first posture data is within the fifth predetermined range or the sixth predetermined range. If the heading angle is within the fifth predetermined range, control the end of the robotic arm to move in the positive direction along the Z axis of the robotic arm so that the robotic arm performs an arm retraction action. If the heading angle is within the sixth predetermined range, control the end of the robotic arm to move in the negative direction along the Z axis of the robotic arm so that the robotic arm performs an arm abduction action.

[0082] The XYZ axis of the robot arm and its positive and negative directions can be flexibly set according to the needs of specific application scenarios. For example, the X axis can be a coordinate axis parallel to the cross section of the robot arm, the Y axis can be parallel to the center axis of the robot arm, and the Z axis is a coordinate axis perpendicular to the X axis and the Y axis.

[0083] The first predetermined range and the second predetermined range can be determined according to the roll angle effective range and the first predetermined threshold. For example, if the roll angle effective range is the interval [a1, a2], the first predetermined threshold is set to a0, the first predetermined range can be [a1, a1+a0], and the second predetermined range can be [a2-a0, a2].

[0084] The third predetermined range and the fourth predetermined range can be determined according to the effective range of the pitch angle and the second predetermined threshold. For example, if the effective range of the pitch angle is the interval [b1, b2], the second predetermined threshold is set to b0, the third predetermined range can be [b1, b1+b0], and the fourth predetermined range can be [b2-b0, b2].

[0085] The fifth predetermined range and the sixth predetermined range can be determined according to the heading angle valid range and the third predetermined threshold. For example, if the heading angle valid range is the interval [c1, c2], the second predetermined threshold is set to c0, the fifth predetermined range can be [c1, c1+c0], and the sixth predetermined range can be [c2-c0, c2].

[0086] In a specific application, the values ​​of the first predetermined threshold a0, the second predetermined threshold b0 and the third predetermined threshold c0 may be the same or different. The specific values ​​of the first predetermined threshold a0, the second predetermined threshold b0 and the third predetermined threshold c0 may be flexibly set according to various factors such as application scenarios and different users.

[0087] In some embodiments, the first predetermined threshold, the second predetermined threshold and the third predetermined threshold The first predetermined threshold can be set as follows: when the user wears the armband, the user can place the arm in a default position (the default position is within the maximum range of the user's arm that can make a first movement), and the inertial measurement unit on the armband measures the posture data (i.e., roll angle, pitch angle and heading angle) at the default position and transmits it to the electronic device, and the electronic device uses the roll angle in the posture data as the first predetermined threshold, the pitch angle as the second predetermined threshold, and the heading angle as the third predetermined threshold.

[0088] As can be seen from the above, through the disclosed embodiment, the user can control the designated finger of the manipulator to perform designated actions by wearing the armband, and can also control the manipulator to perform various arm actions such as wrist rotation, arm flexion and extension, arm adduction and abduction, etc. The disclosed embodiment is not only simple to operate, but also has high accuracy in manipulating the manipulator. It is also flexible to use, and can flexibly set finger actions, arm actions, etc. according to the needs of the application scenario. It has high flexibility in use and can be implemented using low-cost hardware.

[0089] Figure 5 The structure diagram of the robot remote control device based on the arm ring provided by the embodiment of the present disclosure is shown. Figure 5, the armband-based robotic remote control device 500 may include:

[0090] An acquisition unit 501 is used to acquire a first electromyographic signal from an electromyographic sensor on the armband and first motion data from an inertial measurement unit on the armband when the user wears the armband;

[0091] A manipulator control unit 502, configured to control a first finger of the manipulator to perform a first action based on a first electromyographic signal;

[0092] The robot arm control unit 503 is used to control the robot arm to perform a second action based on the first motion data.

[0093] Furthermore, the manipulator control unit 502 can be specifically used to: extract characteristic data of the first electromyographic signal; use the electromyographic model to process the characteristic data of the first electromyographic signal to identify the user's desired finger movement and generate corresponding finger movement instructions, the finger movement instructions including information of the first finger and its movement instructions, the first finger including one or more of the thumb, index finger, middle finger, ring finger and little finger of the manipulator; and, control the first finger of the manipulator to perform a first movement according to the finger movement instruction.

[0094] Furthermore, the robotic arm control unit 503 can be specifically used to: analyze the first motion data to obtain first posture data indicating a first posture of the user's arm, the first posture data including a roll angle, a pitch angle and a heading angle; and, based on the first posture data, control the robotic arm to perform a second action, the second action including one or more of the following: wrist rotation, arm flexion and extension, and arm adduction and abduction.

[0095] Furthermore, the robotic arm control unit 503 can be specifically used to: verify whether the roll angle in the first posture data is within the predetermined effective range of the roll angle, whether the pitch angle is within the predetermined effective range of the pitch angle, and whether the heading angle is within the predetermined effective range of the heading angle; if the roll angle in the first posture data is within the effective range of the roll angle, the pitch angle is within the effective range of the pitch angle, and the heading angle is within the effective range of the heading angle, then control the robotic arm to perform a second action based on the first posture data; if the roll angle in the first posture data is not within the effective range of the roll angle, the pitch angle is not within the effective range of the pitch angle, or the heading angle is not within the effective range of the heading angle, then control the robotic arm not to perform any action.

[0096] Furthermore, the armband-based robot remote control device 500 may also include: an effective range determination unit 504, which is used to: determine the effective range of the roll angle according to the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs the wrist rotation action; and / or, determine the effective range of the pitch angle according to the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user wears the armband and performs the arm flexion and extension action; and / or, determine the effective range of the heading angle according to the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs the arm abduction and adduction action.

[0097] For other technical details about the armband-based robot remote control device 500, please refer to the section on the armband-based robot remote control method above, which will not be repeated here.

[0098] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of the aforementioned armband-based robot remote control method are executed.

[0099] Figure 6 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present disclosure. Figure 6 The electronic device 600 may include: one or more processors 601, and also includes a memory 602 storing one or more programs, which are executed by the one or more processors 601 to implement the method flow shown in the above embodiments of the present disclosure and / or program units corresponding to each unit in the device.

[0100] The various components are interconnected using different buses and can be mounted on a common motherboard or in other ways as needed. The processor 601 can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories.

[0101] The processor 601 may include one or more single-core processors or multi-core processors. The processor 601 may include any combination of general-purpose processors or dedicated processors (such as image processors, application processors, baseband processors, etc.).

[0102] The memory 602 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the following in the embodiments of the present disclosure: Figure 3The processor 601 executes the program instructions / units corresponding to the armband-based robot remote control method. The processor 601 executes the non-transient software programs, instructions and units stored in the memory 602, thereby executing the above method embodiments. Figure 3 The program, instructions and units corresponding to the armband-based robot remote control method are shown.

[0103] The electronic device 600 may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0104] The input device 603 can receive input digital or character information, and generate signal input related to user settings and function control, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 605 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0105] The above-mentioned programs (also referred to as software, software applications, or codes) include machine instructions for programmable processors, and these computer programs may be implemented using object-oriented programming languages, assembly or machine languages.

[0106] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the dissemination path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the Internet. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be used, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, device or device.

[0107] The technical solution provided by the present disclosure is described in detail above. The principles and implementation methods of the present disclosure are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present disclosure. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present disclosure.

[0108] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A robot remote control method based on an armband, characterized in that: The robot comprises a mechanical arm and a mechanical hand connected to each other, and an electromyographic sensor and an inertial measurement unit are installed on the arm ring; the method comprises: When the user wears the armband, obtaining a first electromyographic signal from an electromyographic sensor on the armband and first motion data from an inertial measurement unit on the armband; Controlling a first finger of the manipulator to perform a first action based on the first electromyographic signal; controlling the robotic arm to perform a second action based on the first motion data; Wherein, controlling the mechanical arm to perform a second action based on the first motion data comprises: analyzing the first motion data to obtain first posture data indicating a first posture of the user's arm, the first posture data comprising a roll angle, a pitch angle and a heading angle; and controlling the mechanical arm to perform a second action based on the first posture data, the second action comprising one or more of the following: wrist rotation, arm flexion and extension, and arm adduction and abduction; Wherein, controlling the robotic arm to perform a second action based on the first posture data includes one or more of the following: Determine whether the roll angle in the first posture data is within a first predetermined range or a second predetermined range, and if the roll angle is within the first predetermined range, control the end of the robot arm to move in the positive direction of the X-axis of the robot arm so that the robot arm performs a left-hand wrist rotation action; if the roll angle is within the second predetermined range, control the end of the robot arm to move in the negative direction of the X-axis of the robot arm so that the robot arm performs a right-hand wrist rotation action; Determine whether the pitch angle in the first posture data is within a third predetermined range or a fourth predetermined range, and if the pitch angle is within the third predetermined range, control the end of the robotic arm to move along the positive direction of the Y axis of the robotic arm so that the robotic arm performs an arm bending action, and if the pitch angle is within the fourth predetermined range, control the end of the robotic arm to move along the negative direction of the Y axis of the robotic arm so that the robotic arm performs an arm extending action; Determine whether the heading angle in the first posture data is within a fifth predetermined range or a sixth predetermined range, and if the heading angle is within the fifth predetermined range, control the end of the robotic arm to move in the positive direction of the Z axis of the robotic arm so that the robotic arm performs an arm adduction action; if the heading angle is within the sixth predetermined range, control the end of the robotic arm to move in the negative direction of the Z axis of the robotic arm so that the robotic arm performs an arm abduction action; Wherein, the first predetermined range and the second predetermined range can be determined according to the effective range of the roll angle and the first predetermined threshold, and the effective range of the roll angle is determined according to the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs a wrist rotation action; the effective range of the roll angle is the interval [a1, a2], the first predetermined threshold is set to a0, the first predetermined range is [a1, a1+a0], and the second predetermined range is [a2-a0, a2]; The third predetermined range and the fourth predetermined range are determined according to the effective range of the pitch angle and the second predetermined threshold, and the effective range of the pitch angle is determined according to the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user wears the armband and performs arm flexion and extension; the effective range of the pitch angle is the interval [b1, b2], the second predetermined threshold is set to b0, the third predetermined range is [b1, b1+b0], and the fourth predetermined range is [b2-b0, b2]; The fifth predetermined range and the sixth predetermined range can be determined according to the effective range of the heading angle and the third predetermined threshold value, and the effective range of the heading angle is determined according to the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs the arm abduction and adduction action; the effective range of the heading angle is the interval [c1, c2], the second predetermined threshold value is set to c0, the fifth predetermined range is [c1, c1+c0], and the sixth predetermined range is [c2-c0, c2].

2. The method according to claim 1, characterized in that The controlling the manipulator to perform a first action based on the first electromyographic signal includes: Extracting characteristic data of the first electromyographic signal; Processing the characteristic data of the first electromyographic signal by using an electromyographic model to identify the finger movement desired by the user and generate a corresponding finger movement instruction, wherein the finger movement instruction includes information of the first finger and its movement instruction, wherein the first finger includes one or more of the thumb, index finger, middle finger, ring finger and little finger of the manipulator; The first finger of the manipulator is controlled to perform the first action according to the finger action instruction.

3. The method according to claim 1, characterized in that The controlling the robotic arm to perform a second action based on the first posture data comprises: Verifying whether the roll angle in the first posture data is within a predetermined roll angle valid range, whether the pitch angle is within a predetermined pitch angle valid range, and whether the heading angle is within a predetermined heading angle valid range; If the roll angle in the first posture data is within the roll angle valid range, the pitch angle is within the pitch angle valid range, and the heading angle is within the heading angle valid range, then controlling the robotic arm to perform the second action based on the first posture data; If the roll angle in the first posture data is not within the effective range of the roll angle, the pitch angle is not within the effective range of the pitch angle, or the heading angle is not within the effective range of the heading angle, the robotic arm is controlled not to perform any action.

4. The method according to claim 3, characterized in that The effective range of the roll angle is determined according to the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs a wrist rotation action; and / or, The effective range of the pitch angle is determined according to the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user performs arm flexion and extension after wearing the armband; and / or, The effective range of the heading angle is determined according to the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs arm abduction and adduction actions.

5. A robot remote control device based on an armband, characterized in that: The robot comprises a mechanical arm and a mechanical hand connected to each other, and an electromyographic sensor and an inertial measurement unit are installed on the arm ring; the robot remote control device based on the arm ring comprises: an acquisition unit, configured to acquire, when a user wears the armband, a first electromyographic signal from an electromyographic sensor on the armband and first motion data from an inertial measurement unit on the armband; A manipulator control unit, configured to control a first finger of the manipulator to perform a first action based on the first electromyographic signal; a robotic arm control unit, configured to control the robotic arm to perform a second action based on the first motion data; Wherein, controlling the mechanical arm to perform a second action based on the first motion data comprises: analyzing the first motion data to obtain first posture data indicating a first posture of the user's arm, the first posture data comprising a roll angle, a pitch angle and a heading angle; and controlling the mechanical arm to perform a second action based on the first posture data, the second action comprising one or more of the following: wrist rotation, arm flexion and extension, and arm adduction and abduction; Wherein, controlling the robotic arm to perform a second action based on the first posture data includes one or more of the following: Determine whether the roll angle in the first posture data is within a first predetermined range or a second predetermined range, and if the roll angle is within the first predetermined range, control the end of the robot arm to move in the positive direction of the X-axis of the robot arm so that the robot arm performs a left-hand wrist rotation action; if the roll angle is within the second predetermined range, control the end of the robot arm to move in the negative direction of the X-axis of the robot arm so that the robot arm performs a right-hand wrist rotation action; Determine whether the pitch angle in the first posture data is within a third predetermined range or a fourth predetermined range, and if the pitch angle is within the third predetermined range, control the end of the robotic arm to move along the positive direction of the Y axis of the robotic arm so that the robotic arm performs an arm bending action, and if the pitch angle is within the fourth predetermined range, control the end of the robotic arm to move along the negative direction of the Y axis of the robotic arm so that the robotic arm performs an arm extending action; Determine whether the heading angle in the first posture data is within a fifth predetermined range or a sixth predetermined range, and if the heading angle is within the fifth predetermined range, control the end of the robotic arm to move in the positive direction of the Z axis of the robotic arm so that the robotic arm performs an arm adduction action; if the heading angle is within the sixth predetermined range, control the end of the robotic arm to move in the negative direction of the Z axis of the robotic arm so that the robotic arm performs an arm abduction action; Wherein, the first predetermined range and the second predetermined range can be determined according to the effective range of the roll angle and the first predetermined threshold, and the effective range of the roll angle is determined according to the maximum roll angle and the minimum roll angle measured by the inertial measurement unit when the user wears the armband and performs a wrist rotation action; the effective range of the roll angle is the interval [a1, a2], the first predetermined threshold is set to a0, the first predetermined range is [a1, a1+a0], and the second predetermined range is [a2-a0, a2]; The third predetermined range and the fourth predetermined range are determined according to the effective range of the pitch angle and the second predetermined threshold, and the effective range of the pitch angle is determined according to the maximum pitch angle and the minimum pitch angle measured by the inertial measurement unit when the user wears the armband and performs arm flexion and extension; the effective range of the pitch angle is the interval [b1, b2], the second predetermined threshold is set to b0, the third predetermined range is [b1, b1+b0], and the fourth predetermined range is [b2-b0, b2]; The fifth predetermined range and the sixth predetermined range can be determined according to the effective range of the heading angle and the third predetermined threshold value, and the effective range of the heading angle is determined according to the maximum heading angle and the minimum heading angle measured by the inertial measurement unit when the user wears the armband and performs the arm abduction and adduction action; the effective range of the heading angle is the interval [c1, c2], the second predetermined threshold value is set to c0, the fifth predetermined range is [c1, c1+c0], and the sixth predetermined range is [c2-c0, c2].

6. An electronic device, characterized in that: include: One or more processors and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a program, wherein the program comprises instructions, and when the instructions are executed by one or more processors of a computing device, the instructions cause the computing device to execute the method according to any one of claims 1 to 4.

8. A robot remote control system, characterized in that: The robot comprises a mechanical arm and a mechanical hand connected to each other, and the robot remote control system comprises: an arm ring and an electronic device, wherein the electronic device is built into the robot or electrically connected to the robot; The armband includes an armband body and an electromyographic sensor and an inertial measurement unit installed on the armband body; The electronic device includes: a controller and a communication module, the communication module is electrically connected to the controller, the communication module can communicate with the electromyography sensor and the inertial measurement unit respectively, and the controller is used to execute the method according to any one of claims 1 to 4.

Images