An upper limb prosthesis control method and system based on multi-mode signals

Abstract

The present invention relates to the technical field of medical rehabilitation equipment, and in particular to a multi-mode signal-based upper limb prosthesis control method and system thereof; in the present invention, myoelectric signals are used to control the start and stop of the system, and visual signals are used to obtain a target object and its spatial position through a visual camera, thereby obtaining an inverse kinematic solution of the upper limb prosthesis arm joint, controlling the movement of the upper limb prosthesis arm joint, and realizing multifunctional control of the upper limb prosthesis through limited control signals, enriching the functions that can be realized by the upper limb prosthesis, effectively compensating for some functions lost by upper limb amputees, and reducing the abandonment rate of the upper limb prosthesis; the present invention divides the upper limb prosthesis control process into stages, and adopts different signal sources in different control stages, which can not only solve the problem of control redundancy, but also effectively utilize the signal source according to the characteristics of different signal sources.

Description

Technical Field

[0001] The present invention relates to the technical field of medical rehabilitation equipment, and in particular to a multi-mode signal-based upper limb prosthesis control method and system. Background Art

[0002] At present, there are about 85 million disabled people in my country, of which about 2.26 million are limb amputees, including 450,000 upper limb amputees, and the number of amputees is on the rise. The emergence and application of prosthetic limbs can effectively make up for some of the functions of amputees, but compared with human limbs, the movement effect of prosthetic limbs still has a great trend, especially in the control of prosthetic hands. If more prosthetic hand movement states can be realized within limited degrees of freedom, such as finger movements and finger forces, the abandonment rate of prosthetic limbs will be effectively reduced.

[0003] Currently, the main signal sources for controlling upper limb prostheses include visual, myoelectric, and voice signals. Although multiple signal sources are used in the system, the control of the prosthesis is mostly redundant. For example, the movement of the elbow joint of an upper limb prosthesis can be controlled by both visual and myoelectric control. However, the control of an upper limb prosthesis with multiple degrees of freedom cannot maintain accuracy. Moreover, for high-level amputees wearing upper limb prostheses, the accuracy and stability of intention recognition based on myoelectric signals also pose considerable challenges. Summary of the Invention

[0004] The main technical problem solved by the present invention is to provide an upper limb prosthesis control method based on multi-mode signals, divide the upper limb prosthesis control process into stages, and use different signal sources in different control stages, which can not only solve the problem of control redundancy, but also effectively utilize the signal source according to the characteristics of different signal sources; it also provides an upper limb prosthesis control system based on multi-mode signals.

[0005] To solve the above technical problems, the present invention adopts a technical solution: providing an upper limb prosthesis control method based on multi-mode signals, which includes the following steps:

[0006] Step S1: activating the control of the upper limb prosthesis through electromyographic signals;

[0007] Step S2: Acquire the target object and its spatial position through a visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint;

[0008] Step S3: fine-tuning the upper limb prosthetic arm joint and the prosthetic hand by controlling the posture signal or voice signal;

[0009] Step S4: Turn off the control of the upper limb prosthesis through the electromyographic signal.

[0010] As an improvement of the present invention, in step S1 , the myoelectric signal of a user's specific muscle is collected as the myoelectric control signal for starting the control operation.

[0011] As a further improvement of the present invention, in step S1, a threshold value of the myoelectricity of a specific muscle is set as a set threshold value of the myoelectricity control signal, and the opening and closing of the control work is determined according to the set threshold value of the myoelectricity control signal.

[0012] As a further improvement of the present invention, in step S2, the target object and its spatial position are acquired by a visual camera, and the inverse kinematics solution of the upper limb prosthetic arm joint is calculated according to the motion model of the upper limb prosthesis, thereby controlling the movement of the upper limb prosthetic arm joint.

[0013] As a further improvement of the present invention, in step S2, the corresponding motion function is called in the prosthetic hand knowledge base according to the target object to control the upper limb prosthesis to complete the corresponding action.

[0014] As a further improvement of the present invention, in step S2, the method for obtaining the target object and its spatial position through the visual camera is that the visual camera adopts an RGB-D camera, which simultaneously obtains the RGB data stream and the Depth data stream, and then performs object prediction based on the RGB data through feature extraction and feature enhancement image processing methods, thereby obtaining the type of the target object and its planar position; then, according to the planar position of the target object, depth extraction is performed in the Depth data, and then position fusion is performed in combination with the plane coordinates to obtain the three-dimensional coordinates of the target object, thereby obtaining the target object and its spatial position.

[0015] As a further improvement of the present invention, in step S2, the method for calculating the inverse kinematic solution of the upper limb prosthetic arm joints based on the motion model of the upper limb prosthesis is: taking the visual camera coordinate system as the world coordinate system, combining the information of the installation position of the visual camera, establishing the transformation relationship between the camera coordinate system and the upper limb prosthetic end coordinate system, and calculating the inverse kinematic solution of each joint of the upper limb prosthetic arm according to the spatial position of the target object, thereby driving the upper limb prosthetic arm to reach the specified position.

[0016] As a further improvement of the present invention, in step S3, when the grasping mode of the target object is uncertain, the movement of the prosthetic hand can be fine-tuned and controlled through posture signals or voice signals based on the object information of the target object, or when the distance between the end of the upper limb prosthesis and the target object position exceeds the set distance, the movement of the upper limb prosthetic arm joint can be fine-tuned and controlled through posture signals or voice signals.

[0017] As a further improvement of the present invention, in step S1 , the emergency stop and reset of the upper limb prosthesis are controlled by electromyographic signals.

[0018] An upper limb prosthesis control system based on multi-mode signals, comprising:

[0019] Myoelectric signal module, used to control the opening and closing of upper limb prostheses;

[0020] The visual signal module is used to acquire the target object and its spatial position through the visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint;

[0021] Other signal modules are used to control the fine adjustment of the upper limb prosthetic arm joints and prosthetic hands.

[0022] The beneficial effects of the present invention are as follows: compared with the prior art, the electromyographic signal in the present invention is used to control the start and stop of the system, the visual signal is used to obtain the target object and its spatial position through a visual camera, and the inverse kinematic solution of the upper limb prosthetic arm joint is obtained to control the movement of the upper limb prosthetic arm joint. The multifunctional control of the upper limb prosthesis is achieved through limited control signals, the functions that can be achieved by the upper limb prosthesis are enriched, the partial functions lost by upper limb amputees are effectively compensated, and the abandonment rate of the upper limb prosthesis is reduced; the present invention divides the upper limb prosthesis control process into stages, and different control stages use different signal sources, which can not only solve the problem of control redundancy, but also effectively utilize the signal source according to the characteristics of different signal sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the staged control flow of the present invention;

[0024] Figure 2 Controlling movement flow of the upper limb prosthesis of the present invention;

[0025] Figure 3 Schematic diagram of the visual processing flow of the present invention. DETAILED DESCRIPTION

[0026] Please refer to Figures 1 to 3 The present invention provides an upper limb prosthesis control method based on multi-mode signals, comprising the following steps:

[0027] Step S1: Start the control of the upper limb prosthesis through electromyographic signals;

[0028] Step S2: Acquire the target object and its spatial position through a visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint;

[0029] Step S3: fine-tuning the upper limb prosthetic arm joint and prosthetic hand by controlling the posture signal or voice signal;

[0030] Step S4: Turn off the control of the upper limb prosthesis through the electromyographic signal.

[0031] In the present invention, myoelectric signals are used to control the start and stop of the system, and visual signals are used to obtain the target object and its spatial position through a visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, control the movement of the upper limb prosthetic arm joint, and realize multifunctional control of the upper limb prosthesis through limited control signals, enrich the functions that can be realized by the upper limb prosthesis, effectively compensate for some of the functions lost by upper limb amputees, and reduce the abandonment rate of upper limb prostheses; the present invention divides the upper limb prosthesis control process into stages, and different control stages use different signal sources, which can not only solve the problem of control redundancy, but also effectively utilize the signal source according to the characteristics of different signal sources.

[0032] In step S1, the myoelectric signal of a specific muscle of the user is collected as the myoelectric control signal for starting the control work; further, a threshold value of the myoelectricity of a specific muscle is set as the set threshold value of the myoelectric control signal, and the opening and closing of the control work is determined according to the set threshold value of the myoelectric control signal.

[0033] In step S2, the target object and its spatial position are acquired through a visual camera, and the inverse kinematics solution of the upper limb prosthetic arm joint is calculated based on the motion model of the upper limb prosthesis, thereby controlling the motion of the upper limb prosthetic arm joint; according to the target object, the corresponding motion function is called in the prosthetic hand knowledge base to control the upper limb prosthesis to complete the corresponding action.

[0034] Specifically, if Figure 1As shown in FIG, the present invention divides the control process of the upper limb prosthesis into the following: (1) Initial stage: When the amputee needs to use the upper limb prosthesis, this stage uses electromyographic signal control - the electromyographic signal is equivalent to a switch in this stage, and the control system starts to work. This function does not need to extract electromyographic features for intention recognition, but only needs to judge based on the threshold of the electromyography of a specific muscle, which can easily ensure the accuracy and stability of recognition; (2) Coarse control stage: After the initial stage is completed, the coarse control stage is entered. This stage uses visual control - through the visual camera and processing algorithm, the attribute information such as the type of the target object (bottle, key, etc.) in the field of view and its spatial position information are detected, providing an information basis for the control of the prosthesis and human-computer interaction, solving the inverse kinematics solution of the upper limb prosthesis arm joint, controlling the movement of the upper limb prosthesis arm joint, and then calling the appropriate operation according to the target object in the prosthesis hand knowledge base. Movement function (grasping, pinching, etc.); (3) Fine adjustment stage: When the coarse control stage is completed, the fine adjustment stage is entered. In this stage, posture or voice signal control is used. The amputee can fine-tune the arm joints according to the movement effect of the coarse control stage. When the movement error is large, other signal sources can be combined to fine-tune the arm joints. The most important task of this stage is to finely control the movement of the prosthetic hand according to the object information (shape, size, softness and hardness, etc.) of the target object when the grasping mode of the target object is uncertain; (4) Ending stage: When the fine adjustment stage is completed, the end stage is entered. In this stage, electromyographic signals are used. The purpose is to jump out of the system working state and prevent the visual system from being in the state of identifying objects and spatial positioning; (5) Full process monitoring: In the motion control process of the upper limb prosthesis, this system can complete the emergency stop or reset function through electromyographic control of the prosthesis to prevent the upper limb prosthesis from moving disorderly.

[0035] In the present invention, in step S2, the method for calculating the inverse kinematics solution of the upper limb prosthesis arm joints according to the motion model of the upper limb prosthesis is as follows: taking the visual camera coordinate system as the world coordinate system, combining the information of the installation position of the visual camera, establishing the conversion relationship between the camera coordinate system and the upper limb prosthesis end coordinate system, and calculating the inverse kinematics solution of each joint of the upper limb prosthesis arm according to the spatial position of the target object, thereby driving the upper limb prosthesis arm to reach the specified position; specifically, Figure 2 As shown in the figure, the control scheme for each joint of the arm in the upper limb prosthesis comes from the spatial position of the target object and the kinematic model of the upper limb prosthesis. The specific control details are: taking the camera coordinate system as the world coordinate system, combined with information such as the installation position of the RGB-D camera, a conversion relationship between the camera coordinate system and the upper limb prosthesis terminal coordinate system is established. According to the spatial position of the target object, the inverse kinematic solution of each joint of the arm is calculated, and finally the terminal device (prosthetic hand, etc.) is driven to the specified position. At the same time, according to the type of target object, the corresponding action can be called in the prosthetic hand control knowledge base. For example, when the target object is a mineral water bottle, the grasping action will be called in the prosthetic hand knowledge base.

[0036] In the present invention, in step S2, the method for obtaining the target object and its spatial position by a visual camera is that the visual camera adopts an RGB-D camera, which simultaneously obtains the RGB data stream and the Depth data stream, and then performs object prediction based on the RGB data through the image processing method of feature extraction and feature enhancement, thereby obtaining the type of the target object and its plane position; then, according to the plane position of the target object, depth extraction is performed in the Depth data, and then position fusion is performed in combination with the plane coordinates to obtain the three-dimensional coordinates of the target object, thereby obtaining the target object and its spatial position; specifically, if Figure 3 As shown in the figure, an RGB-D camera is used in the visual task to obtain RGB data streams and Depth data streams at the same time. Based on the RGB data, object prediction is mainly performed through image processing algorithms such as feature extraction and feature enhancement, and the target type and its plane position are finally obtained. According to the plane position, depth extraction is performed in the Depth data, and then position fusion is performed with the plane coordinates to obtain the three-dimensional coordinates of the target object, finally completing the acquisition of the visual task target and determining its spatial position.

[0037] In practical applications, for the motion control of upper limb prostheses, the RGB-D camera is fixed at the user's sternum because this position has fewer degrees of freedom and has little effect on the recognition accuracy of the target's spatial position.

[0038] In the present invention, in step S1 , the emergency stop and reset of the upper limb prosthesis are controlled by electromyographic signals.

[0039] In step S3, when the grasping mode of the target object is uncertain, the movement of the prosthetic hand can be fine-tuned and controlled through posture signals or voice signals based on the object information of the target object; or when the distance between the end of the upper limb prosthesis and the target object position exceeds the set distance, the movement of the upper limb prosthetic arm joint can be fine-tuned and controlled through posture signals or voice signals.

[0040] The present invention provides an upper limb prosthesis control system based on multi-mode signals, comprising:

[0041] Myoelectric signal module, used to control the opening and closing of upper limb prostheses;

[0042] The visual signal module is used to acquire the target object and its spatial position through the visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint;

[0043] Other signal modules are used to control the fine adjustment of the upper limb prosthetic arm joints and prosthetic hands.

[0044] The present invention primarily divides the control process of an upper limb prosthesis into stages, with different signal sources employed in different control stages. Signal sources such as myoelectric and visual systems are primarily used, with interfaces for signal sources such as posture sensors reserved. Myoelectric signals are used to control the start and stop of the system. The visual system perceives the surrounding environment based on a visual camera (acquiring current scene information and detecting attribute information such as the type and spatial position of objects within the field of view (e.g., bottles, keys), providing an information basis for precise prosthetic control and human-computer interaction). Combined with the kinematic model of the upper limb prosthesis, the inverse kinematic solution for the arm joint of the upper limb prosthesis is calculated, freeing up signal sources (e.g., myoelectric or posture sensors) for traditional upper limb prosthetic arm joint motion control. These freed signal sources can be applied to achieve detailed prosthetic hand movements (e.g., grasping or pinching). This approach enables multifunctional control of the upper limb prosthesis using limited control signals, enriching the functions achievable by the upper limb prosthesis, effectively compensating for the partial functions lost by upper limb amputees, and reducing the abandonment rate of upper limb prostheses.

[0045] In the present invention, based on the spatial position of the target object and the kinematic model of the upper limb prosthesis, the inverse kinematic solution of each arm joint in the upper limb prosthesis is calculated to control the movement of the arm joints, thereby liberating other signal sources such as electromyography and posture sensors.

[0046] In the present invention, the upper limb prosthesis control process is divided into levels, and different signal sources are used according to the operation requirements of different control stages (fine, coarse, start or stop, etc.), ultimately forming an upper limb prosthesis control method based on multi-mode signals.

[0047] The present invention has the following advantages:

[0048] 1. The upper limb prosthesis control process is divided into stages, mainly into the starting stage, coarse control stage, fine adjustment stage, ending stage and full process monitoring. Different signal sources are used in different control stages.

[0049] 2. Myoelectric signals are used as the signal source in the starting stage, ending stage and full-process monitoring. Different from the myoelectric-based motion recognition in myoelectric prostheses, the myoelectric signals in this invention are only used to start and stop the user's control system, which can improve the accuracy and stability of myoelectric signal recognition and improve the use effect of upper limb prostheses.

[0050] 3. In the coarse control stage, a visual camera is used as the signal source. The use of a visual camera can help the upper limb prosthetic system perceive the environment and identify the target object's attribute information (including type, etc.) and its spatial position.

[0051] 4. The control of the upper limb prosthesis in the present invention reserves other control interfaces (posture sensor and voice, etc.), which are responsible for the control in the fine-tuning stage and will enrich the upper limb prosthesis control system.

[0052] 5. The control signal of the present invention uses visual signals and also utilizes electromyographic signals as auxiliary control signals, while reserving interfaces for other signal sources, thus forming a multi-source controlled upper limb prosthesis system with vision as the main source and electromyographic and other types of signal sources as the auxiliary sources.

[0053] 6. The electromyographic signals used in the present invention are mainly used to collect the electromyographic signals of the user's specific muscles (pectoralis major) to control the operation of the system. This function does not require the extraction of complex electromyographic features for intention recognition. It only needs to judge based on the threshold of the electromyography of the specific muscle, which can easily ensure the accuracy and stability of recognition.

[0054] 7. The visual system of the present invention is mainly based on the RGB-D camera hardware platform and software platforms such as deep learning. The RGB-D camera is responsible for collecting RGB data streams and Depth data streams, and the RGB data and Depth data are processed by the deep learning algorithm to obtain the target object and its spatial position.

[0055] 8. The upper limb prosthesis system of the present invention will obtain the spatial position of the target object in the field of view based on the visual system, and combine it with the upper limb prosthesis kinematic model to obtain the inverse kinematic solution of the upper limb prosthetic arm joint, control the movement of the upper limb prosthetic arm joint, and then call the appropriate movement function (grasping, pinching, etc.) in the prosthetic hand knowledge base according to the target object.

[0056] In the present invention, the upper limb prosthesis control process is divided, and the appropriate signal source is selected according to the task requirements of each process; a multi-source control scheme for upper limb prostheses is adopted with visual signals as the main method and electromyographic signals and other signal sources as the auxiliary methods; visual sensors are used in upper limb prosthesis control to liberate the signal sources of traditional prostheses to control arm joint movements; the liberated signal sources are used to control the prosthetic hand, and the prosthetic hand is controlled to perform actions that are not pre-set in the prosthetic hand knowledge base, thereby enriching the usage scenarios of the upper limb prosthesis; the type of target object in the image and its plane position are obtained through image processing algorithms, and based on the plane position, the corresponding depth in the depth data is obtained, and finally the acquisition of the target object in the visual task and the determination of its spatial position are completed.

[0057] In the present invention, the camera coordinate system is used as the world coordinate system, and a transformation relationship between the camera coordinate system and the upper limb prosthetic terminal coordinate system is established. According to the spatial position of the target object, the inverse kinematic solution of each joint of the arm is calculated, and finally the terminal device (prosthetic hand, etc.) is driven to the specified position.

[0058] The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for controlling an upper limb prosthesis based on multimodal signals, characterized in that: The steps include: Step S1: Start the control of the upper limb prosthesis through electromyographic signals; Step S2: Acquire the target object and its spatial position through a visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint; Step S3: fine-tuning the upper limb prosthetic arm joint and the prosthetic hand by controlling the posture signal or voice signal; Step S4: shutting down the control of the upper limb prosthesis through the electromyographic signal; In step S2, the target object and its spatial position are acquired by a visual camera, and the inverse kinematics solution of the upper limb prosthetic arm joint is calculated according to the motion model solution of the upper limb prosthetic arm, thereby controlling the motion of the upper limb prosthetic arm joint; In step S3, when the grasping mode of the target object is uncertain, the movement of the prosthetic hand can be fine-tuned and controlled through posture signals or voice signals based on the object information of the target object; or when the distance between the end of the upper limb prosthesis and the target object position exceeds the set distance, the movement of the upper limb prosthetic arm joint can be fine-tuned and controlled through posture signals or voice signals.

2. The upper limb prosthesis control method based on multi-mode signals according to claim 1, characterized in that: In step S1 , the myoelectric signal of a user's specific muscle is collected as a myoelectric control signal for starting the control operation.

3. The upper limb prosthesis control method based on multi-mode signals according to claim 2, characterized in that: In step S1 , a threshold value of the myoelectricity of a specific muscle is set as a set threshold value of the myoelectricity control signal, and the on and off of the control operation is determined according to the set threshold value of the myoelectricity control signal.

4. The upper limb prosthesis control method based on multi-mode signals according to claim 1, characterized in that: In step S2, the corresponding motion function is called in the prosthetic hand knowledge base according to the target object, and the upper limb prosthesis is controlled to complete the corresponding action.

5. The upper limb prosthesis control method based on multi-mode signals according to claim 4, characterized in that: In step S2, the method for obtaining the target object and its spatial position through the visual camera is that the visual camera adopts an RGB-D camera, which simultaneously obtains the RGB data stream and the Depth data stream, and then performs object prediction based on the RGB data through feature extraction and feature enhancement image processing methods, thereby obtaining the type of the target object and its plane position; then, according to the plane position of the target object, depth extraction is performed in the Depth data, and then position fusion is performed in combination with the plane coordinates to obtain the three-dimensional coordinates of the target object, thereby obtaining the target object and its spatial position.

6. The upper limb prosthesis control method based on multi-mode signals according to claim 5, characterized in that: In step S2, the method for calculating the inverse kinematic solutions of the upper limb prosthetic arm joints based on the motion model of the upper limb prosthesis is as follows: taking the visual camera coordinate system as the world coordinate system, combining the information of the installation position of the visual camera, establishing the transformation relationship between the camera coordinate system and the upper limb prosthetic end coordinate system, and calculating the inverse kinematic solutions of each joint of the upper limb prosthetic arm according to the spatial position of the target object, thereby driving the upper limb prosthetic arm to reach the specified position.

7. The upper limb prosthesis control method based on multi-mode signals according to claim 1 or 6, characterized in that: In step S1 , the emergency stop and reset of the upper limb prosthesis are controlled by myoelectric signals.

8. An upper limb prosthesis control system based on multi-mode signals, characterized in that: An upper limb prosthesis control method based on multi-mode signals according to any one of claims 1 to 7 is adopted; the upper limb prosthesis control system comprises: Myoelectric signal module, used to control the opening and closing of upper limb prostheses; The visual signal module is used to acquire the target object and its spatial position through the visual camera, obtain the inverse kinematics solution of the upper limb prosthetic arm joint, and control the movement of the upper limb prosthetic arm joint; Other signal modules are used to control the fine adjustment of the upper limb prosthetic arm joints or prosthetic hands.

Images