A control method, device and medium based on intelligent prosthesis

The intelligent prosthetic limb control method, which combines electromyography signals and image recognition technology, solves the problem of difficulty in controlling grasping force, and achieves precise grasping and stable operation.

CN114469465BActive Publication Date: 2024-11-19浪潮工业互联网股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111628374.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-11-19
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

Existing intelligent prosthetics have difficulty controlling the gripping force when grasping objects, which can lead to damage to fragile items or unstable gripping.

Method used

By using electromyography (EMG) signal acquisition and image recognition technology, combined with a pre-trained recognition model, the system identifies the target to be grasped and determines the appropriate grasping pattern and force, and makes real-time adjustments using a gliding sensor and electrode unit.

Benefits of technology

It achieves accurate identification and force control of the grasping target, avoids damage to fragile items and unstable grasping, and improves the operational accuracy and safety of the intelligent prosthesis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114469465B_ABST
    Figure CN114469465B_ABST
Patent Text Reader

Abstract

The present application discloses a control method, device and medium based on intelligent prosthesis, the method includes: the intelligent prosthesis collects electromyographic signals at the residual limb of the patient through an electromyographic signal acquisition device, processes and analyzes the electromyographic signals, and determines a first action mode according to the analysis results; collects environmental images and human eye images through an image acquisition device, and determines the action execution target in the environmental image; identifies and processes the action execution target through a pre-trained first recognition model to determine a second action mode; identifies and processes the action execution target through a pre-trained second recognition model to determine the action execution strength. The grasping target is accurately identified, and at the same time, the intelligent prosthesis can be controlled to adopt different grasping strengths for different grasping targets, thereby avoiding damage to fragile items caused by excessive grasping strength. At the same time, the grasping strength is adjusted in real time to avoid the falling of the grasping target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent prostheses, and in particular to a control method, device and medium based on intelligent prostheses. Background Art

[0002] With the development of society and the continuous improvement of industrial level, more and more patients have undergone amputation due to trauma caused by machines or car accidents, which has brought many inconveniences to the patients' lives. Therefore, in order to solve the inconveniences in the lives of amputees, more and more intelligent prostheses have been developed.

[0003] At present, intelligent prostheses usually work by electromyography. When the muscles of the patient's residual limbs contract, complex biochemical reactions will occur, generating a tiny potential difference that can be measured on the skin surface. This electromyographic potential difference signal is transmitted to the microsensor, amplified by the amplifier in the electrode, and becomes a control signal. It is input into the microcomputer, and then the microcomputer issues an activity instruction to drive the opening and closing of the joints of the intelligent prosthesis through a drive system such as a micromotor.

[0004] However, when patients use smart prostheses to grasp objects, it is difficult to control the grasping force, which can easily cause some fragile objects to break due to excessive grasping force. At the same time, it is also difficult for patients to perceive the grasping force of the smart prosthesis. Summary of the invention

[0005] In order to solve the above problems, the present application proposes a control method, device and medium based on intelligent prosthesis, including:

[0006] In the first aspect, the present application proposes a control method based on an intelligent prosthesis, comprising: the intelligent prosthesis collects electromyographic signals at the patient's residual limb through an electromyographic signal acquisition device, and processes and analyzes the electromyographic signals to obtain an analysis result; determines a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grasping mode, a columnar object grasping mode, and a fingertip pinching mode; collects an environmental image and a human eye image through an image acquisition device on the intelligent prosthesis, and determines an action execution target in the environmental image according to the human eye image; identifies and processes the action execution target through a pre-trained first recognition model to obtain a first recognition result, and determines a second action mode according to the first recognition result, wherein the second action mode includes at least one of the following: side grasping, top grasping, and bottom grasping; identifies and processes the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determines the action execution intensity according to the second recognition result.

[0007] In one example, the action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined based on the first recognition result, specifically including: inputting an environmental image containing the action execution target into a pre-trained first recognition model to extract graphic features of the action execution target and contact surface features surrounding the action execution target through the first recognition model, wherein the contact surface features are appearance features of the contact object supporting or pulling the action execution target; constructing a three-dimensional image corresponding to the action execution target based on the graphic features, and obtaining appearance parameters of the three-dimensional image; determining a selection range of the second action mode based on the appearance parameters and pre-stored hand parameters of the intelligent prosthesis, wherein the selection range includes: side grabbing, top grabbing, and bottom grabbing; and selecting the optimal execution mode within the selection range as the second action mode based on the contact surface features.

[0008] In one example, the action execution target is identified and processed by a pre-trained second recognition model to obtain a second recognition result, and the action execution force is determined based on the second recognition result, specifically including: inputting an environmental image containing the action execution target into a pre-trained second recognition model to extract characteristic parameters of the action execution target through the second recognition model; based on the characteristic parameters, querying a database for object parameters whose similarity with the characteristic parameters reaches a preset threshold, wherein the object parameters include at least one of the following: object type, object weight, and object material; analyzing and calculating the object parameters to obtain the minimum grasping static friction of the action execution target, and using the minimum grasping static friction as the action execution force.

[0009] In one example, after analyzing and calculating the object parameters to obtain the minimum grasping static friction of the action execution target, and using the minimum grasping net friction as the action execution force, the method further includes: obtaining a force adjustment value based on the minimum grasping static friction and a pre-stored force adjustment level; attempting to grasp the action execution target based on the first action mode, the second action mode and the action execution force; collecting sliding signals in real time through the sliding sensor of the intelligent prosthetic hand, and judging whether there is sliding based on the sliding signals; if not, keeping the action execution force unchanged; if so, adjusting the action execution force step by step based on the pre-stored force adjustment level and the force adjustment value until it is judged that there is no sliding based on the sliding signal.

[0010] In one example, an environmental image and a human eye image are captured by an image capture device on the intelligent prosthesis, and an action execution target in the environmental image is determined based on the human eye image, specifically including: capturing a patient's human eye image through the image capture device on the intelligent prosthesis, obtaining a cross-section and a vertical section of the patient's eyeball based on the human eye image, and taking the extension direction of the cross-section as a gaze direction; constructing a reference axis parallel to the vertical section with the position of the image capture device as the origin; obtaining a target acquisition direction of the image capture device based on the gaze direction and the reference axis, and controlling the image capture device to rotate to the target acquisition direction; capturing an environmental image at the target acquisition direction through the image capture device, and taking an object target that coincides with the center point of the environmental image as an action execution target.

[0011] In one example, determining the first action mode according to the analysis result specifically includes: according to the analysis result, querying the action mode with the highest matching degree with the analysis result in the database through a pre-trained fuzzy matching model, the action mode including at least one of the following: spherical object grasping mode, columnar object grasping mode, fingertip pinching mode; taking the action mode with the highest matching degree as the first action mode.

[0012] In one example, an intelligent prosthesis collects electromyographic signals at a patient's residual limb through an electromyographic signal acquisition device, and processes and analyzes the electromyographic signals. Before obtaining the analysis results, the method further includes: the intelligent prosthesis obtains the patient's action intention, and collects multiple training electromyographic signals at the patient's residual limb according to the action intention, wherein the action intention includes at least one of the following: grasping a spherical object, grasping a column, and pinching with fingertips; the multiple training electromyographic signals and the corresponding action intentions are input into an initial fuzzy matching model to perform fuzzy rule training to obtain multiple fuzzy rules; and a fuzzy matching model is constructed according to the multiple fuzzy rules.

[0013] In one example, after the action execution target is identified and processed by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined based on the second recognition result, the method further includes: obtaining a corresponding bioelectric intensity based on the action execution intensity; simulating a stimulation current consistent with the bioelectric intensity through the electrode unit on the intelligent prosthesis; and introducing the stimulation current into the patient's residual limb.

[0014] On the other hand, the present application proposes a control device based on an intelligent prosthesis, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions: collect electromyographic signals at the patient's residual limb through an electromyographic signal acquisition device, and process and analyze the electromyographic signals to obtain analysis results; determine a first action mode based on the analysis results, wherein the first action mode includes at least one of the following: a spherical object grasping mode, a columnar object grasping mode, and a fingertip pinching mode; collect an environmental image and a human eye image through an image acquisition device on the intelligent prosthesis, and determine an action execution target in the environmental image based on the human eye image; identify and process the action execution target through a pre-trained first recognition model to obtain a first recognition result, and determine a second action mode based on the first recognition result, wherein the second action mode includes at least one of the following: side grasping, top grasping, and bottom grasping; identify and process the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determine the action execution intensity based on the second recognition result.

[0015] On the other hand, the present application proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to: collect electromyographic signals at the patient's residual limb through an electromyographic signal acquisition device, and process and analyze the electromyographic signals to obtain an analysis result; determine a first action mode based on the analysis result, wherein the first action mode includes at least one of the following: a spherical object grasping mode, a columnar object grasping mode, and a fingertip pinching mode; collect environmental images and human eye images through an image acquisition device on the intelligent prosthesis, and determine an action execution target in the environmental image based on the human eye image; identify and process the action execution target through a pre-trained first recognition model to obtain a first recognition result, and determine a second action mode based on the first recognition result, wherein the second action mode includes at least one of the following: side grasping, top grasping, and bottom grasping; identify and process the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determine the action execution intensity based on the second recognition result.

[0016] The control method, device and medium based on intelligent prosthesis proposed in this application can bring the following beneficial effects: accurately identify the grasping target, and at the same time, control the intelligent prosthesis to adopt different grasping forces for different grasping targets, thereby avoiding damage to fragile items caused by excessive grasping force, and at the same time, adjusting the grasping force in real time to avoid the falling of the grasping target. In addition, a corresponding stimulation electrical signal is sent to the patient according to the grasping force, so that the patient can understand the current grasping force. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A flow chart of a control method based on an intelligent prosthesis in an embodiment of the present application;

[0019] Figure 2 This is a schematic diagram of an image acquisition device in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of a control device based on an intelligent prosthesis in an embodiment of the present application. Description of the drawings:

[0022] 10-intelligent prosthesis, 20-image acquisition device, 30-eyeball, 40-cross section, 50-vertical section, 60-action execution target, 70-reference axis, 80-target acquisition direction, 90-rotation angle. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] First of all, it should be noted that the control method based on intelligent prosthesis recorded in this application can be stored in a corresponding system in the form of a program or algorithm, and the system can be installed in the intelligent prosthesis. To support the operation of the system, the intelligent prosthesis should be equipped with corresponding components, such as a processor, a memory, a communication module, etc., to support the above-mentioned program or algorithm. At the same time, the intelligent prosthesis should include other hardware to support all the technical solutions recorded in this application, such as: an electromyographic signal acquisition device, an image acquisition device, a sliding sensor, an electrode unit, etc.

[0025] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, a control method based on an intelligent prosthesis provided in an embodiment of the present application includes:

[0027] S101: The intelligent prosthesis collects electromyographic signals from the residual limb of the patient through an electromyographic signal collection device, and processes and analyzes the electromyographic signals to obtain analysis results.

[0028] Specifically, the intelligent prosthesis is connected to the patient's residual limb, and the intelligent prosthesis described in the present application has an electromyographic sensing function. It is provided with an electromyographic signal acquisition device, which can be attached to the outer skin of the patient's residual limb. When the patient has a certain grasping consciousness, there will be corresponding electromyographic signals on the residual limb, which can be collected by the electromyographic signal acquisition device.

[0029] Furthermore, the intelligent prosthesis is provided with a processor, which can perform corresponding analysis and processing on the collected electromyographic signals. It should be noted that before the electromyographic signals are processed, the electromyographic signals should also be amplified by an amplifier to ensure the strength of the electromyographic signals.

[0030] Then, the analysis results are obtained.

[0031] S102: Determine a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode.

[0032] Specifically, based on the analysis results, the intelligent prosthesis uses a pre-trained fuzzy matching model to query the database for the action mode that has the highest matching degree with the analysis results, and the action modes include at least one of the following: spherical object grasping mode, columnar object grasping mode, and fingertip pinching mode.

[0033] Furthermore, the action mode with the highest matching degree is taken as the first action mode, that is, any one of the spherical object grasping mode, the columnar object grasping mode, and the fingertip pinching mode.

[0034] It should be noted that a memory may be provided inside the intelligent prosthesis as a database. The intelligent prosthesis may also establish a wireless connection with a remote server and database through a communication module, thereby accessing the database. The specific implementation form is not specifically limited in the embodiments of the present application.

[0035] In addition, when the patient consciously uses a certain grasping mode, a specific electromyographic signal will be generated at the residual limb. However, for the same grasping mode, the generated electromyographic signals are not completely consistent. Therefore, in the embodiment of the present application, a fuzzy matching model is introduced to realize the above solution.

[0036] S103: collecting an environment image and a human eye image through an image collection device on the intelligent prosthesis, and determining an action execution target in the environment image according to the human eye image.

[0037] Specifically, Figure 2 As shown, the intelligent prosthesis 10 is provided with an image acquisition device 20, which includes a camera and a rotating device. The camera can be rotated according to the rotating device to acquire images at different angles.

[0038] It should be noted that the image acquisition device 20 here can automatically track the position of the patient's eyeball 30 after the intelligent prosthesis determines the first action mode according to the analysis results, that is, it determines the position of the eyeball 30 by rotating 360°, and after determining that the camera is aimed at the eyeball 30, it can acquire the human eye image.

[0039] Furthermore, the intelligent prosthesis 10 collects the patient's eye image, that is, the image of the eyeball 30 , through the image acquisition device 20 on the intelligent prosthesis. The acquisition method may adopt eyeball feature recognition to align the camera.

[0040] It should be noted that a processing unit with corresponding computing power should be provided inside the intelligent prosthesis 10, which can store the corresponding program for eye feature recognition, that is, the image acquisition device 20 is controlled by the processing unit, and after the processing unit determines that the camera has been aimed at the eyeball 30, the rotating device is controlled to stop rotating. In addition, if the processing unit fails to recognize the eyeball 30 after controlling the rotating device to rotate 360°, for example, the patient's arm is currently in a drooping state and the eyeball 30 cannot be recognized, the voice prompt device provided on the intelligent prosthesis 10 can be used to prompt the patient to lift the residual limb in the form of voice, thereby driving the position of the intelligent prosthesis 10 to rise until the human eye image is collected in the above manner, or the intelligent prosthesis 10 is automatically raised a certain distance through the control of the processing unit until the human eye image is collected in the above manner.

[0041] It should also be noted that the above technical solution relies on the processing unit inside the intelligent prosthesis 10 to perform corresponding processing calculations. In addition, a connection with a remote server can be established through a communication module set in the intelligent prosthesis 10 to utilize the computing power of the remote server to support the implementation of the above technical solution.

[0042] Furthermore, the cross-section 40 and the vertical section 50 of the patient's eyeball are obtained based on the human eye image, and the extension direction of the cross-section 40 is used as the gaze direction. It should be noted that the extension direction here is the direction of the eyeball's forward gaze, and extending in this direction can intersect with the action execution target 60.

[0043] Furthermore, the intelligent prosthesis constructs a reference axis 70 parallel to the vertical section with the position of the image acquisition device as the origin.

[0044] Furthermore, based on the gaze direction (i.e., the extension direction of the cross section 40) and the reference axis 70, the target acquisition direction 80 of the image acquisition device is obtained, that is, the value of the rotation angle 90 is calculated with the reference axis as the center point, and the camera is rotated according to the rotation angle 90 to align with the target acquisition direction 80. According to the reference axis 70 and the gaze direction (i.e., the extension direction of the cross section 40), the rotation angle 90 can be calculated by trigonometric functions.

[0045] Then, the image acquisition device 20 acquires the environment image at the target acquisition direction 80, and the object target that coincides with the center point of the environment image is used as the action execution target. That is, when the image acquisition device 20 rotates to the target acquisition direction 60, the direction in which the camera is pointed intersects with the patient's gaze direction, and at this time, the action execution target must exist at the center of the acquired environment image.

[0046] S104: Identify and process the action execution target through a pre-trained first recognition model to obtain a first recognition result, and determine a second action mode based on the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing.

[0047] Specifically, the intelligent prosthesis inputs an environmental image containing an action execution target into a pre-trained first recognition model to extract graphic features of the action execution target and contact surface features surrounding the action execution target through the first recognition model.

[0048] It should be noted that the graphic features are the various elements that constitute the action execution target, including but not limited to arcs, circles, rectangles, trapezoids, etc. The contact surface features are the appearance features of the supporting items that support or pull the action execution target, including but not limited to table tops, pull ropes, etc. That is, the contact surface features are the appearance features of the contact objects that support or pull the action execution target.

[0049] Furthermore, a three-dimensional image corresponding to the action execution target is constructed according to the graphic features, and appearance parameters of the three-dimensional image are obtained, the appearance parameters including but not limited to: length, width, height, diameter, cross-sectional area, etc.

[0050] Then, according to the appearance parameters and the pre-stored hand parameters of the intelligent prosthesis, the selection range of the second action mode is determined, wherein the selection range includes: side grabbing, top grabbing, and bottom grabbing. For example, according to the appearance parameters, the action execution target is determined to be a cylinder, and according to the hand parameters, it is determined that the hand can wrap around and hold all or part of the cylinder. Therefore, the selection range at this time can include all of the above methods.

[0051] Then, according to the contact surface features, the best execution mode is selected within the selection range as the second action mode. For example, according to the contact surface features, the above-mentioned cylinder is placed on the table, and the bottom grabbing can be filtered out at this time. For the grabbing of the cylinder, the stability of the side grabbing is much higher than the stability of the top grabbing, that is, the side grabbing can be selected as the second action mode according to the pre-setting for grabbing this type of object.

[0052] S105: The action execution target is recognized by using a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result.

[0053] Specifically, the intelligent prosthesis inputs an environment image containing an action execution target into a pre-trained second recognition model to extract feature parameters of the action execution target through the second recognition model.

[0054] Furthermore, according to the characteristic parameters, item parameters whose similarity with the characteristic parameters reaches a preset threshold are searched in the database and / or the Internet, and the item parameters include at least one of the following: item type, item weight, and item material.

[0055] Then, the object parameters are analyzed and calculated to obtain the minimum static friction force of the action execution target, and the minimum static friction force of the action execution target is used as the action execution force.

[0056] It should be noted that the above-mentioned object parameters can reflect the relevant parameters of the action execution target.

[0057] In addition, by adopting two recognition models, namely the first recognition model and the second recognition model, two data processing operations can be performed in parallel, thereby reducing the response time of the intelligent prosthesis.

[0058] In addition, in one embodiment, after analyzing and calculating the object parameters to obtain the minimum grabbing friction force of the action execution target and using the minimum grabbing static friction force as the action execution force, the following can also be done:

[0059] The force adjustment value is obtained according to the minimum grabbing static friction force and the pre-stored force adjustment gear number. For example, if the pre-stored force adjustment gear number is 5 and the minimum grabbing static friction force is 10N, the minimum grabbing static friction force can be evenly divided according to the force adjustment gear number to obtain a force adjustment value of 2N.

[0060] Then, according to the first action mode, the second action mode and the action execution strength, an attempt is made to grasp the action execution target.

[0061] The sliding sensor of the intelligent prosthetic hand can collect sliding signals in real time, and determine whether sliding occurs based on the sliding signals.

[0062] If not, the action execution force is kept unchanged, and the action execution target is captured in front of the patient's eyes.

[0063] If yes, it means that there is sliding at this time, and the action execution force needs to be increased. That is, the action execution force is adjusted step by step according to the pre-stored force adjustment level and the force adjustment value until it is determined that there is no sliding according to the sliding signal. For example, if the current action execution force is 10N, it is increased by 2N each time, up to 5 times, until there is no sliding.

[0064] This technical solution prevents certain fragile items from being broken due to excessive gripping force, while at the same time ensuring that the items will not fall due to insufficient gripping force.

[0065] In one embodiment, the intelligent prosthesis collects electromyographic signals from the residual limb of the patient through an electromyographic signal collection device, and processes and analyzes the electromyographic signals. Before obtaining the analysis results, the method further includes:

[0066] The intelligent prosthesis obtains the patient's movement intention and collects multiple training electromyographic signals at the patient's residual limb according to the movement intention, wherein the movement intention includes at least one of the following: grasping a spherical object, grasping a columnar object, and pinching with fingertips.

[0067] It should be noted that the action intention can be used as an instruction through the patient's voice or other means so that the intelligent prosthesis can obtain the action intention. At the same time, the action intention should contain a specific type, that is, the type supported by the intelligent prosthesis.

[0068] Furthermore, the intelligent prosthesis inputs a plurality of training electromyographic signals and corresponding action intentions into an initial fuzzy matching model to perform fuzzy rule training and obtain a plurality of fuzzy rules.

[0069] Then, a fuzzy matching model is constructed according to a plurality of fuzzy rules, and the corresponding first motion mode can be determined through the patient's electromyographic signal and the fuzzy matching model.

[0070] In one embodiment, after the action execution target is recognized by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result, the method further includes:

[0071] The corresponding bioelectric intensity is obtained according to the intensity of the action.

[0072] Furthermore, the electrode units on the intelligent prosthesis simulate stimulation currents that are consistent with the bioelectric intensity.

[0073] Then, a stimulation current is introduced into the residual limb of the patient. For example, the stimulation current can be changed from 0.05A to 0.25A.

[0074] Through the above technical solution, the patient can sense the strength of the stimulation current at the residual limb and then sense the magnitude of the force of the action.

[0075] In one embodiment, Figure 3 As shown, the present application also provides a control device based on an intelligent prosthesis, comprising:

[0076] at least one processor; and,

[0077] a memory communicatively connected to the at least one processor; wherein,

[0078] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions:

[0079] The electromyographic signal is collected by an electromyographic signal collection device at the residual limb of the patient, and the electromyographic signal is processed and analyzed to obtain an analysis result;

[0080] Determining a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode;

[0081] The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image;

[0082] The action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing;

[0083] The action execution target is recognized and processed by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result.

[0084] In one embodiment, the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0085] The electromyographic signal is collected by an electromyographic signal collection device at the residual limb of the patient, and the electromyographic signal is processed and analyzed to obtain an analysis result;

[0086] Determining a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode;

[0087] The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image;

[0088] The action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing;

[0089] The action execution target is recognized and processed by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result.

[0090] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0091] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0092] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0093] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0100] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A control method based on intelligent prosthesis, characterized in that: include: The intelligent prosthesis collects electromyographic signals from the residual limb of the patient through the electromyographic signal collection device, and processes and analyzes the electromyographic signals to obtain analysis results; Determining a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode; The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image; The action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing; Recognize the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determine the action execution intensity according to the second recognition result; The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image, specifically including: The image acquisition device on the intelligent prosthesis acquires the patient's eye image, obtains the cross section and the vertical section of the patient's eyeball according to the eye image, and uses the extension direction of the cross section as the gaze direction; Taking the position of the image acquisition device as the origin, constructing a reference axis parallel to the vertical section; Obtaining a target acquisition direction of the image acquisition device according to the gaze direction and the reference axis, and controlling the image acquisition device to rotate to the target acquisition direction; The image acquisition device is used to acquire an environmental image in the target acquisition direction, and an object target that coincides with the center point of the environmental image is used as an action execution target; The action execution target is recognized by using a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained first recognition model, so as to extract graphic features of the action execution target and contact surface features surrounding the action execution target through the first recognition model, wherein the contact surface features are appearance features of a contact object supporting or pulling the action execution target; constructing a three-dimensional image corresponding to the action execution target according to the graphic features, and acquiring appearance parameters of the three-dimensional image; Determine a selection range of the second action mode according to the appearance parameters and the pre-stored hand parameters of the intelligent prosthesis, wherein the selection range includes: side grabbing, top grabbing, and bottom grabbing; According to the contact surface characteristics, selecting the best execution mode within the selection range as the second action mode; The action execution target is identified by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained second recognition model to extract feature parameters of the action execution target through the second recognition model; According to the characteristic parameters, querying the database for item parameters whose similarity with the characteristic parameters reaches a preset threshold, wherein the item parameters include at least one of the following: item type, item weight, and item material; Analyze and calculate the object parameters to obtain the minimum grabbing static friction force of the action execution target, and use the minimum grabbing static friction force as the action execution force; After analyzing and calculating the object parameters to obtain the minimum grabbing static friction force of the action execution target and using the minimum grabbing static friction force as the action execution force, the method further includes: Obtaining a force adjustment value according to the minimum grabbing static friction force and the pre-stored force adjustment level number; Attempting to grasp the action execution target according to the first action mode, the second action mode, and the action execution strength; The sliding sensor of the intelligent prosthetic hand is used to collect sliding signals in real time, and to determine whether sliding occurs according to the sliding signals; If not, the execution intensity of the action is kept unchanged; If so, the action execution force is adjusted step by step according to the pre-stored force adjustment level number and the force adjustment value, until it is determined according to the sliding signal that there is no sliding.

2. The control method based on intelligent prosthesis according to claim 1, characterized in that: Determining the first action mode according to the analysis result specifically includes: According to the analysis result, the action mode with the highest matching degree with the analysis result is searched in the database through the pre-trained fuzzy matching model, and the action mode includes at least one of the following: a spherical object grasping mode, a columnar object grasping mode, and a fingertip pinching mode; The action mode with the highest matching degree is used as the first action mode.

3. The control method based on intelligent prosthesis according to claim 1, characterized in that: The intelligent prosthesis collects electromyographic signals at the residual limb of the patient through the electromyographic signal collection device, and processes and analyzes the electromyographic signals. Before obtaining the analysis results, the method further includes: The intelligent prosthesis obtains the patient's action intention and collects a plurality of training electromyographic signals at the residual limb of the patient according to the action intention, wherein the action intention includes at least one of the following: grasping a spherical object, grasping a columnar object, and pinching with fingertips; Inputting the plurality of training electromyographic signals and the corresponding action intentions into an initial fuzzy matching model to perform fuzzy rule training to obtain a plurality of fuzzy rules; A fuzzy matching model is constructed according to the multiple fuzzy rules.

4. The control method based on intelligent prosthesis according to claim 1, characterized in that: After the action execution target is identified by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result, the method further includes: Obtaining a corresponding bioelectric intensity according to the execution intensity of the action; Simulating a stimulation current consistent with the bioelectric intensity by using an electrode unit on the intelligent prosthesis; The stimulation current is introduced into the residual limb of the patient.

5. A control device based on intelligent prosthesis, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the following instructions: The electromyographic signal is collected by an electromyographic signal collection device at the residual limb of the patient, and the electromyographic signal is processed and analyzed to obtain an analysis result; Determining a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode; The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image; The action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing; Recognize the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determine the action execution intensity according to the second recognition result; The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image, specifically including: The image acquisition device on the intelligent prosthesis acquires the patient's eye image, obtains the cross section and the vertical section of the patient's eyeball according to the eye image, and uses the extension direction of the cross section as the gaze direction; Taking the position of the image acquisition device as the origin, constructing a reference axis parallel to the vertical section; Obtaining a target acquisition direction of the image acquisition device according to the gaze direction and the reference axis, and controlling the image acquisition device to rotate to the target acquisition direction; The image acquisition device is used to acquire an environmental image in the target acquisition direction, and an object target that coincides with the center point of the environmental image is used as an action execution target; The action execution target is recognized by using a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained first recognition model, so as to extract graphic features of the action execution target and contact surface features surrounding the action execution target through the first recognition model, wherein the contact surface features are appearance features of a contact object supporting or pulling the action execution target; constructing a three-dimensional image corresponding to the action execution target according to the graphic features, and acquiring appearance parameters of the three-dimensional image; Determine a selection range of the second action mode according to the appearance parameters and the pre-stored hand parameters of the intelligent prosthesis, wherein the selection range includes: side grabbing, top grabbing, and bottom grabbing; According to the contact surface characteristics, selecting the best execution mode within the selection range as the second action mode; The action execution target is identified by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained second recognition model to extract feature parameters of the action execution target through the second recognition model; According to the characteristic parameters, querying the database for item parameters whose similarity with the characteristic parameters reaches a preset threshold, wherein the item parameters include at least one of the following: item type, item weight, and item material; Analyze and calculate the object parameters to obtain the minimum grabbing static friction force of the action execution target, and use the minimum grabbing static friction force as the action execution force; After analyzing and calculating the object parameters to obtain the minimum grabbing static friction force of the action execution target and using the minimum grabbing static friction force as the action execution force, the method further includes: Obtaining a force adjustment value according to the minimum grabbing static friction force and the pre-stored force adjustment level number; Attempting to grasp the action execution target according to the first action mode, the second action mode, and the action execution strength; The sliding sensor of the intelligent prosthetic hand is used to collect sliding signals in real time, and to determine whether sliding occurs according to the sliding signals; If not, the execution intensity of the action is kept unchanged; If so, the action execution force is adjusted step by step according to the pre-stored force adjustment level number and the force adjustment value, until it is determined according to the sliding signal that there is no sliding.

6. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: The electromyographic signal is collected by an electromyographic signal collection device at the residual limb of the patient, and the electromyographic signal is processed and analyzed to obtain an analysis result; Determining a first action mode according to the analysis result, wherein the first action mode includes at least one of the following: a spherical object grabbing mode, a columnar object grabbing mode, and a fingertip pinching mode; The image acquisition device on the intelligent prosthesis collects an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image; The action execution target is recognized and processed by a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, wherein the second action mode includes at least one of the following: side grabbing, top grabbing, and bottom grabbing; Recognize the action execution target through a pre-trained second recognition model to obtain a second recognition result, and determine the action execution intensity according to the second recognition result; The image acquisition device on the intelligent prosthesis acquires an environmental image and a human eye image, and determines an action execution target in the environmental image according to the human eye image, specifically including: The image acquisition device on the intelligent prosthesis acquires the patient's eye image, obtains the cross section and the vertical section of the patient's eyeball according to the eye image, and uses the extension direction of the cross section as the gaze direction; Taking the position of the image acquisition device as the origin, constructing a reference axis parallel to the vertical section; Obtaining a target acquisition direction of the image acquisition device according to the gaze direction and the reference axis, and controlling the image acquisition device to rotate to the target acquisition direction; The image acquisition device is used to acquire an environmental image in the target acquisition direction, and an object target that coincides with the center point of the environmental image is used as an action execution target; The action execution target is recognized by using a pre-trained first recognition model to obtain a first recognition result, and a second action mode is determined according to the first recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained first recognition model, so as to extract graphic features of the action execution target and contact surface features surrounding the action execution target through the first recognition model, wherein the contact surface features are appearance features of a contact object supporting or pulling the action execution target; constructing a three-dimensional image corresponding to the action execution target according to the graphic features, and acquiring appearance parameters of the three-dimensional image; Determine a selection range of the second action mode according to the appearance parameters and the pre-stored hand parameters of the intelligent prosthesis, wherein the selection range includes: side grabbing, top grabbing, and bottom grabbing; According to the contact surface characteristics, selecting the best execution mode within the selection range as the second action mode; The action execution target is identified by a pre-trained second recognition model to obtain a second recognition result, and the action execution intensity is determined according to the second recognition result, specifically including: Inputting an environment image containing the action execution target into a pre-trained second recognition model to extract feature parameters of the action execution target through the second recognition model; According to the characteristic parameters, querying the database for item parameters whose similarity with the characteristic parameters reaches a preset threshold, wherein the item parameters include at least one of the following: item type, item weight, and item material; Analyze and calculate the object parameters to obtain the minimum grabbing static friction force of the action execution target, and use the minimum grabbing static friction force as the action execution force; After analyzing and calculating the object parameters to obtain the minimum grabbing static friction force of the action execution target and using the minimum grabbing static friction force as the action execution force, the method further includes: Obtaining a force adjustment value according to the minimum grabbing static friction force and the pre-stored force adjustment level number; Attempting to grasp the action execution target according to the first action mode, the second action mode, and the action execution strength; The sliding sensor of the intelligent prosthetic hand is used to collect sliding signals in real time, and to determine whether sliding occurs according to the sliding signals; If not, the execution intensity of the action is kept unchanged; If so, the action execution force is adjusted step by step according to the pre-stored force adjustment level number and the force adjustment value, until it is determined according to the sliding signal that there is no sliding.