Motion intention recognition method and system based on myoelectricity and muscle ultrasound
By combining electromyography and muscle ultrasound signals, a variety of recognition modes and automatic recognition modes are provided, which solves the accuracy and stability of surface electromyography signals when identifying fine movements of the hand, and improves the accuracy of motor intention recognition and people Reliability of computer interaction.
Patent Information
- Application Number
- CN202410928635.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-26
Smart Images

Figure CN119025955A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of human-computer interaction technology, and in particular to a method and system for identifying movement intention based on electromyography and muscle ultrasound. Background Art
[0002] In the past few decades, hand motion recognition has become one of the most important technologies in the field of human-computer interaction and rehabilitation engineering. Decoding the user's movement intentions, such as making a fist and opening the hand, from human physiological signals such as surface electromyography (SEMG) and electroencephalography (EEG) can provide intuitive neural control for prosthetic hands and other auxiliary rehabilitation devices. Surface electromyography signals have been widely used in hand motion recognition due to their simple acquisition, non-invasiveness and high temporal resolution. However, the spatial resolution of surface electromyography is low, and the surface electromyography signals collected on the skin surface are a combination of electrical signals generated by different muscle fibers. This makes it difficult to detect small and deep muscle activities from surface electromyography signals, resulting in a low accuracy rate in identifying fine hand movements. In addition, due to some inherent limitations of surface electromyography, it is easily affected by environmental electromagnetic interference, skin impedance changes, etc., which greatly hinders its clinical application. Summary of the invention
[0003] The embodiment of the present application provides a method for identifying movement intention based on electromyography and muscle ultrasound to solve the problems in the prior art, such as low accuracy in identifying fine hand movements, and some inherent limitations of surface electromyography, which is easily affected by environmental electromagnetic interference, changes in skin impedance, etc., which greatly hinders its clinical application.
[0004] Correspondingly, the embodiments of the present application also provide a movement intention recognition system based on electromyography and muscle ultrasound, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.
[0005] In order to solve the above technical problems, the embodiment of the present application discloses a method for identifying movement intention based on electromyography and muscle ultrasound, the method comprising:
[0006] Obtaining indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method respectively, and displaying the indicators on the system interface; wherein the indicators include the motion recognition accuracy rate;
[0007] In response to the user selecting an output mode according to the indicator, the following operations are performed:
[0008] If the user selects the electromyography unimodal mode, the electromyography unimodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyography unimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyography signals and processing the electromyography signals using a preset machine algorithm;
[0009] If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information;
[0010] If the user selects the dual-modal mode, the dual-modal motion intention recognition method is used to obtain the motion intention recognition result; the dual-modal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information and using a preset machine algorithm to process the electromyographic signals and muscle ultrasonic information;
[0011] If the user selects the automatic recognition mode, the indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method will be automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator will be output.
[0012] The embodiment of the present application also discloses a system for identifying movement intention based on electromyography and muscle ultrasound, the system comprising:
[0013] An index generation module is used to obtain the indexes of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method, and display the indexes on the system interface; wherein the indexes include the motion recognition accuracy rate;
[0014] The identification output module is used to respond to the user's operation of selecting a corresponding output mode according to the indicator and perform the following operations:
[0015] If the user selects the electromyography unimodal mode, the electromyography unimodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyography unimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyography signals and processing the electromyography signals using a preset machine algorithm;
[0016] If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information;
[0017] If the user selects the dual-modal mode, the dual-modal motion intention recognition method is used to obtain the motion intention recognition result; the dual-modal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information and using a preset machine algorithm to process the electromyographic signals and muscle ultrasonic information;
[0018] If the user selects the automatic recognition mode, the indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method will be automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator will be output.
[0019] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more of the methods described in the embodiments of the present application are implemented.
[0020] The embodiments of the present application further disclose a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the embodiments of the present application are implemented.
[0021] In the embodiment of the present application, the user can select any output mode of the electromyographic unimodal mode, the muscle ultrasound unimodal mode, the dual-modal mode and the automatic recognition mode based on the indicators displayed on the system interface, and the method of use is flexible. The electromyographic unimodal mode is identified based on electromyographic signals, the muscle ultrasound unimodal mode is identified based on muscle ultrasound information, and the dual-modal mode is jointly identified based on electromyographic signals and muscle ultrasound information. Among them, muscle ultrasound information can realize non-invasive detection of deep muscles, making up for the low spatial resolution of electromyographic signals, and it is difficult to detect the problem of small and deep muscle activities from electromyographic signals. In the embodiment of the present application, the advantages of the high temporal resolution of electromyographic signals and the high spatial resolution of muscle ultrasound information are fully combined, and the comprehensive detection of muscle electrophysiological and structural morphological information during limb movement can be realized, and the accuracy of movement intention recognition can be improved. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator in the above three modes according to the indicators of each movement intention recognition method, and the movement intention recognition result with the highest accuracy in the three modes can be obtained. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or mechanical arms, and improve the safety and reliability of prosthetic use and mechanical arm control.
[0022] Additional aspects and advantages of the embodiments of the present application will be given in the following description, which will become apparent from the following description or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0024] Figure 1 A system block diagram of a method for identifying movement intention based on electromyography and muscle ultrasound provided in an embodiment of the present application;
[0025] Figure 2 A flowchart for obtaining action recognition results in each mode provided in the embodiments of the present application;
[0026] Figure 3 A schematic diagram of the structure of a system for identifying movement intention based on electromyography and muscle ultrasound provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0029] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0030] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0031] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. For the technical problems existing in the prior art, the method and system for identifying motion intention based on electromyography and muscle ultrasound provided in this application are intended to solve at least one of the technical problems of the prior art.
[0032] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0033] An embodiment of the present application provides a possible implementation method and a flowchart of a method for identifying movement intention based on electromyography and muscle ultrasound. The scheme can be executed by any electronic device, and optionally, can be executed on a server or terminal device.
[0034] Combination Figure 1 , the method may include the following steps:
[0035] The indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are obtained respectively, and the indicators are displayed on the system interface. Among them, the indicators include the motion recognition accuracy.
[0036] In the embodiment of the present application, the indicators of the electromyographic unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are displayed on the system interface in the form of offline data. Taking the action recognition accuracy as an example, after the system is turned on, the system interface displays the action recognition accuracy of the electromyographic unimodal motion intention recognition method (which can be referred to as the electromyographic signal action recognition accuracy), the action recognition accuracy of the muscle ultrasound unimodal motion intention recognition method (which can be referred to as the muscle ultrasound action recognition accuracy), and the action recognition accuracy of the dual-modal motion intention recognition method (which can be referred to as the electromyographic ultrasound fusion action recognition accuracy).
[0037] Each time the user uses the system, he can refer to the indicators on the system interface to select the output mode, where the output modes include electromyography single-mode mode, muscle ultrasound single-mode mode, dual-mode mode and automatic recognition mode.
[0038] In response to the user selecting the corresponding output mode according to the indicator, the following operations are performed:
[0039] If the user selects the EMG unimodal mode, the EMG unimodal motion intention recognition method is used to obtain the motion intention recognition result; the EMG unimodal motion intention recognition method obtains the motion intention recognition result by collecting EMG signals and using a preset machine algorithm to process the EMG signals. Finally, it is displayed on the system interface, such as Figure 1 The electromyographic signal action recognition results are shown.
[0040] If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the action intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the action intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information. Finally, it is displayed on the system interface, such as Figure 1 The muscle ultrasound action recognition results are shown.
[0041] If the user selects the dual-mode mode, the dual-mode motion intention recognition method is used to obtain the motion intention recognition result; the dual-mode motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information and using a preset machine algorithm to process the electromyographic signals and muscle ultrasonic information. Finally, it is displayed on the system interface, such as Figure 1 The results of myoelectric and ultrasonic fusion action recognition are shown. In this mode, the accuracy of movement intention recognition is improved by combining the electrophysiological information of muscle movement (i.e., electromyographic signals) and muscle structure and morphological information (i.e., muscle ultrasonic information).
[0042] If the user selects the automatic recognition mode, the indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator is output. Figure 1 As shown in , the action intention recognition accuracy of the above three methods can be compared, and the action recognition results output by the method with the highest accuracy are displayed.
[0043] Optionally, the preset machine learning algorithm may be Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), Convolutional Neural Networks (CNN) or Long Short-Term Memory (LSTM).
[0044] In the embodiment of the present application, the user can select any output mode of the electromyographic unimodal mode, the muscle ultrasound unimodal mode, the dual-modal mode and the automatic recognition mode based on the indicators displayed on the system interface, and the method of use is flexible. The electromyographic unimodal mode is identified based on electromyographic signals, the muscle ultrasound unimodal mode is identified based on muscle ultrasound information, and the dual-modal mode is jointly identified based on electromyographic signals and muscle ultrasound information. Among them, muscle ultrasound information can realize non-invasive detection of deep muscles, making up for the low spatial resolution of electromyographic signals, and it is difficult to detect the problem of small and deep muscle activities from electromyographic signals. In the embodiment of the present application, the advantages of the high temporal resolution of electromyographic signals and the high spatial resolution of muscle ultrasound information are fully combined, and the comprehensive detection of muscle electrophysiological and structural morphological information during limb movement can be realized, and the accuracy of movement intention recognition can be improved. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator in the above three modes according to the indicators of each movement intention recognition method, and the movement intention recognition result with the highest accuracy in the three modes can be obtained. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or mechanical arms, and improve the safety and reliability of prosthetic use and mechanical arm control.
[0045] In an optional embodiment, the indicators also include the amplitude and signal-to-noise ratio of the electromyographic signal and muscle ultrasonic information.
[0046] In an embodiment of the present application, the system has the function of evaluating the quality of electromyography and muscle ultrasound information. The system can display the amplitude, signal-to-noise ratio, and motion recognition accuracy of the electromyography signal and muscle ultrasound information, as well as the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method, and the bimodal motion intention recognition method, which can help users select the corresponding output mode according to the above preparation.
[0047] In an optional embodiment, the indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are respectively obtained, and the indicators are displayed on the system interface, including:
[0048] Processing one or more segments of electromyographic signals by using the electromyographic unimodal motion intention recognition method to obtain a plurality of first motion intention recognition results, and calculating the motion recognition accuracy of the electromyographic unimodal motion intention recognition method according to the plurality of first motion intention recognition results;
[0049] Processing a plurality of muscle ultrasound information using a muscle ultrasound single-mode motion intention recognition method to obtain a plurality of second motion intention recognition results, and calculating the motion recognition accuracy of the muscle ultrasound single-mode motion intention recognition method according to the plurality of second motion intention recognition results;
[0050] Using a dual-modal motion intention recognition method to process multiple sets of time-synchronized electromyographic signals and muscle ultrasonic information, a plurality of third motion intention recognition results are obtained, and the motion recognition accuracy of the dual-modal motion intention recognition method is calculated based on the plurality of third motion intention recognition results;
[0051] The motion recognition accuracy of the electromyography unimodal motion intention recognition method, the motion recognition accuracy of the muscle ultrasound unimodal motion intention recognition method, and the motion recognition accuracy of the dual-modal motion intention recognition method are displayed on the system interface.
[0052] In an optional embodiment, one or more segments of electromyographic signals are processed using an electromyographic unimodal motion intention recognition method to obtain multiple first motion intention recognition results, and the motion recognition accuracy of the electromyographic unimodal motion intention recognition method is calculated based on the multiple first motion intention recognition results, including:
[0053] Using electromyographic equipment to obtain one or more segments of electromyographic signals on the skin surface of the user when the user performs different hand movements;
[0054] Using a preset machine algorithm to extract features of the electromyographic signal, and identifying the features of the electromyographic signal to obtain a plurality of first action intention recognition results;
[0055] The proportion of the correct number of the multiple first action intention recognition results in the total number is calculated to obtain the action recognition accuracy of the electromyography single-modal motion intention recognition method.
[0056] In an embodiment of the present application, an electromyographic device is used to collect multiple segments of electromyographic signals, and for each sampling point, the action performed by the user at the sampling point is used as a label to generate a sample. A data set of electromyographic signals is generated based on multiple samples. The data set of electromyographic signals is input into a preset machine algorithm, and the first action intention recognition result corresponding to each sample and the action recognition accuracy of the electromyographic unimodal motion intention recognition method are output. Among them, the first action intention recognition result of each sample is compared with the label to determine whether it is correct, and the number of correct samples is counted. Finally, the action recognition accuracy of the electromyographic unimodal motion intention recognition method is obtained by dividing the number of correct samples by the total number of samples.
[0057] In an optional embodiment, a plurality of muscle ultrasound information is processed using a muscle ultrasound single-mode motion intention recognition method to obtain a plurality of second motion intention recognition results, and the motion recognition accuracy of the muscle ultrasound single-mode motion intention recognition method is calculated according to the plurality of second motion intention recognition results, including:
[0058] Using an ultrasound device to obtain ultrasound information of multiple muscles on the skin surface of a user when the user performs different hand movements;
[0059] Using a preset machine algorithm to extract features of each muscle ultrasonic information, and identifying the features of each muscle ultrasonic information to obtain multiple second action intention recognition results;
[0060] The proportion of the correct number of the multiple second action intention recognition results in the total number is calculated to obtain the action recognition accuracy of the muscle ultrasound single-mode motion intention recognition method.
[0061] In an embodiment of the present application, an ultrasonic device is used to collect multiple muscle ultrasonic information. For each muscle ultrasonic information, the action performed by the user under the muscle ultrasonic information is used as a label to generate a sample. A data set of muscle ultrasonic information is generated based on multiple samples. The data set of muscle ultrasonic information is input into a preset machine algorithm, and the second action intention recognition result corresponding to each sample and the action recognition accuracy of the muscle ultrasonic single-modal motion intention recognition method are output. Among them, the second action intention recognition result of each sample is compared with the label to determine whether it is correct, and the number of correct samples is counted. Finally, the action recognition accuracy of the muscle ultrasonic single-modal motion intention recognition method is obtained by dividing the number of correct samples by the total number of samples.
[0062] In an optional embodiment, a dual-modal motion intention recognition method is used to process multiple groups of time-synchronized electromyographic signals and muscle ultrasonic information to obtain multiple third motion intention recognition results, and the motion recognition accuracy of the dual-modal motion intention recognition method is calculated based on the multiple third motion intention recognition results, including:
[0063] Using electromyographic equipment and ultrasonic equipment to synchronously obtain multiple groups of electromyographic signals and muscle ultrasonic information on the skin surface of the user when performing different hand movements;
[0064] Using a preset machine algorithm to fuse the electromyographic signal and muscle ultrasound information in each group, and identifying each group of fused information to obtain multiple third action intention recognition results;
[0065] The proportion of the correct number of the third action intention recognition results in the total number is calculated to obtain the action recognition accuracy of the dual-modal motion intention recognition method.
[0066] In an embodiment of the present application, an electromyographic device and an ultrasonic device are used to synchronously collect electromyographic signals and muscle ultrasonic information when the user performs different actions. The electromyographic signals and muscle ultrasonic information collected synchronously each time are grouped. For each group of electromyographic signals and muscle ultrasonic information, the action performed by the user is used as a label to generate a sample. A data set of electromyographic and ultrasonic fusion is generated based on multiple samples. The electromyographic and ultrasonic fusion data set is input into a preset machine algorithm, and the third action intention recognition result corresponding to each sample and the action recognition accuracy of the dual-modal motion intention recognition method are output. Among them, the third action intention recognition result of each sample is compared with the label to determine whether it is correct, and the number of correct samples is counted. Finally, the action recognition accuracy of the dual-modal motion intention recognition method is obtained by dividing the number of correct samples by the total number of samples.
[0067] In combination with the above, users can choose any of the four output modes: electromyography single mode, muscle ultrasound single mode, dual mode and automatic recognition mode according to their needs. Figure 2 As shown, an electromyographic device is used to collect electromyographic signals, and an ultrasonic device is used to collect muscle ultrasonic information. In the electromyographic single-mode mode, feature extraction and motion prediction are performed on the electromyographic signal, and then the motion result is output. In the muscle ultrasonic single-mode mode, feature extraction and motion prediction are performed on the muscle ultrasonic information, and then the motion result is output. In the dual-mode mode, feature extraction and fusion are performed on the electromyographic signal and the muscle ultrasonic information, and motion prediction is performed based on the fused information, and then the motion result is output. In the automatic recognition mode, the motion result based on the method with the highest accuracy is output according to the motion recognition accuracy of each method obtained in advance by the system.
[0068] In an optional embodiment, the way of fusing the electromyographic signal and the muscle ultrasound information in each group includes signal layer fusion, feature layer fusion and decision layer fusion;
[0069] Among them, signal layer fusion refers to statistically analyzing each sampling point of the electromyographic signal and each pixel value of the muscle ultrasound information to achieve the fusion of the electromyographic signal and the muscle ultrasound information;
[0070] Feature layer fusion refers to extracting the features of the electromyographic signal and the muscle ultrasonic information respectively, and fusing the features of the electromyographic signal and the muscle ultrasonic information;
[0071] Decision layer fusion refers to obtaining the electromyographic action intention recognition results and the muscle ultrasonic action intention recognition results based on the electromyographic signal and the muscle ultrasonic information respectively, and performing joint judgment and processing based on the electromyographic action intention recognition results and the muscle ultrasonic action intention recognition results.
[0072] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a system for identifying movement intention based on electromyography and muscle ultrasound, such as Figure 3 As shown, the system comprises:
[0073] The index generation module 301 is used to obtain the indexes of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method, and display the indexes on the system interface; wherein the indexes include the motion recognition accuracy rate;
[0074] The identification output module 302 is used to respond to the user's operation of selecting a corresponding output mode according to the indicator and perform the following operations:
[0075] If the user selects the electromyographic unimodal mode, the electromyographic unimodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyographic unimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and processing the electromyographic signals using a preset machine algorithm;
[0076] If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information;
[0077] If the user selects the dual-modal mode, the dual-modal motion intention recognition method is used to obtain the motion intention recognition result; the dual-modal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information, and using a preset machine algorithm to process the electromyographic signals and the muscle ultrasonic information;
[0078] If the user selects the automatic recognition mode, the indicators of the myoelectric unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator is output.
[0079] In the embodiment of the present application, the user can select any output mode of the electromyographic unimodal mode, the muscle ultrasound unimodal mode, the dual-modal mode and the automatic recognition mode based on the indicators displayed on the system interface, and the method of use is flexible. The electromyographic unimodal mode is identified based on electromyographic signals, the muscle ultrasound unimodal mode is identified based on muscle ultrasound information, and the dual-modal mode is jointly identified based on electromyographic signals and muscle ultrasound information. Among them, muscle ultrasound information can realize non-invasive detection of deep muscles, making up for the low spatial resolution of electromyographic signals, and it is difficult to detect the problem of small and deep muscle activities from electromyographic signals. In the embodiment of the present application, the advantages of the high temporal resolution of electromyographic signals and the high spatial resolution of muscle ultrasound information are fully combined, and the comprehensive detection of muscle electrophysiological and structural morphological information during limb movement can be realized, and the accuracy of movement intention recognition can be improved. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator in the above three modes according to the indicators of each movement intention recognition method, and the movement intention recognition result with the highest accuracy in the three modes can be obtained. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or mechanical arms, and improve the safety and reliability of prosthetic use and mechanical arm control.
[0080] The exercise intention recognition system based on electromyography and muscle ultrasound provided in the embodiment of the present application can achieve Figure 1 and Figure 2 To avoid repetition, the various processes implemented in the method embodiment are not described here.
[0081] The motion intention recognition system based on electromyography and muscle ultrasound of the embodiment of the present application can execute the motion intention recognition method based on electromyography and muscle ultrasound provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module and unit in the motion intention recognition system based on electromyography and muscle ultrasound in each embodiment of the present application correspond to the steps in the motion intention recognition method based on electromyography and muscle ultrasound in each embodiment of the present application. For the detailed functional description of each module of the motion intention recognition system based on electromyography and muscle ultrasound, please refer to the description of the corresponding motion intention recognition method based on electromyography and muscle ultrasound shown in the previous text, which will not be repeated here.
[0082] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the method for identifying the intention of movement based on electromyography and muscle ultrasound shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the method for identifying the intention of movement based on electromyography and muscle ultrasound provided by the present application allows the user to select any output mode of the electromyography unimodal mode, the muscle ultrasound unimodal mode, the dual-modal mode and the automatic recognition mode based on the indicators displayed on the system interface, and the method of use is flexible. The electromyography unimodal mode is identified based on electromyography signals, the muscle ultrasound unimodal mode is identified based on muscle ultrasound information, and the dual-modal mode is jointly identified based on electromyography signals and muscle ultrasound information. Among them, muscle ultrasound information can realize non-invasive detection of deep muscles, making up for the problem that the spatial resolution of electromyography signals is low and it is difficult to detect small and deep muscle activities from electromyography signals. The embodiments of the present application fully combine the advantages of high temporal resolution of electromyographic signals and high spatial resolution of muscle ultrasonic information, which can achieve comprehensive detection of muscle electrophysiological and structural morphological information during limb movement, and improve the accuracy of movement intention recognition. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator among the above three modes for recognition based on the indicators of each movement intention recognition method, and can obtain the movement intention recognition result with the highest accuracy among the three modes. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or robotic arms, and improve the safety and reliability of prosthetic use and robotic arm control.
[0083] In an optional embodiment, an electronic device is also provided, such as Figure 4 As shown, Figure 4 The electronic device 400 shown may be a server, including: a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the electronic device 400 may further include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.
[0084] Processor 401 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0085] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0086] The memory 403 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0087] The memory 403 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the contents shown in the above method embodiment.
[0088] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0089] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.
[0090] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.
[0091] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0092] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0093] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0094] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0095] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method and system for identifying movement intention based on electromyography and muscle ultrasound provided in the above-mentioned various optional implementations.
[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0098] The modules involved in the embodiments of the present application may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the recognition output module may also be described as "a recognition output module for performing the following operations in response to the user selecting the corresponding output mode according to the indicator".
[0099] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A method for identifying movement intention based on electromyography and muscle ultrasound, characterized in that: The method comprises: Obtaining indicators of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method respectively, and displaying the indicators on the system interface; wherein the indicators include the motion recognition accuracy rate; In response to the user selecting an output mode according to the indicator, the following operations are performed: If the user selects the electromyographic unimodal mode, the electromyographic unimodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyographic unimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and processing the electromyographic signals using a preset machine algorithm; If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information; If the user selects the dual-modal mode, the dual-modal motion intention recognition method is used to obtain the motion intention recognition result; the dual-modal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information, and using a preset machine algorithm to process the electromyographic signals and the muscle ultrasonic information; If the user selects the automatic recognition mode, the indicators of the myoelectric unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator is output.
2. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 1, characterized in that: The method of respectively obtaining the indicators of the electromyography single-mode motion intention recognition method, the muscle ultrasound single-mode motion intention recognition method and the dual-mode motion intention recognition method, and displaying the indicators on the system interface, includes: Processing one or more segments of electromyographic signals using the electromyographic unimodal motion intention recognition method to obtain a plurality of first motion intention recognition results, and calculating the motion recognition accuracy of the electromyographic unimodal motion intention recognition method based on the plurality of first motion intention recognition results; Processing a plurality of muscle ultrasound information using the muscle ultrasound single-mode motion intention recognition method to obtain a plurality of second motion intention recognition results, and calculating the motion recognition accuracy of the muscle ultrasound single-mode motion intention recognition method according to the plurality of second motion intention recognition results; Processing multiple groups of time-synchronized electromyographic signals and muscle ultrasonic information using the dual-modal motion intention recognition method to obtain multiple third motion intention recognition results, and calculating the motion recognition accuracy of the dual-modal motion intention recognition method based on the multiple third motion intention recognition results; The motion recognition accuracy of the myoelectric unimodal motion intention recognition method, the motion recognition accuracy of the muscle ultrasound unimodal motion intention recognition method, and the motion recognition accuracy of the dual-modal motion intention recognition method are displayed on the system interface.
3. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that: The method of using the electromyographic single-mode motion intention recognition method to process one or more segments of electromyographic signals to obtain multiple first motion intention recognition results, and calculating the motion recognition accuracy of the electromyographic single-mode motion intention recognition method according to the multiple first motion intention recognition results, includes: Using electromyographic equipment to obtain one or more segments of electromyographic signals on the skin surface of the user when the user performs different hand movements; Extracting features of the electromyographic signal using a preset machine algorithm, and identifying the features of the electromyographic signal to obtain a plurality of first action intention recognition results; The proportion of the correct number of the multiple first action intention recognition results in the total number is calculated to obtain the action recognition accuracy of the electromyographic single-modal motion intention recognition method.
4. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that: The method of using the muscle ultrasound single-mode motion intention recognition method to process a plurality of muscle ultrasound information to obtain a plurality of second motion intention recognition results, and calculating the motion recognition accuracy of the muscle ultrasound single-mode motion intention recognition method according to the plurality of second motion intention recognition results, comprises: Using an ultrasound device to obtain ultrasound information of multiple muscles on the skin surface of a user when the user performs different hand movements; Using a preset machine algorithm to extract features of each muscle ultrasonic information, and identifying the features of each muscle ultrasonic information to obtain multiple second action intention recognition results; The proportion of the correct number of the multiple second action intention recognition results in the total number is calculated to obtain the action recognition accuracy of the muscle ultrasound single-mode motion intention recognition method.
5. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that: The method of using the dual-modal motion intention recognition method to process multiple groups of time-synchronized electromyographic signals and muscle ultrasonic information to obtain multiple third motion intention recognition results, and calculating the motion recognition accuracy of the dual-modal motion intention recognition method according to the multiple third motion intention recognition results, includes: Using electromyographic equipment and ultrasonic equipment to synchronously obtain multiple groups of electromyographic signals and muscle ultrasonic information on the skin surface of the user when performing different hand movements; Using a preset machine algorithm to fuse the electromyographic signal and muscle ultrasound information in each group, and identifying each group of fused information to obtain multiple third action intention recognition results; The proportion of the correct number of the plurality of third motion intention recognition results in the total number is calculated to obtain the motion recognition accuracy of the dual-modal motion intention recognition method.
6. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 5, characterized in that: The methods of fusing the electromyographic signals and muscle ultrasound information in each group include signal layer fusion, feature layer fusion and decision layer fusion; The signal layer fusion refers to performing statistical analysis on each sampling point of the electromyographic signal and each pixel value of the muscle ultrasonic information to achieve fusion of the electromyographic signal and the muscle ultrasonic information; The feature layer fusion refers to extracting the features of the electromyographic signal and the muscle ultrasonic information respectively, and fusing the features of the electromyographic signal and the features of the muscle ultrasonic information; The decision layer fusion refers to obtaining the electromyographic action intention recognition result and the muscle ultrasonic action intention recognition result based on the electromyographic signal and the muscle ultrasonic information respectively, and performing joint judgment and processing based on the electromyographic action intention recognition result and the muscle ultrasonic action intention recognition result.
7. The method for identifying movement intention based on electromyography and muscle ultrasound according to claim 1, characterized in that: The indicators also include the amplitude and signal-to-noise ratio of the electromyographic signal and muscle ultrasonic information.
8. A system for identifying movement intention based on electromyography and muscle ultrasound, characterized in that: The system comprises: An index generation module, used to obtain the indexes of the electromyography unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method, and display the indexes on the system interface; wherein the indexes include the motion recognition accuracy rate; The identification output module is used to respond to the user's operation of selecting a corresponding output mode according to the indicator and perform the following operations: If the user selects the electromyographic unimodal mode, the electromyographic unimodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyographic unimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and processing the electromyographic signals using a preset machine algorithm; If the user selects the muscle ultrasound single-mode mode, the muscle ultrasound single-mode motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound single-mode motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and using a preset machine algorithm to process the muscle ultrasound information; If the user selects the dual-modal mode, the dual-modal motion intention recognition method is used to obtain the motion intention recognition result; the dual-modal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasonic information, and using a preset machine algorithm to process the electromyographic signals and the muscle ultrasonic information; If the user selects the automatic recognition mode, the indicators of the myoelectric unimodal motion intention recognition method, the muscle ultrasound unimodal motion intention recognition method and the dual-modal motion intention recognition method are automatically compared, and the motion intention recognition result obtained by the method corresponding to the optimal indicator is output.
9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
System and method for synchronously detecting multi-modal biological signals
CN103584919A
Electroencephalogram-myoelectricity dual-mode neural signal-based man-machine interaction system and method
CN108681396A
Emergency rescue gesture recognition method and device based on surface electromyogram signals
CN116400812A
Prosthesis control method, apparatus, system and device, and storage medium
WO2020118797A1
Cited By
Flexible integrated array of myoelectricity and ultrasound collaborative perception and preparation method thereof
CN120078442A
A fine motion control method for prosthetic hand based on muscle fiber micro perception
CN122848989B
