Motion intention recognition method and system based on electromyography and muscle ultrasound

Abstract

A motion intention recognition method and system based on electromyography and muscle ultrasound. A user can select, on the basis of indexes, any output mode of electromyography single modality, muscle ultrasound single modality, dual modality, and automatic recognition. The electromyography single modality mode performs recognition on the basis of an electromyographic signal. The muscle ultrasound single modality mode performs recognition on the basis of muscle ultrasound information. The dual modality mode performs combined recognition on the basis of an electromyographic signal and muscle ultrasound information. The muscle ultrasound information makes up for the deficiency that the spatial resolution of the electromyographic signal being low makes it difficult to detect small and deep muscle activities from the electromyographic signal. By fully combining the advantages of high time resolution of the electromyographic signal and high spatial resolution of the muscle ultrasound information, comprehensive detection of muscle electrophysiological and structural morphology information during limb movement is realized, and the recognition accuracy is improved. The automatic recognition mode can automatically select a mode corresponding to the optimal index for recognition to obtain an optimal recognition result.

Description

A method and system for motor intention recognition based on electromyography and muscle ultrasound Technical Field

[0001] This application relates to the field of human-computer interaction technology, specifically to a method and system for recognizing movement intentions based on electromyography and muscle ultrasound. Background Technology

[0002] Over the past few decades, hand movement recognition has become one of the most important technologies in the fields of human-computer interaction and rehabilitation engineering. Decoding a user's motor intentions, such as clenching a fist or opening a hand, from physiological signals like surface electromyography (SEMG) and electroencephalography (EEG) can provide intuitive neural control for prosthetic hands and other assistive rehabilitation devices. SEMG signals have been widely used in hand movement recognition due to their ease of acquisition, non-invasiveness, and high temporal resolution. However, SEMG suffers from low spatial resolution, as the signals acquired on the skin surface are combinations of electrical signals generated by different muscle fibers. This makes it difficult to detect small and deep muscle activities from SEMG signals, resulting in a low accuracy rate in recognizing fine hand movements. Furthermore, SEMG has inherent limitations, being susceptible to environmental electromagnetic interference and changes in skin impedance, which significantly hinders its clinical application.

[0003] Summary of the Invention

[0004] This application provides a method for recognizing movement intentions based on electromyography and muscle ultrasound, in order to solve the problems in the prior art, such as the low accuracy of recognizing fine hand movements and the inherent limitations of surface electromyography, which is easily affected by environmental electromagnetic interference, changes in skin impedance, etc., which greatly hinder its clinical application.

[0005] Accordingly, embodiments of this application also provide a motion intention recognition system based on electromyography and muscle ultrasound, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above methods.

[0006] To address the aforementioned technical problems, this application discloses a method for recognizing movement intentions based on electromyography and muscle ultrasound, the method comprising:

[0007] The system acquires the metrics for electromyography-based single-modal motion intention recognition, muscle ultrasound-based single-modal motion intention recognition, and bimodal motion intention recognition, and displays these metrics on the system interface. The metrics include motion recognition accuracy.

[0008] In response to the user's selection of the output mode based on the metrics, perform the following operations:

[0009] If the user selects the electromyography (EMG) monomodal mode, the EMG monomodal motion intention recognition method is used to obtain the motion intention recognition result. The EMG monomodal motion intention recognition method obtains the motion intention recognition result by collecting EMG signals and processing the EMG signals using a preset machine algorithm.

[0010] If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intent recognition method is used to obtain the motion intent recognition result. The muscle ultrasound monomodal motion intent recognition method obtains the motion intent recognition result by collecting muscle ultrasound information and processing the muscle ultrasound information using a preset machine algorithm.

[0011] If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the motion intention recognition result. The bimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasound information and using a preset machine algorithm to process the electromyographic signals and muscle ultrasound information.

[0012] If the user selects the automatic recognition mode, the system will automatically compare the metrics of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and output the motion intention recognition result obtained by the method with the best metrics.

[0013] This application also discloses a motion intention recognition system based on electromyography and muscle ultrasound, the system comprising:

[0014] The index generation module is used to acquire the indexes of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and display the indexes on the system interface; among them, the indexes include the motion recognition accuracy.

[0015] The output identification module is used to respond to the user's selection of the corresponding output mode based on the indicator, and performs the following operations:

[0016] If the user selects the electromyography (EMG) monomodal mode, the EMG monomodal motion intention recognition method is used to obtain the motion intention recognition result. The EMG monomodal motion intention recognition method obtains the motion intention recognition result by collecting EMG signals and processing the EMG signals using a preset machine algorithm.

[0017] If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intent recognition method is used to obtain the motion intent recognition result. The muscle ultrasound monomodal motion intent recognition method obtains the motion intent recognition result by collecting muscle ultrasound information and processing the muscle ultrasound information using a preset machine algorithm.

[0018] If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the motion intention recognition result. The bimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasound information and using a preset machine algorithm to process the electromyographic signals and muscle ultrasound information.

[0019] If the user selects the automatic recognition mode, the system will automatically compare the metrics of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and output the motion intention recognition result obtained by the method with the best metrics.

[0020] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements one or more of the methods described in this application.

[0021] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements one or more of the methods described in this application.

[0022] In this embodiment, users can select any output mode from the following based on the indicators displayed on the system interface: electromyography (EMG) monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode, offering flexible usage. The EMG monomodal mode identifies based on EMG signals, the muscle ultrasound monomodal mode identifies based on muscle ultrasound information, and the bimodal mode combines EMG signals and muscle ultrasound information for joint recognition. Muscle ultrasound information enables non-invasive detection of deep muscles, overcoming the low spatial resolution of EMG signals, which makes it difficult to detect small, deep muscle activities. This embodiment fully combines the advantages of high temporal resolution of EMG signals and high spatial resolution of muscle ultrasound information, enabling comprehensive detection of muscle electrophysiology and structural morphology information during limb movement, thus improving the accuracy of movement intention recognition. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator among the three modes based on the indicators of each movement intention recognition method, obtaining the most accurate movement intention recognition result among the three modes. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or robotic arms, improving the safety and reliability of prosthetic use and robotic arm control.

[0023] Additional aspects and advantages of the embodiments of this application will be set forth in the following description, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 is a system block diagram of the motion intention recognition method based on electromyography and muscle ultrasound provided in an embodiment of this application;

[0026] Figure 2 is a flowchart of the action recognition result acquisition process under various modes provided in the embodiments of this application;

[0027] Figure 3 is a schematic diagram of the motion intention recognition system based on electromyography and muscle ultrasound provided in an embodiment of this application;

[0028] Figure 4 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0030] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0031] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0032] The solutions provided in this application can be executed by any electronic device, such as a terminal device or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. Regarding the technical problems existing in the prior art, the motion intention recognition method and system based on electromyography and muscle ultrasound provided in this application aim to solve at least one of the technical problems in the prior art.

[0033] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0034] This application provides a possible implementation method and a flowchart of a motion intention recognition method based on electromyography and muscle ultrasound. This scheme can be executed by any electronic device, and optionally, it can be executed on a server or a terminal device.

[0035] Referring to Figure 1, the method may include the following steps:

[0036] The system acquires metrics for three different motion intention recognition methods: electromyography (EMG) monomodal motion intention recognition, muscle ultrasound monomodal motion intention recognition, and bimodal motion intention recognition, and displays these metrics on the system interface. The metrics include motion recognition accuracy.

[0037] In this embodiment, the metrics of the electromyography (EMG) monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method are displayed on the system interface as offline data. Taking the motion recognition accuracy rate as an example, after the system is started, the system interface displays the motion recognition accuracy rate of the EMG monomodal motion intention recognition method (which can be simply referred to as EMG signal motion recognition accuracy rate), the motion recognition accuracy rate of the muscle ultrasound monomodal motion intention recognition method (which can be simply referred to as muscle ultrasound motion recognition accuracy rate), and the motion recognition accuracy rate of the bimodal motion intention recognition method (which can be simply referred to as EMG-ultrasound fusion motion recognition accuracy rate).

[0038] Each time a user uses the system, they can refer to the indicators on the system interface to select the output mode. The output modes include electromyography monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode.

[0039] In response to the user's selection of the corresponding output mode based on the metrics, perform the following operations:

[0040] If the user selects the electromyography (EMG) monomodal mode, the EMG monomodal motion intention recognition method is used to obtain the motion intention recognition result. This method acquires EMG signals and processes them using a preset machine algorithm to obtain the motion intention recognition result. Finally, the result is displayed on the system interface, as shown in Figure 1.

[0041] If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intention recognition method is used to obtain the motion intention recognition result. This method acquires muscle ultrasound information and processes it using a preset machine algorithm to obtain the motion intention recognition result. The result is then displayed on the system interface, as shown in Figure 1.

[0042] If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the action intention recognition result. This method acquires electromyographic (EMG) signals and muscle ultrasound information, and processes these signals and information using a pre-defined machine algorithm to obtain the action intention recognition result. The result is then displayed on the system interface, as shown in Figure 1, representing the fused EMG / ultrasound motion recognition result. In this mode, the accuracy of motion intention recognition is improved by combining the electrophysiological information of muscle movement (i.e., EMG signals) and muscle structural morphology information (i.e., muscle ultrasound information).

[0043] If the user selects the automatic recognition mode, the system automatically compares the metrics of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and outputs the motion intention recognition result obtained by the method with the best metric. As shown in Figure 1, the motion intention recognition accuracy of the above three methods can be compared, and the motion recognition result output by the method with the highest accuracy can be displayed.

[0044] Optionally, the preset machine learning algorithm can be Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), Convolutional Neural Networks (CNN), or Long Short-Term Memory (LSTM).

[0045] In this embodiment, users can select any output mode from the following based on the indicators displayed on the system interface: electromyography (EMG) monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode, offering flexible usage. The EMG monomodal mode identifies based on EMG signals, the muscle ultrasound monomodal mode identifies based on muscle ultrasound information, and the bimodal mode combines EMG signals and muscle ultrasound information for joint recognition. Muscle ultrasound information enables non-invasive detection of deep muscles, overcoming the low spatial resolution of EMG signals, which makes it difficult to detect small, deep muscle activities. This embodiment fully combines the advantages of high temporal resolution of EMG signals and high spatial resolution of muscle ultrasound information, enabling comprehensive detection of muscle electrophysiology and structural morphology information during limb movement, thus improving the accuracy of movement intention recognition. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator among the three modes based on the indicators of each movement intention recognition method, obtaining the most accurate movement intention recognition result among the three modes. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or robotic arms, improving the safety and reliability of prosthetic use and robotic arm control.

[0046] In an optional embodiment, the metrics also include the amplitude and signal-to-noise ratio of the electromyographic signal and muscle ultrasound information.

[0047] In this embodiment, the system has electromyography (EMG) and muscle ultrasound information quality assessment functions. The system can display the amplitude and signal-to-noise ratio of EMG signals and muscle ultrasound information, as well as the motion recognition accuracy of three methods: EMG single-modal motion intention recognition method, muscle ultrasound single-modal motion intention recognition method, and dual-modal motion intention recognition method. This can help users select the corresponding output mode based on the above preparation.

[0048] In an optional embodiment, the metrics of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method are acquired respectively, and the metrics are displayed on the system interface, including:

[0049] The electromyography monomodal motion intention recognition method is used to process one or more electromyography signals to obtain multiple first motion intention recognition results, and the motion recognition accuracy of the electromyography monomodal motion intention recognition method is calculated based on the multiple first motion intention recognition results.

[0050] Multiple muscle ultrasound information was processed using a muscle ultrasound single-modal motion intention recognition method to obtain multiple second motion intention recognition results, and the motion recognition accuracy of the muscle ultrasound single-modal motion intention recognition method was calculated based on the multiple second motion intention recognition results.

[0051] A dual-modal motion intention recognition method was used to process multiple sets of time-synchronized electromyographic signals and muscle ultrasound information to obtain multiple third-motion intention recognition results. The motion recognition accuracy of the dual-modal motion intention recognition method was then calculated based on the multiple third-motion intention recognition results.

[0052] The motion recognition accuracy rates of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method are displayed on the system interface.

[0053] In an optional embodiment, a single-modal electromyography (EMG) motion intent recognition method is used to process one or more segments of EMG signals to obtain multiple first motion intent recognition results. The motion recognition accuracy of the EMG single-modal motion intent recognition method is then calculated based on these multiple first motion intent recognition results, including:

[0054] Use electromyography (EMG) devices to acquire one or more segments of EMG signals from the skin surface when a user makes different hand movements;

[0055] The features of the electromyographic signals are extracted using a preset machine algorithm, and the features of the electromyographic signals are identified to obtain multiple first action intention recognition results.

[0056] The percentage of correct results among multiple first-motor intention recognition results is calculated to obtain the motion recognition accuracy of the electromyography single-modal motion intention recognition method.

[0057] In this embodiment, an electromyography (EMG) device is used to collect multiple segments of EMG signals. For each sampling point, the user's action at that sampling point is used as a label to generate a sample. A dataset of EMG signals is generated based on multiple samples. The dataset of EMG signals is input into a preset machine algorithm, which outputs the first action intent recognition result corresponding to each sample, as well as the action recognition accuracy of the EMG single-modal motion intent recognition method. Specifically, the first action intent recognition result of each sample is compared with the label to determine whether it is correct, and the number of correct samples is counted. Finally, the action recognition accuracy of the EMG single-modal motion intent recognition method is obtained by dividing the number of correct samples by the total number of samples.

[0058] In an optional embodiment, a muscle ultrasound single-modal motion intent recognition method is used to process multiple muscle ultrasound information to obtain multiple second motion intent recognition results. The motion recognition accuracy of the muscle ultrasound single-modal motion intent recognition method is then calculated based on these multiple second motion intent recognition results, including:

[0059] The ultrasound device was used to obtain ultrasound information of multiple muscles on the skin surface when the user made different hand movements;

[0060] The features of each muscle ultrasound information are extracted using a preset machine algorithm, and the features of each muscle ultrasound information are identified to obtain multiple second action intention recognition results.

[0061] The percentage of correct results among multiple second action intent recognition results is calculated to obtain the action recognition accuracy of the muscle ultrasound single-modal motion intent recognition method.

[0062] In this embodiment, an ultrasound device is used to collect multiple muscle ultrasound data. For each muscle ultrasound data point, the user's action under that muscle ultrasound data is used as a label to generate a sample. A dataset of muscle ultrasound data is generated based on multiple samples. The dataset of muscle ultrasound data is input into a preset machine algorithm, which outputs the second action intent recognition result corresponding to each sample, as well as the action recognition accuracy of the muscle ultrasound single-modal motion intent recognition method. Specifically, the second action intent recognition result of each sample is compared with the label to determine whether it is correct, and the number of correct samples is counted. Finally, the action recognition accuracy of the muscle ultrasound single-modal motion intent recognition method is obtained by dividing the number of correct samples by the total number of samples.

[0063] In an optional embodiment, a bimodal motion intent recognition method is used to process multiple sets of time-synchronized electromyographic signals and muscle ultrasound information to obtain multiple third motion intent recognition results. The motion recognition accuracy of the bimodal motion intent recognition method is then calculated based on these multiple third motion intent recognition results, including:

[0064] The device uses electromyography (EMG) and ultrasound to simultaneously acquire multiple sets of EMG signals and muscle ultrasound information on the skin surface when the user makes different hand movements.

[0065] The preset machine algorithm is used to fuse the electromyographic signals and muscle ultrasound information in each group, and the fused information in each group is identified to obtain multiple third action intention recognition results.

[0066] The percentage of correct results among multiple third-mode motion intent recognition results is calculated to obtain the motion recognition accuracy of the bimodal motion intent recognition method.

[0067] In this embodiment, electromyography (EMG) and ultrasound devices are used to simultaneously acquire EMG signals and muscle ultrasound information when the user performs different movements. Each synchronously acquired EMG signal and muscle ultrasound information is considered a group. For each group of EMG signals and muscle ultrasound information, the user's movement is used as a label to generate a sample. An EMG and ultrasound fusion dataset is generated based on multiple samples. The EMG and ultrasound fusion dataset is input into a preset machine algorithm, which outputs the third-mode motion intent recognition result for each sample, as well as the motion recognition accuracy of the bimodal motion intent recognition method. Specifically, the third-mode motion intent recognition result for each sample is compared with the label to determine its correctness, and the number of correct samples is counted. Finally, the motion recognition accuracy of the bimodal motion intent recognition method is obtained by dividing the number of correct samples by the total number of samples.

[0068] In summary, users can choose from four output modes based on their needs: electromyography (EMG) monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode. As shown in Figure 2, EMG signals are acquired using an EMG device, and muscle ultrasound information is acquired using an ultrasound device. In EMG monomodal mode, feature extraction and motion prediction are performed on the EMG signals, and then the motion result is output. In muscle ultrasound monomodal mode, feature extraction and motion prediction are performed on the muscle ultrasound information, and then the motion result is output. In bimodal mode, feature extraction and fusion are performed on the EMG and muscle ultrasound information, and motion prediction is performed based on the fused information, and then the motion result is output. In automatic recognition mode, the motion result based on the method with the highest pre-obtained motion recognition accuracy is output.

[0069] In an optional embodiment, the fusion of electromyographic signals and muscle ultrasound information in each group can be performed in the following ways: signal layer fusion, feature layer fusion, and decision layer fusion.

[0070] Among them, signal layer fusion refers to the statistical analysis of each sampling point of electromyography signal and each pixel value of muscle ultrasound information to achieve the fusion of electromyography signal and muscle ultrasound information.

[0071] Feature layer fusion refers to extracting features from electromyography (EMG) signals and muscle ultrasound information separately, and then fusing the features of EMG signals and muscle ultrasound information.

[0072] Decision-level fusion refers to obtaining electromyographic motor intention recognition results and muscle ultrasound motor intention recognition results based on electromyographic signals and muscle ultrasound information, respectively, and then making joint judgments and processing based on the electromyographic motor intention recognition results and muscle ultrasound motor intention recognition results.

[0073] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application also provide a motion intention recognition system based on electromyography and muscle ultrasound, as shown in Figure 3. The system includes:

[0074] The index generation module 301 is used to acquire the indexes of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and display the indexes on the system interface; wherein, the indexes include the motion recognition accuracy.

[0075] The identification output module 302 is used to respond to the user's selection of the corresponding output mode based on the indicator and perform the following operations:

[0076] If the user selects the electromyography monomodal mode, the electromyography monomodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyography monomodal motion intention recognition method obtains the motion intention recognition result by collecting electromyography signals and processing the electromyography signals using a preset machine algorithm.

[0077] If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound monomodal motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and processing the muscle ultrasound information using a preset machine algorithm.

[0078] If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the motion intention recognition result; the bimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasound information, and processing the electromyographic signals and muscle ultrasound information using a preset machine algorithm.

[0079] If the user selects the automatic recognition mode, the system will automatically compare the indicators of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and output the motion intention recognition result obtained by the method corresponding to the optimal indicator.

[0080] In this embodiment, users can select any output mode from the following based on the indicators displayed on the system interface: electromyography (EMG) monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode, offering flexible usage. The EMG monomodal mode identifies based on EMG signals, the muscle ultrasound monomodal mode identifies based on muscle ultrasound information, and the bimodal mode combines EMG signals and muscle ultrasound information for joint recognition. Muscle ultrasound information enables non-invasive detection of deep muscles, overcoming the low spatial resolution of EMG signals, which makes it difficult to detect small, deep muscle activities. This embodiment fully combines the advantages of high temporal resolution of EMG signals and high spatial resolution of muscle ultrasound information, enabling comprehensive detection of muscle electrophysiology and structural morphology information during limb movement, thus improving the accuracy of movement intention recognition. The automatic recognition mode automatically selects the mode corresponding to the optimal indicator among the three modes based on the indicators of each movement intention recognition method, obtaining the most accurate movement intention recognition result among the three modes. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or robotic arms, improving the safety and reliability of prosthetic use and robotic arm control.

[0081] The motion intention recognition system based on electromyography and muscle ultrasound provided in this application embodiment can realize the various processes implemented in the method embodiments of Figures 1 and 2. To avoid repetition, these will not be described again here.

[0082] The motion intention recognition system based on electromyography and muscle ultrasound in this application embodiment can execute the motion intention recognition method based on electromyography and muscle ultrasound provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the motion intention recognition system based on electromyography and muscle ultrasound in each embodiment of this application correspond to the steps in the motion intention recognition method based on electromyography and muscle ultrasound in each embodiment of this application. For detailed functional descriptions of each module of the motion intention recognition system based on electromyography and muscle ultrasound, please refer to the descriptions in the corresponding motion intention recognition methods based on electromyography and muscle ultrasound shown above. They will not be repeated here.

[0083] Based on the same principles as the methods shown in the embodiments of this application, this application also provides an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the motion intention recognition method based on electromyography and muscle ultrasound shown in any optional embodiment of this application by calling the computer program. Compared with the prior art, the motion intention recognition method based on electromyography and muscle ultrasound provided by this application allows users to select any output mode from electromyography monomodal mode, muscle ultrasound monomodal mode, bimodal mode, and automatic recognition mode based on the indicators displayed on the system interface, making the method flexible. The electromyography monomodal mode identifies based on electromyography signals, the muscle ultrasound monomodal mode identifies based on muscle ultrasound information, and the bimodal mode identifies based on a combination of electromyography signals and muscle ultrasound information. Among these, muscle ultrasound information can achieve non-invasive detection of deep muscles, compensating for the problem of low spatial resolution of electromyography signals, making it difficult to detect small and deep muscle activities from electromyography signals. This embodiment of the application fully combines the advantages of high temporal resolution of electromyography (EMG) signals and high spatial resolution of muscle ultrasound information, enabling comprehensive detection of muscle electrophysiological and structural morphological information during limb movement and improving the accuracy of movement intention recognition. The automatic recognition mode automatically selects the mode corresponding to the optimal index among the three modes based on the indicators of each movement intention recognition method, obtaining the most accurate movement intention recognition result among the three modes. The above method can be used to enhance the human-computer interaction performance of prosthetic hands or robotic arms, improving the safety and reliability of prosthetic use and robotic arm control.

[0084] In an optional embodiment, an electronic device is also provided, as shown in FIG4. The electronic device 400 shown in FIG4 can be a server, including a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the electronic device 400 may further include a transceiver 404. It should be noted that in practical applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of this application.

[0085] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0086] Bus 402 may include a pathway for transmitting information between the aforementioned components. Bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 402 may be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 4, but this does not indicate that there is only one bus or one type of bus.

[0087] The memory 403 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0088] The memory 403 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the foregoing method embodiments.

[0089] The electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic device shown in Figure 4 is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0090] The server provided in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0091] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0092] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0093] It should be noted that the computer-readable storage medium described above in this application can also be a computer-readable signal medium or a combination of computer-readable storage media and computer-readable storage media. Computer-readable storage media can be, for example,—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0094] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0095] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0096] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the motion intention recognition method and system based on electromyography and muscle ultrasound provided in the various optional implementations described above.

[0097] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the identification output module can also be described as "an identification output module that performs the following operations in response to a user's selection of a corresponding output mode based on the specified indicators."

[0100] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for recognizing movement intention based on electromyography and muscle ultrasound, characterized in that, The method includes: The system acquires the metrics for electromyography-based single-modal motion intention recognition, muscle ultrasound-based single-modal motion intention recognition, and bimodal motion intention recognition, and displays these metrics on the system interface; wherein, the metrics include motion recognition accuracy. In response to the user's selection of the output mode based on the aforementioned metrics, perform the following operations: If the user selects the electromyography monomodal mode, the electromyography monomodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyography monomodal motion intention recognition method obtains the motion intention recognition result by collecting electromyography signals and processing the electromyography signals using a preset machine algorithm. If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound monomodal motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and processing the muscle ultrasound information using a preset machine algorithm. If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the motion intention recognition result; the bimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasound information, and processing the electromyographic signals and muscle ultrasound information using a preset machine algorithm. If the user selects the automatic recognition mode, the system will automatically compare the indicators of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and output the motion intention recognition result obtained by the method corresponding to the optimal indicator.

2. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 1, characterized in that, The process of acquiring indicators for the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and displaying these indicators on the system interface, includes: The electromyography monomodal motion intention recognition method is used to process one or more segments of electromyography signals to obtain multiple first motion intention recognition results, and the motion recognition accuracy of the electromyography monomodal motion intention recognition method is calculated based on the multiple first motion intention recognition results. The muscle ultrasound single-modal motion intention recognition method is used to process multiple muscle ultrasound information to obtain multiple second motion intention recognition results, and the motion recognition accuracy of the muscle ultrasound single-modal motion intention recognition method is calculated based on the multiple second motion intention recognition results. The bimodal motion intent recognition method is used to process multiple sets of time-synchronized electromyographic signals and muscle ultrasound information to obtain multiple third-motion intent recognition results. Based on these multiple third-motion intent recognition results, a bimodal motion intent recognition value is calculated. The accuracy of action recognition using other methods; The motion recognition accuracy rates of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method are displayed on the system interface.

3. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that, The process of using the electromyography monomodal motion intention recognition method to process one or more segments of electromyography signals to obtain multiple first motion intention recognition results, and calculating the motion recognition accuracy of the electromyography monomodal motion intention recognition method based on the multiple first motion intention recognition results, includes: Use electromyography (EMG) devices to acquire one or more segments of EMG signals from the skin surface when a user makes different hand movements; The features of the electromyographic signals are extracted using a preset machine algorithm, and the features of the electromyographic signals are identified to obtain multiple first action intention recognition results. The percentage of correct results among multiple first action intent recognition results is calculated to obtain the action recognition accuracy of the electromyographic single-modal motion intent recognition method.

4. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that, The method of processing multiple muscle ultrasound information using the muscle ultrasound single-modal motion intention recognition method to obtain multiple second motion intention recognition results, and calculating the motion recognition accuracy of the muscle ultrasound single-modal motion intention recognition method based on the multiple second motion intention recognition results, includes: The ultrasound device was used to obtain ultrasound information of multiple muscles on the skin surface when the user made different hand movements; The features of each muscle ultrasound information are extracted using a preset machine algorithm, and the features of each muscle ultrasound information are identified to obtain multiple second action intention recognition results. The percentage of correct results among multiple second action intent recognition results is calculated to obtain the action recognition accuracy of the muscle ultrasound single-modal motion intent recognition method.

5. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 2, characterized in that, The method of using the dual-modal motion intention recognition method to process multiple sets of time-synchronized electromyographic signals and muscle ultrasound information to obtain multiple third-motion intention recognition results, and calculating the motion recognition accuracy of the dual-modal motion intention recognition method based on the multiple third-motion intention recognition results, includes: The device uses electromyography (EMG) and ultrasound to simultaneously acquire multiple sets of EMG signals and muscle ultrasound information on the skin surface when the user makes different hand movements. The preset machine algorithm is used to fuse the electromyographic signals and muscle ultrasound information in each group, and the fused information in each group is identified to obtain multiple third action intention recognition results. The percentage of correct results among multiple third-mode motion intent recognition results is calculated to obtain the motion recognition accuracy of the bimodal motion intent recognition method.

6. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 5, characterized in that, The methods for fusing electromyographic signals and muscle ultrasound information in each group include signal layer fusion, feature layer fusion, and decision layer fusion. The signal layer fusion refers to the statistical analysis of each sampling point of the electromyographic signal and each pixel value of the muscle ultrasound information to achieve the fusion of the electromyographic signal and the muscle ultrasound information. The feature layer fusion refers to extracting features from the electromyography signal and the muscle ultrasound information respectively, and then fusing the features of the electromyography signal and the muscle ultrasound information. The decision-level fusion refers to obtaining electromyographic movement intention recognition results and muscle ultrasound movement intention recognition results based on the electromyographic signals and muscle ultrasound information, respectively, and then performing joint judgment and processing based on the electromyographic movement intention recognition results and the muscle ultrasound movement intention recognition results.

7. The method for recognizing movement intention based on electromyography and muscle ultrasound according to claim 1, characterized in that, The indicators also include the amplitude and signal-to-noise ratio of electromyographic signals and muscle ultrasound information.

8. A motion intention recognition system based on electromyography and muscle ultrasound, characterized in that, The system includes: The index generation module is used to acquire indices for the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and display the indices on the system interface; wherein, the indices include the motion recognition accuracy. The identification output module is used to respond to the user's selection of the corresponding output mode based on the aforementioned indicators, and performs the following operations: If the user selects the electromyography monomodal mode, the electromyography monomodal motion intention recognition method is used to obtain the motion intention recognition result; the electromyography monomodal motion intention recognition method obtains the motion intention recognition result by collecting electromyography signals and processing the electromyography signals using a preset machine algorithm. If the user selects the muscle ultrasound monomodal mode, the muscle ultrasound monomodal motion intention recognition method is used to obtain the motion intention recognition result; the muscle ultrasound monomodal motion intention recognition method obtains the motion intention recognition result by collecting muscle ultrasound information and processing the muscle ultrasound information using a preset machine algorithm. If the user selects the bimodal mode, the bimodal motion intention recognition method is used to obtain the motion intention recognition result; the bimodal motion intention recognition method obtains the motion intention recognition result by collecting electromyographic signals and muscle ultrasound information, and processing the electromyographic signals and muscle ultrasound information using a preset machine algorithm. If the user selects the automatic recognition mode, the system will automatically compare the indicators of the electromyography monomodal motion intention recognition method, the muscle ultrasound monomodal motion intention recognition method, and the bimodal motion intention recognition method, and output the motion intention recognition result obtained by the method corresponding to the optimal indicator.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images