Interface control system and method based on sonomyogram

Abstract

There is provided an interface control system based on sonomyogram (SMG), comprising: an SMG data acquisition device comprising a plurality of ultrasonic transducers (UT) arranged on a finger and / or a forearm and at least one data acquisition module; the SMG data processing device is used for processing the SMG data acquired by the SMG data acquisition device so as to obtain image data beneficial to identifying predetermined target muscles; the feature extraction and selection device is used for carrying out feature extraction on the form and / or the motion mode of the target muscle and selecting features beneficial to recognition of a predefined target gesture; the classification device is used for classifying the selected features based on the target gesture so as to recognize a correct gesture; and the interface control device is used for explaining the gesture recognized by the classification device into a control command of an interface on a media based on a local or online deployed control algorithm, so that human-computer interaction HM I is realized. The invention further provides a corresponding method.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine interface (HMI), and in particular to an interface control system and method based on sonomyography (SMG).

[0002] List of abbreviations

[0003] ANN Artificial Neural Network

[0004] AR Augmented Reality

[0005] BP Backpropagation

[0006] FCR flexor carpi radialis

[0007] FCU flexor carpi ulnaris

[0008] FDP flexor digitorum profundus

[0009] FDS flexor digitorum superficialis

[0010] FPL flexor pollicis longus

[0011] HMI Human Machine Interface

[0012] iEMG intramuscular electromyography

[0013] KNN K Nearest Neighbor

[0014] LDA Linear Discriminant Analysis

[0015] LR linear regression

[0016] MLP Multilayer Perceptron

[0017] Mixed Reality

[0018] RF

[0019] SDA Stacked Denoising Autoencoder

[0020] SMG Sonomyography

[0021] EMG electromyography

[0022] sEMG surface electromyography

[0023] SVM Support Vector Machine

[0024] TGC Time Gain Compensation

[0025] UT Ultrasonic Transducer

[0026] US ultrasound imaging Background Art

[0027] Biomedical signals are used to overcome the limitations of traditional HMI systems such as keyboards and mice. Human arm muscle movements can be converted into natural computer control commands. Currently, many studies are devoted to wearable devices to capture the muscle movements of human arms and fingers and recognize them as different gestures. Traditional gesture recognition methods include those based on electromyography (EMG). Since weak currents are generated when muscles contract, sensors attached to appropriate locations on the skin can measure the current of muscles on the surface of the body. EMG is a curve of current intensity changing over time. Intramuscular electromyography (iEMG) is an invasive detection method that can be used to capture muscle activity in various environments of deep muscles. However, it has disadvantages such as invasiveness, unstable electrode position, and small recording volume [2]. Surface electromyography (sEMG) captures electrical signals from muscles through sensors attached to the surface of the skin. sEMG is non-invasive, but there is a crosstalk problem with surrounding muscles and it cannot effectively detect the movement of deep muscles [2, 10]. EMG is easily affected by artifacts and background activity, making it difficult to recognize small finger movements and subtle gestures.

[0028] Muscle contraction causes muscle movement, resulting in changes in muscle structure and morphology[4]. When different fingers are flexed or extended, a unique set of myofascial septa are activated and mechanically deformed[7]. Compared to EMG, ultrasound imaging (US), which is used to sense mechanical deformation of functional myofascial septa, can distinguish deep continuous myofascial septa and obtain low signal-to-noise ratio and robust graded signals[6]. Compared to EMG, the probe size of US is much smaller

[10] . Ultrasound is a sound wave with a frequency above 20kHz. Ultrasound equipment can use the propagation and reflection of ultrasonic pulses to non-destructively detect and measure objects, thereby providing structural information of the object[1]. Ultrasonic pulses can be reflected at different depths in the body. The delay of the reflected wave will be proportional to the depth distance, and its intensity will indicate the type of tissue, because different tissues have characteristic acoustic impedance

[10] . US can detect, for example, muscle thickness, muscle fiber length, pennate angle of pennate muscles, and cross-sectional area[4]. US can not only provide static images of anatomical structures, but also provide real-time dynamic images of internal tissue movements related to physical and physiological activities[3]. The dynamic ultrasound images and dynamic signals obtained from ultrasound scanning of muscles are called SMGs. SMGs can better distinguish individual muscle activity and provide control signals proportional to muscle deformation. SMGs can spatially resolve individual muscles with submillimeter accuracy [7].

[0029] The present invention provides several novel SMG-based interface control systems. These systems calculate features of SMG acquired from gloves and / or forearm wristbands via US and deploy artificial neural networks (ANNs) to recognize different gestures, finger combinations, and fine finger movements, thereby controlling interfaces on different media. Summary of the Invention

[0030] The present invention discloses an interface control system based on sonomyogram (SMG), comprising: an SMG data acquisition device, the SMG data acquisition device comprising a plurality of ultrasonic transducers UT arranged on fingers and / or forearms and at least one data acquisition module; an SMG data processing device, the SMG data processing device processing the SMG data acquired by the SMG data acquisition device to obtain image data that is conducive to identifying predetermined target muscles; a feature extraction and selection device, the feature extraction and selection device extracts features of the morphology and / or movement pattern of the target muscles and selects features that are conducive to identifying predefined target gestures; a classification device, the classification device classifies the selected features based on the target gesture to identify the correct gesture; and an interface control device, the interface control device interprets the gestures recognized by the classification device as control commands of an interface on a medium based on a control algorithm deployed locally or online, thereby realizing human-computer interaction (HMI).

[0031] In some embodiments, the SMG data acquisition device is a wearable device, wherein the SMG data acquisition device is a wearable glove when arranged on a finger, the SMG data acquisition device is a wearable forearm band when arranged on a forearm, and the SMG data acquisition device is a combination of a glove and a forearm band when arranged on both the finger and the forearm, wherein the SMG data acquisition device is a left-hand glove, a right-hand glove, a left forearm band, a right forearm band, or any combination thereof.

[0032] In some embodiments, the SMG data acquisition device uses one or more of the following ultrasound modes: A-mode ultrasound, B-mode ultrasound, and M-mode ultrasound, and the acquired or reconstructed ultrasound dimensions include one or more of the following: one-dimensional, two-dimensional, or three-dimensional ultrasound.

[0033] In some embodiments, the SMG data processing device identifies the target muscle by segmenting the muscle aponeurosis in the SMG data and reconstructing the muscle bundles in the region.

[0034] In some embodiments, the SMG data processing device uses one or more of the following methods: time gain compensation (TGC), Gaussian filtering, Hilbert transform, logarithmic compression, and registration to generate the deformation field.

[0035] In some embodiments, when the SMG data acquisition device is a glove, the morphology and / or motion pattern of the target muscle is obtained from at least one or more of the following parameters: tissue displacement, length, width, area, angle, and grayscale gradient, as well as the magnitude and speed of changes in the above parameters over time.

[0036] In some embodiments, when the SMG data acquisition device is a forearm wristband, for the three-dimensional SMG reconstructed from multiple UTs in the forearm wristband, a three-dimensional model of the target muscle is generated by segmenting the muscle in each cross-sectional slice, and the extracted two-dimensional SMG features are spliced ​​together according to the spatial distribution of the muscle to obtain an SMG feature map, wherein both the three-dimensional model of the target muscle and the SMG feature map can be used as inputs to the AI ​​model.

[0037] In some embodiments, the feature extraction and selection device adopts one or more of the following methods: linear fitting, linear regression (LR), image edge detection, differential operator, high-pass filtering and discrete wavelet transform, wherein the classification device adopts one or more of the following methods: support vector machine SVM, back propagation BP neural network, linear discriminant analysis LDA, K nearest neighbor KNN algorithm, multi-layer perceptron MLP, stacked denoising autoencoder SDA, naive Bayes classifier and decision tree-based classifier.

[0038] In some embodiments, the control command is a custom sign language based on the target gesture, wherein the interface control device searches for the target sign language that matches the recognized gesture in a predefined sign language dictionary, and outputs a value corresponding to the target sign language when the match is successful. When the match is unsuccessful, the interface control device supports updating the sign language dictionary.

[0039] In some embodiments, the target gesture is a binary expression based on relaxation and bending of different fingers, wherein the finger states of the binary expression include: relaxation, bending thumb, bending index finger, bending middle finger, bending ring finger, bending little finger and making a fist.

[0040] In some embodiments, for the finger states expressed in binary, typing of all characters and function keys is achieved on ten keys by combining the bending states, bending times, and bending times of the ten fingers.

[0041] In some embodiments, for the finger state expressed in binary, the rows and columns on the custom key table are indexed respectively through the gestures of the left hand and the gestures of the right hand to find the corresponding keys at the intersection of the rows and columns, thereby realizing the typing of full characters and function keys on the custom key table.

[0042] In some embodiments, for the finger state of binary expression, one or more keyboard configurations are switched by making a fist with one hand while bending the fingers of the other hand. The keyboard configurations include one or more of the following: English keyboard, numeric keyboard, symbol keyboard, other language keyboards and user-defined shortcut keys.

[0043] In some embodiments, the interface control system also includes an interface display device, which presents an interface with a cursor, a finger pointer and / or a virtual keyboard to provide typing guidance, mouse clicks, cursor control and / or keyboard switching, and provides corresponding visual feedback when the user's gesture recognition is successful. The interface display device is one or more of the following: a display, a projection interface, augmented reality AR and mixed reality MR.

[0044] In some embodiments, the SMG data acquisition device includes a combination of gloves for both hands and a forearm wristband, which realizes mouse clicks, cursor movement and full keyboard input by collecting fine movements of the wrist and fingers, and provides real-time visual feedback on cursor movement and finger movement.

[0045] In some embodiments, the interface display device switches between the following three modes through gestures: virtual keyboard mode, touchpad mode, and mouse mode.

[0046] The present invention also discloses an interface control method based on sonomyogram (SMG), comprising: collecting SMG data through a plurality of ultrasonic transducers UT and at least one data acquisition module arranged on the fingers and / or forearms; processing the SMG data to obtain image data that is conducive to identifying predefined target muscles; extracting features of the morphology and / or movement pattern of the target muscles, and selecting features that are conducive to identifying predefined target gestures; classifying the selected features based on the target gestures to identify the correct gestures; and interpreting the identified gestures as control commands of the interface on the media based on a control algorithm deployed locally or online, thereby realizing human-computer interaction (HMI).

[0047] Other features and advantages will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The foregoing and further features of the present invention will become apparent from the following description of preferred embodiments provided by way of example only and taken in conjunction with the accompanying drawings, in which:

[0049] Figure 1 shows a schematic diagram of a forearm wristband according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a forearm target muscle according to an embodiment of the present invention is shown;

[0051] Figure 3 shows a schematic diagram of a glove according to an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of target muscles of the hand according to an embodiment of the present invention is shown;

[0053] Figure 5 A schematic diagram showing SMG data processing according to an embodiment of the present invention is shown;

[0054] Figure 6 A flowchart of gesture recognition based on SMG according to an embodiment of the present invention is shown;

[0055] Figure 7A An interface control system for performing sign language recognition using both hands according to an embodiment of the present invention is shown;

[0056] Figure 7B An interface control system for performing sign language recognition using a single hand according to an embodiment of the present invention is shown;

[0057] Figure 8 shows a schematic flow chart of sign language matching and insertion according to an embodiment of the present invention;

[0058] Figure 9 A schematic diagram showing finger states expressed in binary according to an embodiment of the present invention is shown;

[0059] Figure 10 A schematic diagram of ten-key typing steps according to an embodiment of the present invention is shown;

[0060] Figure 11 A schematic diagram of a method for switching keyboard configurations according to an embodiment of the present invention is shown;

[0061] Figure 12 A schematic diagram showing a ten-key typing guide interface on a display or projection media according to an embodiment of the present invention is shown;

[0062] Figure 13 A schematic diagram illustrating a ten-key typing guide interface on an AR or MR media according to an embodiment of the present invention is shown;

[0063] Figure 14 A schematic diagram of the steps of typing with a key table according to an embodiment of the present invention is shown;

[0064] Figure 15 A schematic diagram showing a key table typing guide interface on a display or projection medium according to an embodiment of the present invention is shown;

[0065] Figure 16 A schematic diagram illustrating a key table typing guide interface on an AR or MR media according to an embodiment of the present invention is shown;

[0066] Figure 17A and 17B A schematic diagram of an interface control system using a cursor and a virtual keyboard according to an embodiment of the present invention is shown;

[0067] Figure 18 A schematic diagram showing mode switching according to an embodiment of the present invention is shown;

[0068] Figure 19 A schematic diagram showing a cursor control on a display and a projection medium according to an embodiment of the present invention is shown;

[0069] Figure 20 FIG. 1 shows a schematic diagram of an SMG-based interface control system according to an embodiment of the present invention;

[0070] Figure 21 FIG. 1 is a schematic diagram showing an exemplary interface control system based on SMG according to an embodiment of the present invention;

[0071] Figure 22 A flow chart of an SMG-based interface control method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0072] As used herein, the terms "first," "second," and "third" are used interchangeably to distinguish one component from another, rather than to indicate the position or importance of a single component. The singular expressions "a," "an," and "the" also include the plural, unless the context clearly dictates otherwise. Terms such as "coupled," "fixed," and "connected to" refer to direct coupling, fixing, or connection, as well as indirect coupling, fixing, or connection through one or more intermediate components or features, unless the context clearly dictates otherwise. The terms "comprise," "include," "compose," "have," or any other variations thereof herein are intended to encompass non-exclusive inclusion. For example, a process, method, article, or device that includes a set of features is not necessarily limited to those features, but may include features not explicitly listed or other features inherent to such a process, method, article, or device. In addition, unless expressly stated to the contrary, "or" is inclusive rather than exclusive. For example, any of the following satisfies condition A or B: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).

[0073] Terms indicating approximation, such as "about," "approximately," "approximately," or "substantially," include values ​​within 10% greater or less than the stated value. When used in the context of an angle or direction, these terms include values ​​within 10 degrees greater or less than the stated angle or direction. For example, "approximately perpendicular" includes directions within 10 degrees of perpendicular in any direction (e.g., clockwise or counterclockwise).

[0074] It will be appreciated that, if any prior art publication is cited herein, such reference does not constitute an admission that the publication forms part of the common general knowledge in the art in any country.

[0075] The hand and fingers, as the most flexible parts of the forearm, can achieve complex interface control through different movement patterns and combinations. Therefore, to recognize different gestures or finger states, it is necessary to obtain rich motion information of the forearm muscle groups, and even the movement patterns of smaller muscles on the back of the hand to identify more detailed finger movements. Wearable devices are deployed at different locations on the forearm. Ultrasonic transducers (UTs) can be flexibly placed on the forearm to monitor muscle activity.

[0076] US is typically acquired using a UT probe. UT is a device equipped with one or more piezoelectric elements. These elements convert electrical pulses into mechanical pulses, thereby generating ultrasound waves. These elements can also convert the mechanical energy of the reflected pulses into electrical signals in the radio frequency (RF) range, using the envelope of the RF signal to construct an image. These phenomena are known as the inverse piezoelectric effect and the direct piezoelectric effect. Rapid switching of electronic components allows the generation and reading of ultrasound waves using the same probe. Modern ultrasound imaging devices use algorithms to form images and apply beamforming and harmonic imaging to improve resolution.

[0077] US has multiple modes, such as A-mode, B-mode, and M-mode. A-mode is also called amplitude mode, which means that the received echo signal is displayed in the form of amplitude. During scanning, a single UT can be fixed at a selected position. The echo signal of A-mode ultrasound is a one-dimensional signal, which has the advantages of flexible layout, small size, low cost, short computational delay, and small amount of data to be processed

[11] . B-mode is called brightness mode, which can display the information reflected from interfaces at different depths in grayscale form. During scanning, a probe composed of a UT linear array (such as 64, 128 UTs, etc.) is used to capture multiple A-mode scan lines. B-mode ultrasound reflects the state of tissue in a two-dimensional image. The signal robustness of B-mode ultrasound is better than that of A-mode ultrasound. M-mode is also called motion mode, which represents the movement of tissue over time by collecting continuous A-mode scan signals during scanning. When M-mode ultrasound is running, ultrasonic pulses are emitted rapidly and continuously. When the boundary of the object generating the reflection moves relative to the probe over time, M-mode ultrasound can determine its movement speed. Compared with B-mode ultrasound, M-mode ultrasound has a higher scanning frequency and can provide more detailed muscle state information than A-mode ultrasound. The envelope of the A-mode scan can be used to construct 2D B-mode and M-mode images. 2D B-mode or M-mode images can also be used to construct real-time 3D images based on algorithms

[12] . The present invention focuses on a system and method for interface control based on sonomyography. Therefore, as long as the US sensor device can be miniaturized, the present invention is not limited to the application of any one or combination of the above modes.

[0078] Figure 1A schematic diagram of a forearm wristband 100 according to an embodiment of the present invention is shown. Multiple UTs 110 are arranged side by side on the forearm wristband 100, generally along its longitudinal extension direction (i.e., along the length of the forearm), to generate a three-dimensional model of the arm muscles in that area. The forearm wristband is used to monitor simple flexion and extension movements of the wrist and fingers. Figure 1 1. A schematic arrangement of multiple UTs 110 and data acquisition modules 120 is shown. It should be understood that the locations and numbers of multiple UTs 110 and data acquisition modules 120 may vary as desired.

[0079] Figure 2 A schematic diagram of a forearm target muscle 200 according to an embodiment of the present invention is shown. Multiple UTs 110 can monitor the movement of the forearm target muscles 200. The forearm target muscles 200 may include the flexor carpi ulnaris (FCU) 210, the flexor carpi radialis (FCR) 220, the flexor digitorum superficialis (FDS) 230, the flexor digitorum profundus (FDP) 240, and the flexor pollicis longus (FPL) 250. The activity of the FCU and FCR can reflect the abduction and adduction of the wrist. The FDP and FDS involve flexion of the index, middle, ring, and little fingers at the metacarpophalangeal, proximal, and distal interphalangeal joints. The FPL can reflect flexion of the thumb. It should be understood that the forearm target muscles 200 described above are merely preferred examples of the present invention and are not intended to limit the types of muscles that can be detected. Certain gestures can involve the movement of a series of muscles, and discrete gestures can be classified by combining the changes in multiple muscles.

[0080] Figure 3 FIG. 3 is a schematic diagram of a glove 300 according to an embodiment of the present invention. The glove 300 can monitor fine movements of the fingers, including flexion and extension of the fingers and left and right tilt of the fingers. The glove 300 is arranged along the tendon 460 ( Figure 4 ) are arranged in the extending direction of . Figure 3 A schematic arrangement of five UTs 310 and data acquisition modules 320 is shown. It should be understood that the location and number of UTs 310 and data acquisition modules 320 may vary as needed.

[0081] Figure 4 A schematic diagram of a hand target muscle 400 according to an embodiment of the present invention is shown. The hand target muscles 400 include four lumbrical muscles 410, 420, 430, and 440 and the adductor pollicis 450. The lumbrical muscles 410-440 are involved in flexion of the metacarpophalangeal joints and extension of the interphalangeal joints. By measuring the first to fourth lumbrical muscles 410-440, fine motor movements of the little finger, ring finger, middle finger, and index finger can be monitored. The activity of the adductor pollicis 450 can reflect the adduction of the thumb.

[0082] The glove 300 and the forearm wristband 100 can be worn in different combinations on both arms to accommodate different application scenarios (e.g., single-handed and double-handed), usage modes (e.g., gesture control and keyboard and mouse control), or different types of users (e.g., healthy individuals and amputees). The data acquisition modules 120, 320 can be designed to be detachable. When the glove and wristband are used on the same side, one of the data acquisition modules can be removed to reduce weight. The data acquisition modules 120, 320 can transmit the collected image data in real time to a terminal device or the cloud for subsequent processing.

[0083] Figure 5 A schematic diagram of SMG data processing according to an embodiment of the present invention is shown. Muscle motion information conveyed by the SMG can be characterized in two ways. One way is to directly use the acquired SMG data as input to the AI ​​model. Another way is to pre-process the SMG data before using it as input to the AI ​​model. For example, the SMG data can first be segmented 510 to obtain a series of images of target muscles 520. Subsequently, SMG feature calculation 530 is performed for one or more target muscles 520 to obtain an SMG feature map 540. For two-dimensional SMG acquired through the glove 300, the target muscle 520 can be found by segmenting the muscle aponeurosis and reconstructing the muscle bundles in the area. Based on the complete muscle structure information, the morphology and / or motion pattern of the target muscle can be calculated, for example, using the following parameters: tissue displacement, length, width, area, angle, grayscale gradient, and / or the magnitude and / or velocity of the above parameters changing over time. For three-dimensional SMG reconstructed from multiple UTs 110 in the forearm wristband 100, a three-dimensional model of the target muscle 520 can be generated by segmenting the muscle in each cross-sectional slice. By splicing the extracted two-dimensional SMG features according to the spatial distribution of the muscle, an SMG feature map 540 can be obtained. Both the three-dimensional model of the target muscle 520 and the SMG feature map 540 can be used as inputs of the AI ​​model.

[0084] Figure 6The flowchart of gesture recognition based on SMG according to an embodiment of the present invention is shown. The gesture recognition based on SMG generally includes the following steps: SMG data collection 610, SMG data processing 620, feature extraction and selection 630 and classification 640. Based on images acquired in different modes, SMG data processing 620 can use various methods, such as time gain compensation (TGC), Gaussian filtering, Hilbert transform, logarithmic compression, and registration (e.g., based on optical flow field algorithms) to generate deformation fields. Feature extraction and selection 630 can use, for example, linear fitting, linear regression (LR), image edge detection, differential operators (e.g., Sobel, Prewitt, Laplacian, Roberts, and Canny operators), high-pass filtering, discrete wavelet transform, etc. Classification 640 can use a variety of classifiers, such as support vector machines (SVMs), back-propagation (BP) neural networks, linear discriminant analysis (LDA), K-nearest neighbor (KNN) algorithms, multi-layer perceptrons (MLPs), stacked denoising autoencoders (SDA), naive Bayes classifiers, and decision tree-based classifiers [5,8]. The performance of the classification system can be evaluated 650 based on various methods, such as F-scores. The aforementioned steps can be optimized and adjusted 660 based on the evaluation results. When using B-mode or M-mode ultrasound, in order to reduce the amount of computation, it is possible to verify whether the pre-defined target gesture can still guarantee the accuracy of full resolution when reducing the lateral spatial resolution (the arrangement direction of the UT array) [3,9]. This can be based on the fact that when the depth range of the tissue of interest is within the near field (Fresnel) region, the ultrasound beam size can be approximately the same as the UT size [3]. This also means that it is feasible to successfully recognize the target gesture with only a small number of UTs. When designing gestures, it should be considered that the recognition accuracy of discrete gestures is significantly higher than that of continuous gestures and has less computational complexity, especially to make full use of simplified gestures based on binary expressions.

[0085] Figure 7A An interface control system for sign language recognition using both hands according to an embodiment of the present invention is shown. Ultrasonic imaging is performed using a wristband 100 on both or one forearm, and SMG data of the forearm muscle groups is obtained. Simple muscle patterns and corresponding gestures are recognized in real time using AI algorithms and image processing. Sign language can be used as a direct gesture-based typing or command method. International sign languages ​​are too complex and diverse, so custom sign languages ​​are used to input words, characters, or commands.

[0086] Figure 7B This example illustrates an interface control system that uses single-handed sign language recognition according to an embodiment of the present invention. A single forearm wristband allows people with disabilities to easily control simple interfaces. They can use similar custom sign language to type or send commands. The difference is that the sign language dictionary only stores the sign language for a single hand.

[0087] Custom sign language implements interface control, such as typing, by retrieving the value corresponding to the user's current sign language from the user-defined sign language dictionary. Figure 8 A schematic flow chart of sign language matching and insertion according to an embodiment of the present invention is shown. When a user makes a custom sign language, the system attempts to match it in the sign language dictionary. If a corresponding custom sign language is found, the corresponding value is output. If the match is unsuccessful, the current sign language will be recognized as a new sign language. The user can use other inputs (such as through voice) to customize the value of the new sign language and insert it into the sign language dictionary. The updated dictionary will be used to retrain the artificial intelligence model to ensure that the newly defined sign language can be retrieved in a timely manner.

[0088] In addition to sign language, the relaxation and bending of different fingers can also be used as binary expressions. Figure 9 A schematic diagram of finger states expressed through binary expression according to an embodiment of the present invention is shown. If each hand is only allowed to bend a single finger, there are seven basic finger states, including: relaxed 910, bent thumb 920, bent index finger 930, bent middle finger 940, bent ring finger 950, bent pinky 960, and fist 970. These basic finger states can be combined between two hands to convey more information. Compared to complex sign language, SMG-based gesture recognition is easier to implement limited and fixed finger states. The way of combining fingers ensures the diversity of expression.

[0089] Figure 10 A schematic diagram illustrates the steps of ten-key typing according to an embodiment of the present invention. Ten-key typing is a form of finger combination typing. For anyone who has used ten-key input on a phone's numeric keypad, this typing method is easy to learn. Each finger bend has a defined letter, symbol group, or function. For the left hand, bending the pinky, ring finger, middle finger, index finger, and thumb respectively represents ",.?!", "ABC," "DEF," "GHI," and "JKL." For the right hand, bending the thumb, index finger, middle finger, ring finger, and pinky respectively represents "MNO," "PQRS," "TUV," "WXYZ," and "Caps." The "Caps" command is used for caps lock, consistent with traditional keyboards. Bending a finger different times selects the corresponding character sequence within the symbol group. For example, if a user wants to type "POLYU," they should first bend their right pinky once to lock capital letters, bend their right index finger once to type "P," bend their right thumb three times to type "O," and then bend their left thumb three times to type "L." Similarly, bending the other fingers sequentially selects "Y," "U," and "!".

[0090] Furthermore, the ten numbers "1234567890" correspond to the left pinky finger and the right pinky finger, respectively. Holding the fingers bent for more than one second will input the corresponding number. Bending the right pinky finger twice will backspace, and making a fist will input a space. It should be understood that the above input method is merely exemplary; other input methods can be defined for the ten keys by combining the bending state, bending times, and bending duration of the ten fingers.

[0091] Figure 11 A schematic diagram of a method for switching keyboard configurations according to an embodiment of the present invention is shown. In order to achieve more interaction possibilities, the keyboard configuration can be switched by making a fist with one hand. When the right hand makes a fist, the fingers of the left hand bend from left to right, representing keyboard configurations 1-5. When the left hand makes a fist, the fingers of the right hand bend from left to right, representing keyboard configurations 6-10. Keyboard configuration 1 is the default English keyboard. Keyboard configurations can include numeric keyboards, symbol keyboards, keyboards for other languages, and even user-defined shortcut keys. It should be understood that the above-mentioned definitions of keyboard configurations are only exemplary, and other keyboard configurations can also be defined.

[0092] Figure 12 A schematic diagram of a ten-key typing guidance interface on a display or projection medium according to an embodiment of the present invention is shown. Typing guidance interfaces are designed for different media, not only guiding users in inputting commands but also visualizing and providing feedback to users on their interactions, enhancing the sense of interaction. The media can include one or more of the following: display, projection, augmented reality (AR), and mixed reality (MR).

[0093] For traditional display and projection devices, ten virtual buttons will be displayed at the bottom of the interface, representing the fingers of both hands. When the muscle activity of finger flexion is recognized, the virtual button corresponding to the finger will be highlighted when pressed. The character group to be selected and the currently selected character will be displayed under the input cursor until another finger flexion is detected to complete the current character input. In the keyboard selection interface, the button on the side of the fist will be temporarily folded, and the keyboard options will be displayed on the other side.

[0094] Figure 13 A schematic diagram of a ten-key typing guide interface on an AR or MR media according to an embodiment of the present invention is shown. AR and MR devices are highly interactive, and the displayed content can interact directly with reality. Therefore, the AR or MR device typing guide can interact with gestures and hand movements to provide more powerful visual feedback. When the entity of the hand does not appear in the field of view of AR or MR, the typing guide is similar to the interface of traditional display and projection, displayed at the bottom of the field of view. When the entity of the hand appears in the field of view of AR or MR, the typing guide will be directly adsorbed above each finger. When the finger is bent, the finger area and the character group above it will be highlighted to remind the user that the gesture is successfully recognized.

[0095] Figure 14 A schematic diagram of the steps of key table typing according to an embodiment of the present invention is shown. Key table typing combines the bent fingers of both hands to represent characters. The fingers of the left hand are used to locate the row numbers in the key table. The six basic finger states other than making a fist are relaxed 910, bent thumb 920, bent index finger 930, bent middle finger 940, bent ring finger 950 and bent little finger 960, which are respectively located to the rows "AE", "FJ", "KO", "PT", "UY" and "Z-!" (combined with Figure 15 ). The fingers of the right hand are used to locate the column number in the key table. Therefore, combining the two hands can locate a specific character in the key table. Bending the right hand finger once represents lower case, and bending it twice represents upper case. For example, suppose the user wants to output "PolyU!". In this case, he / she should first bend his / her left middle finger once and right thumb twice to output "P", bend the left index finger once and the right little finger once to output lower case "o", and then bend the left index finger once and the right index finger once to output lower case "l". Similar to bending other fingers, the remaining "y", "U" and "!" are output in sequence. It should be understood that other keyboards indexed by rows and columns can also be defined. For the finger states expressed in binary, the rows and columns on the custom key table are indexed by the gestures of the left hand and the gestures of the right hand respectively to find the corresponding key values, thereby realizing the typing of full characters and function keys on the custom key table.

[0096] Additionally, making a fist with both hands simultaneously can input a space, while making a fist with one hand can enter keyboard configuration switching mode, similar to ten-key typing. Keyboard configuration 1 is the default English keyboard. Keyboard configurations can include a numeric keypad, a symbol keyboard, keyboards for other languages, and even user-defined shortcuts. It should be understood that the above keyboard configuration definitions are merely exemplary, and other keyboard configurations can also be defined.

[0097] Figure 15 A schematic diagram of a key table typing guide interface on a display or projection medium according to an embodiment of the present invention is shown. For traditional display and projection devices, ten virtual buttons are displayed at the bottom of the interface, representing the fingers of two hands. The button on the left indicates the character corresponding to the row, while the button on the right indicates the key table above it. The rows and columns in the key table have indicator bars that can be slid to visually locate the characters. When the muscle activity of the finger flexion is recognized, the virtual button corresponding to the finger will be highlighted when it is pressed. At the same time, the indicator bar will also move to the corresponding position, and the character where the two indicator bars of the row and column intersect is the user's output. In the keyboard configuration switching interface, the button on the side of the fist will be temporarily folded, and the keyboard options will be displayed on the other side, which is the same as ten-key typing.

[0098] Figure 16A schematic diagram of a key table typing guide interface on an AR or MR media according to an embodiment of the present invention is shown. The typing guide in the AR or MR device can interact with gestures and hand movements to provide more powerful visual feedback. When the physical hand does not appear in the AR or MR field of view, the typing guide is similar to the interface of traditional display and projection, displayed at the bottom of the field of view. When the physical hand appears in the AR or MR field of view, the typing guide will be directly adsorbed above each finger. When the finger is bent, the finger area and the character group above it will be highlighted to remind the user that the action has been successfully recognized.

[0099] The aforementioned sign language and finger-based interface control system can monitor wrist muscle activity or simple finger movements using only a forearm strap or glove. To enable a more natural keyboard and mouse interface control system using SMG-based acquisition devices without requiring additional learning, a combination of gloves and forearm straps can be used to achieve gesture recognition down to the finger level.

[0100] Figure 17A and 17B A schematic diagram of an interface control system using a cursor and a virtual keyboard according to an embodiment of the present invention is shown. The wearable ultrasound acquisition device can be combined in the following two ways, for example. One combination includes two gloves 300 on the left and right and a forearm wristband 100 on only the dominant hand, such as Figure 17A Another combination includes two left and right gloves 300 and two left and right forearm straps 100, as shown. Figure 17B As shown in Figure 3, the glove 300 and forearm wristband 100 are equipped with UTs 110 and 310 to capture ultrasound images of target muscles. The collected data is transmitted in real time to a terminal device or the cloud for subsequent processing via a data acquisition module 320 located inside the glove 300. The redundant data acquisition module 120 on the forearm wristband 100 is removed. The combination of the forearm wristband 100 and glove 300 can monitor fine movements of the wrist and fingers.

[0101] Figure 18 18 shows a schematic diagram of mode switching according to an embodiment of the present invention. A mode switching gesture, such as waving a hand 1810, can be designed to switch between virtual keyboard mode 1820, touchpad mode 1830 and mouse mode 1840. Figure 18As shown, touchpad mode 1830 and mouse mode 1840 can be used for cursor control, while virtual keyboard mode 1820 can be used for keyboard input. For cursor control, mouse mode 1840 primarily monitors wrist and finger movements by collecting SMG data from the forearm muscles and lumbrical muscles. Initially, the middle finger, ring finger, and pinky finger of the dominant hand are bent, while the index finger and thumb are relaxed. When motion is detected, an event classifier determines the current motion mode. An index finger up and down movement is a click event, four-finger flexion is a downward movement event, four-finger extension is an upward movement event, a wrist twist to the left is a leftward movement event, and a wrist twist to the right is a rightward movement event. In touchpad mode 1830, the initial state is the same as in mouse mode. The index finger's flexion and extension, as well as left and right tilting movements, are monitored in real time to control cursor movement. In virtual keyboard mode 1820, the abduction and adduction of the two thumbs, as well as the fine movements of the other eight fingers, are monitored to reflect the finger position on the virtual keyboard in real time for each media. Simultaneously, the event classifier identifies each finger press event.

[0102] Figure 19 A schematic diagram of controlling a cursor 1920 on a display and projection media according to an embodiment of the present invention is shown. For display and projection media, the cursor 1920 is controlled to move on the interface using touchpad mode 1830 or mouse mode 1840. A virtual keyboard is displayed at the bottom of the interface. Ten finger indicators 1910 appear on the keyboard to indicate the current position of the fingers in virtual keyboard mode 1820. To provide visual feedback to the user, the finger indicators 1910 perform real-time movement based on muscle activity. When the finger indicator 1910 hovers over a virtual button, the button will enlarge. When a click event on the finger is detected, the button below the finger indicator 1910 will be highlighted to indicate the user's successful input.

[0103] Figure 20 A schematic diagram of an SMG-based interface control system 2000 according to an embodiment of the present invention is shown. The interface control system 2000 includes an SMG data acquisition device 2010. The SMG data acquisition device 2010 comprises multiple ultrasonic transducers (UTs) arranged on the fingers and / or forearm and at least one data acquisition module for collecting SMG data. When arranged on a finger, the SMG data acquisition device 2010 is a wearable glove; when arranged on the forearm, it is a wearable forearm wristband. When arranged on both a finger and a forearm, the SMG data acquisition device 2010 is a combination of a glove and a forearm wristband. As previously described, the SMG data acquisition device 2010 can include a left-hand glove, a right-hand glove, a left forearm wristband, a right forearm wristband, or any combination thereof. The multiple UTs can utilize multiple discrete UTs or UT array probes. The data acquisition module 2010 can be deployed on one or more devices.

[0104] The interface control system 2000 also includes an SMG data processing device 2020, a feature extraction and selection device 2030, and a classification device 2040. Devices 2020-2040 mainly perform gesture recognition based on SMG. The SMG data processing device 2020 processes the SMG data collected by the SMG data acquisition device 2010 to obtain image data that is conducive to identifying predefined target muscles. The feature extraction and selection device 2030 extracts features from the morphology and / or movement pattern of the target muscles and selects features that are conducive to identifying predefined target gestures. The classification device 2040 classifies the selected features based on the target gesture to identify the correct gesture. One or more of the devices 2020-2040 can be deployed on one or more AI models. Figure 6 The description section lists the possible implementation methods such as algorithms and models involved, which will not be repeated here.

[0105] The interface control system 2000 also includes an interface control device 2050. The interface control device 2050 can interpret the recognized gestures as control commands of the interface based on a control algorithm deployed locally or online, thereby realizing the HMI. Preferably, the interface control system 2000 also includes an interface display device 2060. The interface display device 2060 can be a display as a medium, a projection interface, an interface with a cursor, a finger pointer and / or a virtual keyboard presented in AR or MR, which can provide intuitive visual feedback. Optionally, the interface control system 2000 also includes a tactile feedback device. Feedback on the user's actions can be provided tactilely. Tactile feedback can be a group of devices arranged on the data acquisition device 2010 that can generate microcurrents or tiny mechanical vibrations. For example, when a finger "taps" a specific keyboard correctly or incorrectly, a tiny vibration feedback is sent to the finger.

[0106] In the present invention, in order to use SMG to realize multi-mode keyboard and mouse command output, the classification device 2040 preferably uses an MLP-based event classifier to identify events such as "move", "click", "double-click" and "mode switch". When the current motion mode is classified as a "move" event, the SMG data will be input into the AI ​​model of different modes for motion command output. In mouse mode 1840, the model outputs the direction of cursor movement based on the four motion modes of the hand, and outputs the speed of cursor movement based on high-order SMG features. In touchpad mode 1830, the model outputs the position of the cursor relative to the initial cursor. In virtual keyboard mode 1820, the model outputs the position of the finger indicator 1910 relative to the initial position. Figure 21 FIG. 4 is a schematic diagram showing an exemplary SMG-based interface control system according to an embodiment of the present invention.

[0107] Figure 22FIG2 shows a flow chart of an exemplary interface control method 2200 based on SMG according to an embodiment of the present invention. Figure 20 The description part will not be repeated here.

[0108] The present invention can be used to control robots (prosthetic limbs, exoskeletons, surgical robots, etc.), electronic devices (smartphones, keyboards, mice, etc.), game characters, sign language interpreters, and more. It can also provide muscle activity information that can be integrated with musculoskeletal simulation software to provide useful information for rehabilitation and physical therapy. A novel system for detecting muscle activity during bodybuilding training improves training output efficiency. Furthermore, the present invention can help elderly people monitor their muscle activity to prevent falls.

[0109] The present invention is shown and described in detail above with reference to the accompanying drawings, but the above embodiments should be considered as illustrative rather than restrictive. The present invention also includes various combinations, modifications and variations of the exemplary embodiments without departing from the spirit and scope of the present invention.

[0110] References

[0111] The following is a list of references occasionally cited in this specification. The disclosures of each of these references are incorporated herein by reference in their entirety.

[0112] [1] Li, Jianmin, Kun Zhu, and Lizhi Pan. "Wrist and finger motionrecognition via M-mode ultrasound signal: A feasibility study." BiomedicalSignal Processing and Control 71(2022):103112.

[0113] [2] Zheng, Yang, et al. "Automatic detection of contracting muscle regions via the deformation field of transverse ultrasound images: afeasibility study." Annals of Biomedical Engineering 49.1 (2021): 354-366.

[0114] [3]Fernandes,Alexander James,Yuu Ono,and Eranga Ukwatta."Evaluationof finger flexion classification at reduced lateral spatial resolutions ofultrasound."IEEE Access 9(2021):24105-24118.

[0115] [4]Shi,Jun,et al."Recognition of finger flexion motion fromultrasound image:Afeasibility study."Ultrasound in medicine&biology 38.10(2012):1695-1704.

[0116] [5]Ortenzi,Valerio,et al."Ultrasound imaging for hand prosthesiscontrol:a comparative study of features and classification methods."2015IEEEInternational Conference on Rehabilitation Robotics(ICORR).IEEE,2015.

[0117] [6]Akhlaghi,Nima,et al."Real-time classification of hand motionsusing ultrasound imaging of forearm muscles."IEEE Transactions on BiomedicalEngineering 63.8(2015):1687-1698.

[0118] [7]Engdahl,Susannah,et al."A Novel Method for Achieving Dexterous,Proportional Prosthetic Control using Sonomyography."MEC20 Symposium.2020.

[0119] [8]Yang,Xingchen,et al."Simultaneous prediction of wrist / hand motionvia wearable ultrasound sensing."IEEE Transactions on Neural Systems andRehabilitation Engineering 28.4(2020):970-977.

[0120] [9]Akhlaghi,Nima,et al."Sparsity analysis of a sonomyographic muscle–computer interface."IEEE Transactions on Biomedical Engineering 67.3(2019):688-696.

[0121]

[10] McIntosh,Jess,et al."Echoflex:Hand gesture recognition usingultrasound imaging."Proceedings of the 2017 CHI Conference on Human Factorsin Computing Systems.2017.

[0122]

[11] Yang,Xingchen,et al."Towards wearable A-mode ultrasound sensingfor real-time finger motion recognition."IEEE Transactions on Neural Systemsand Rehabilitation Engineering 26.6(2018):1199-1208.

[0123]

[12] Huang,Qinghua,and Zhaozheng Zeng."A review on real-time 3Dultrasound imaging technology."BioMed research international 2017(2017).

Claims

1. An interface control system based on sonomyography (SMG), comprising: An SMG data acquisition device, the SMG data acquisition device comprising a plurality of ultrasonic transducers UT arranged on the fingers and / or forearms and at least one data acquisition module; an SMG data processing device, which processes the SMG data collected by the SMG data collection device to obtain image data that is conducive to identifying a predetermined target muscle; a feature extraction and selection device, which extracts features from the morphology and / or movement pattern of the target muscle and selects features that are useful for recognizing a predefined target gesture; a classification device, wherein the classification device classifies the selected features based on the target gesture to identify a correct gesture; An interface control device interprets the gestures recognized by the classification device as control commands of an interface on a medium based on a control algorithm deployed locally or online, thereby realizing human-machine interaction (HMI).

2. The system according to claim 1, wherein: The SMG data acquisition device is a wearable device, wherein the SMG data acquisition device is a wearable glove when arranged on a finger, the SMG data acquisition device is a wearable forearm wristband when arranged on a forearm, and the SMG data acquisition device is a combination of a glove and a forearm wristband when arranged on both the finger and the forearm, wherein the SMG data acquisition device is a left-hand glove, a right-hand glove, a left forearm wristband, a right forearm wristband, or any combination thereof.

3. The system according to claim 1, wherein: The SMG data acquisition device adopts one or more of the following ultrasound modes: A-mode ultrasound, B-mode ultrasound and M-mode ultrasound, and the ultrasound dimensions collected or reconstructed include one or more of the following: one-dimensional, two-dimensional or three-dimensional ultrasound.

4. The system according to claim 1, wherein: The SMG data processing device identifies the target muscle by segmenting the muscle aponeurosis in the SMG data and reconstructing the muscle bundles in the area.

5. The system according to claim 1, wherein The SMG data processing device adopts one or more of the following methods: time gain compensation TGC, Gaussian filtering, Hilbert transform, logarithmic compression and registration to generate a deformation field.

6. The system according to claim 2, wherein: When the SMG data acquisition device is a glove, the morphology and / or movement pattern of the target muscle is obtained from at least one or more of the following parameters: tissue displacement, length, width, area, angle and grayscale gradient, as well as the magnitude and speed of changes of the above parameters over time.

7. The system according to claim 2, wherein: When the SMG data acquisition device is a forearm wristband, for the three-dimensional SMG reconstructed by multiple UTs in the forearm wristband, the three-dimensional model of the target muscle is generated by segmenting the muscle in each cross-sectional slice, and the extracted two-dimensional SMG features are spliced ​​together according to the spatial distribution of the muscle to obtain an SMG feature map, wherein the three-dimensional model of the target muscle and the SMG feature map can both be used as inputs of the AI ​​model.

8. The system according to claim 1, wherein: The feature extraction and selection device adopts one or more of the following methods: linear fitting, linear regression (LR), image edge detection, differential operator, high-pass filtering and discrete wavelet transform, wherein the classification device adopts one or more of the following methods: support vector machine SVM, back propagation BP neural network, linear discriminant analysis LDA, K nearest neighbor KNN algorithm, multi-layer perceptron MLP, stacked denoising autoencoder SDA, naive Bayes classifier and decision tree-based classifier.

9. The system according to claim 1, wherein: The control command is a custom sign language based on the target gesture, wherein the interface control device searches for the target sign language that matches the recognized gesture in a predefined sign language dictionary, and outputs a value corresponding to the target sign language when the match is successful, and supports updating the sign language dictionary when the match is unsuccessful.

10. The system according to claim 1, wherein: The target gesture is a binary expression based on relaxation and bending of different fingers, wherein the finger states of the binary expression include: relaxation, bending of thumb, bending of index finger, bending of middle finger, bending of ring finger, bending of little finger and fist.

11. The system according to claim 10, wherein: For the finger states expressed in binary, typing of all characters and function keys is achieved on ten keys by combining the bending states, bending times and bending times of the ten fingers.

12. The system according to claim 10, wherein: For the finger state expressed in binary, the rows and columns on the custom key table are indexed respectively through the left hand gesture and the right hand gesture to find the corresponding key at the intersection of the row and column, thereby realizing the typing of full characters and function keys on the custom key table.

13. The system according to claim 10, wherein: For the finger state of the binary expression, one or more keyboard configurations are switched by making a fist with one hand and bending the fingers of the other hand. The keyboard configurations include one or more of the following: English keyboard, numeric keyboard, symbol keyboard, other language keyboards and user-defined shortcut keys.

14. The system according to claim 1, wherein: The interface control system also includes an interface display device, which presents an interface with a cursor, a finger pointer and / or a virtual keyboard to provide typing guidance, mouse clicks, cursor control and / or keyboard switching, and provides corresponding visual feedback when the user's gesture recognition is successful. The interface display device is one or more of the following: a display, a projection interface, augmented reality AR and mixed reality MR.

15. The system according to claim 14, wherein: The SMG data acquisition device includes a combination of gloves for both hands and a forearm wristband. It realizes mouse clicks, cursor movement and full keyboard input by collecting fine movements of the wrist and fingers, and provides real-time visual feedback on cursor movement and finger movement.

16. The system according to claim 15, wherein: The interface display device switches between the following three modes through gestures: virtual keyboard mode, touchpad mode and mouse mode.

17. A method for controlling an interface based on a sonomyogram (SMG), comprising: Collecting SMG data by means of a plurality of ultrasonic transducers UT and at least one data acquisition module arranged on the fingers and / or forearms; Processing the SMG data to obtain image data useful for identifying a predefined target muscle; Extracting features from the morphology and / or movement pattern of the target muscle and selecting features that are useful for recognizing a predefined target gesture; classifying the selected features based on the target gesture to identify a correct gesture; and The control algorithm based on local or online deployment interprets the recognized gestures as control commands of the interface on the media, thereby realizing human-computer interaction HMI.

Images