Signal identification system and method based on bimodal sensing

Abstract

The invention discloses a signal identification system and method based on bimodal sensing. The system comprises a man-machine interaction bracelet, a hardware module and a server, the man-machine interaction bracelet is used for making contact with the skin of an object and measuring a composite signal of the object, and the composite signal comprises an electromyographic signal and a muscle force signal which are fused; the hardware module is used for extracting the composite signal; the server is used for receiving the composite signal extracted by the hardware module, performing decoupling processing on the composite signal, and obtaining a target recognition result of the object according to a decoupling result; the human-computer interaction bracelet comprises hydrogel, a first electrode and a second electrode, the first electrode and the second electrode are fixed to the surface of the hydrogel, and the first electrode and the second electrode are used for making contact with the skin. The embodiment of the invention is beneficial to reducing the system cost and improving the recognition accuracy. The method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a signal recognition system and method based on dual-mode sensing. Background Art

[0002] Human-computer interaction has been widely used in fields such as smart home, medical treatment, education, entertainment and industrial automation. Accuracy is one of the most important parameters of human-computer interaction systems, especially in clinical applications such as active rehabilitation and intelligent prostheses. The human-computer interaction interface is a key component of the human-computer interaction system. It uses bioelectric signals, limb movements or muscle movements to achieve real-time synchronous communication between human and machine functions. It is the core component that determines the recognition accuracy of the human-computer interaction system. Large-area, body-scale epidermal systems for multiple sensing methods can provide powerful recording capabilities across multiple muscle groups, thereby accurately controlling the execution terminal. For large-area human-computer interaction interface systems, the assembly of sensors poses a major challenge.

[0003] In the related art, human-machine interfaces based on multi-channel electromyography have been widely used in fields such as VR games and rehabilitation medicine. The commercial HMI interface is assembled into a wristband by metal electrodes, elastic bands, circuits and shells. However, the hard metal electrodes and tight elastic bands may cause discomfort when worn, and long-term use may even cause dermatitis, and it is difficult to be compatible with arms of different sizes. The flexible human-machine interface can be mechanically bent and conform to the non-planar and dynamic surfaces of the human body, thereby improving the comfort of the wearer and the quality of the interactive signal obtained. In another embodiment, micromachining technology is used, and at the same time, by combining microporous silicone adhesives, a close bond between the electrode and the skin is achieved. However, the micromachining process requires expensive clean rooms and equipment, which will lead to high manufacturing costs of the human-machine interaction interface and is not conducive to large-scale application and promotion. The above problems all lead to low signal recognition accuracy. Summary of the invention

[0004] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0005] To this end, an object of the present invention is to provide a high-precision signal recognition system and method based on dual-modal sensing.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include the following aspects:

[0007] On the one hand, an embodiment of the present invention provides a signal recognition system based on dual-modal sensing, including: a human-computer interaction bracelet, a hardware module and a server; the human-computer interaction bracelet is used to contact the skin of an object and measure a composite signal of the object, the composite signal including a fused electromyographic signal and a muscle force signal; the hardware module is used to extract the composite signal; the server is used to receive the composite signal extracted by the hardware module, and decouple the composite signal, and obtain the target recognition result of the object according to the decoupling result; the human-computer interaction bracelet includes a hydrogel, a first electrode and a second electrode, the first electrode and the second electrode are fixed on the surface of the hydrogel, and the first electrode and the second electrode are used to contact the skin. The embodiment of the present application extracts the composite signal through the human-computer interaction bracelet made of hydrogel, which can be worn for a long time; at the same time, the signal decoupling is realized by the decoupling algorithm, so as to identify the intention of the object; the system composition is simple and easy to obtain, which alleviates the problem of high cost. The embodiment of the present application is conducive to reducing system cost and improving recognition accuracy.

[0008] In addition, the signal recognition system based on dual-modal sensing according to the above embodiment of the present invention may also have the following additional technical features:

[0009] Furthermore, in the dual-modal sensing-based signal recognition system of an embodiment of the present invention, the first electrode and the second electrode are fixed to the serpentine wire by soldering, and the serpentine wire is fixed to the surface of the hydrogel by electrochemical treatment.

[0010] Furthermore, in one embodiment of the present invention, the hardware module includes a microprocessor unit, an analog front-end unit and a filtering unit; the human-computer interaction bracelet is connected to the analog front-end unit through the filtering unit, and the analog front-end unit is connected to the server through the microprocessor unit.

[0011] Furthermore, in one embodiment of the present invention, the server is used to decouple the composite signal through a wavelet transform algorithm, extract characteristic parameters of the electromyographic signal and the muscle force signal, and identify the characteristic parameters through a trained multi-label neural network model to obtain the target recognition result.

[0012] Furthermore, in one embodiment of the present invention, the hydrogel is modified by a material rich in catechol groups and a polymer monomer.

[0013] On the other hand, an embodiment of the present invention provides a signal recognition method based on dual-modal sensing, which is applied to the above-mentioned signal recognition system based on dual-modal sensing. The method includes:

[0014] A composite signal of the object is measured by a human-computer interaction bracelet, wherein the composite signal includes a fused electromyographic signal and a muscle force signal;

[0015] The composite signal is decoupled, and a target recognition result of the object is obtained according to the decoupling result.

[0016] Furthermore, the signal recognition method based on dual-modal sensing of the embodiment of the present invention further includes:

[0017] If the average value of the electromyographic signal within the first time period is greater than or equal to the first signal threshold, it is determined that the current signal is a gesture signal; the gesture signal is a composite signal that can be used for gesture recognition;

[0018] Based on the current moment, the gesture signal within the second time duration is obtained as a composite signal.

[0019] Furthermore, the signal recognition method based on dual-modal sensing of the embodiment of the present invention further includes:

[0020] The composite signal is decoupled to extract characteristic parameters of the electromyographic and muscle force signals; the characteristic parameters include: peak-to-peak value of the muscle force signal, maximum value of the derivative of the muscle force signal, minimum value of the derivative of the muscle force signal, maximum value of the electromyographic signal, minimum value of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, peak-to-peak value of the electromyographic signal, absolute average value of the electromyographic signal, average value of the electromyographic signal, root mean square of the electromyographic signal, standard deviation of the electromyographic signal, variance of the electromyographic signal, and frequency threshold mean of the electromyographic signal.

[0021] Furthermore, the signal recognition method based on dual-modal sensing of the embodiment of the present invention further includes:

[0022] Inputting the characteristic parameters into a trained multi-label neural network model for recognition to obtain the target recognition result;

[0023] The multi-label neural network model is trained by the following steps:

[0024] Attaching the human-computer interaction bracelet to the skin of the subject to obtain gesture sample signals under different actions;

[0025] Extracting sample feature parameters of the gesture sample signal, inputting the sample feature parameters into the multi-label neural network model, and obtaining a sample recognition result;

[0026] According to the difference between the sample recognition result and the true result of the sample, a loss function is established, and the parameters of the multi-label neural network model are updated according to the loss function to obtain the trained multi-label neural network model.

[0027] Furthermore, in the signal recognition method based on dual-modal sensing of an embodiment of the present invention, the decoupling process of the composite signal includes:

[0028] Preprocessing the composite signal to obtain a base signal;

[0029] Determine a wavelet basis function and a decomposition layer number, and perform multi-layer decomposition on the base signal according to the wavelet basis function and the decomposition layer number to obtain a multi-layer signal; wherein each layer of decomposition corresponds to a coefficient and a frequency band;

[0030] Based on the coefficients, an inverse wavelet transform is performed to obtain decoupled electromyographic signals and muscle force signals; the electromyographic signals include signals corresponding to frequency bands D2, D3, D4, and D5, and the muscle force signals include signals corresponding to frequency bands D6, D7, D8, and A8.

[0031] On the other hand, an embodiment of the present invention provides a signal recognition device based on dual-modal sensing, comprising:

[0032] at least one processor;

[0033] at least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned signal recognition method based on dual-modal sensing.

[0035] On the other hand, an embodiment of the present invention provides a storage medium storing a program executable by a processor, wherein the program executable by the processor is used to implement the above-mentioned signal recognition method based on dual-modal sensing when executed by the processor.

[0036] The system provided by the embodiment of the present invention includes: a human-computer interaction bracelet, a hardware module and a server; the human-computer interaction bracelet is used to contact the skin of the object and measure the composite signal of the object, and the composite signal includes a fused electromyographic signal and a muscle force signal; the hardware module is used to extract the composite signal; the server is used to receive the composite signal extracted by the hardware module, and decouple the composite signal, and obtain the target recognition result of the object according to the decoupling result; the human-computer interaction bracelet includes a hydrogel, a first electrode and a second electrode, the first electrode and the second electrode are fixed on the surface of the hydrogel, and the first electrode and the second electrode are used to contact the skin. The embodiment of the present application extracts the composite signal through the human-computer interaction bracelet made of hydrogel, which can be worn for a long time; at the same time, the signal decoupling is realized by the decoupling algorithm, so as to identify the intention of the object; the system composition is simple and easy to obtain, which alleviates the problem of high cost. The embodiment of the present application is conducive to reducing system cost and improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic diagram of the structure of an embodiment of the dual-mode sensing provided by the present invention;

[0039] Figure 2 A schematic diagram of the sensing principle of an embodiment of the dual-mode sensing provided by the present invention;

[0040] Figure 3 A schematic diagram of the structure of an embodiment of the human-computer interaction bracelet provided by the present invention;

[0041] Figure 4 An electrical schematic diagram of an embodiment of a control module provided by the present invention;

[0042] Figure 5 An electrical schematic diagram of an embodiment of an analog front end unit provided by the present invention;

[0043] Figure 6 A schematic diagram of a flow chart of an embodiment of a multi-label neural network model provided by the present invention;

[0044] Figure 7 A schematic diagram of the decoupling effect of a composite signal according to an embodiment of the present invention;

[0045] FIG8( a ) is a schematic diagram showing a comparison of prediction effects between the system provided by the present invention and the system of the related art;

[0046] FIG8( b ) is a schematic diagram showing the comparison of the prediction probabilities corresponding to FIG8( a );

[0047] FIG9( a ) is a schematic diagram of wearing a human-computer interaction bracelet provided by the present invention;

[0048] FIG9( b ) is a schematic diagram of the measurement results corresponding to FIG9( a );

[0049] Fig.10 (a) is a schematic diagram of eight-channel signals corresponding to the fist gesture provided by the present invention;

[0050] Fig.10 (b) is a schematic diagram of eight-channel signals corresponding to the OK gesture provided by the present invention;

[0051] Fig.10 (c) is a schematic diagram of eight-channel signals corresponding to a gesture provided by the present invention;

[0052] Fig.10 (d) is a schematic diagram of eight-channel signals corresponding to three gestures provided by the present invention;

[0053] Fig.10 (e) is a schematic diagram of eight-channel signals corresponding to five gestures provided by the present invention;

[0054] Fig.10 (f) is a schematic diagram of eight-channel signals corresponding to the upward movement of the wrist provided by the present invention;

[0055] Fig.10 (g) is a schematic diagram of eight-channel signals corresponding to the downward movement of the wrist provided by the present invention;

[0056] Fig.10 (h) is a schematic diagram of eight-channel signals corresponding to the leftward movement of the wrist provided by the present invention;

[0057] Fig.10 (i) is a schematic diagram of eight-channel signals corresponding to the rightward movement of the wrist provided by the present invention;

[0058] FIG11( a ) is a schematic diagram of the recognition accuracy corresponding to an arm with an arm circumference of 27 cm provided by the present invention;

[0059] FIG11( b ) is a schematic diagram of the recognition accuracy corresponding to an arm with an arm circumference of 23 cm provided by the present invention;

[0060] FIG11( c ) is a schematic diagram of recognition accuracy corresponding to the commercial system provided by the present invention;

[0061] FIG. 12( a ) is a schematic diagram of the recognition accuracy rate for arm correspondence provided by the present invention;

[0062] FIG12( b ) is a schematic diagram of the recognition accuracy rate for the calf provided by the present invention;

[0063] FIG12( c ) is a schematic diagram of the recognition accuracy rate for thighs provided by the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0065] First, the relevant concepts involved in this application are explained:

[0066] Human-computer interaction bracelet: The human-computer interaction bracelet is a smart wearable device based on bioelectric signals, limb movements or muscle movements. It analyzes and controls movements by measuring muscle activity. It can monitor muscle activity and fatigue in real time, helping users optimize training effects and prevent sports injuries. In rehabilitation medicine, the human-computer interaction bracelet can be used to evaluate the patient's muscle function and guide rehabilitation training. At the same time, it can also be applied to human-computer interaction, controlling external devices such as wheelchairs and prostheses through muscle signals, improving the quality of life of people with disabilities, and has broad application potential and development prospects.

[0067] Electromyogram (EMG) is an electrical signal generated when muscles contract. When the brain sends a signal to the muscles to move, the motor neurons are activated, which generates action potentials in the muscle fibers. These potential changes combine to form the EMG signal. It can be detected by electrodes, which are generally placed on the surface of the skin (non-invasive testing) or by needle electrodes inserted into the muscles (invasive testing).

[0068] Force Myography (FMG) is mainly used to reflect changes in muscle strength. When a muscle exerts force, its physical properties such as shape and volume will change, and the FMG signal is a record of these changes. It is different from the EMG signal. EMG focuses on the electrical activity of the muscle, while FMG focuses on the morphological changes caused by muscle strength. This signal is widely used in the field of rehabilitation. For example, it helps monitor the muscle strength of prosthetic users so that the prosthesis can better cooperate with the user's movement intention; it can also provide data reference for athletes' strength training in sports training, and better evaluate the state of muscle strength.

[0069] Human-computer interaction has been widely used in fields such as smart home, medical treatment, education, entertainment and industrial automation. Accuracy is one of the most important parameters of human-computer interaction systems, especially in clinical applications such as active rehabilitation and intelligent prostheses. The human-computer interaction interface is a key component of the human-computer interaction system. It uses bioelectric signals, limb movements or muscle movements to achieve real-time synchronous communication between human and machine functions. It is the core component that determines the recognition accuracy of the human-computer interaction system. In order to further develop the human-computer interaction system and expand its application scenarios, it is crucial to improve the performance of the human-computer interaction interface and reduce the preparation cost.

[0070] In the development of human-computer interaction systems, non-invasive signals such as electromyographic signals, muscle force signals, and EEG signals are usually more attractive. Due to the imperfection of a single perception mode, single-modal human-computer interaction systems are still far from achieving stable control of execution terminals that match biological counterparts. Therefore, people use multi-channel or multi-modal methods to improve the robustness and accuracy of human-computer interaction interface systems.

[0071] Large-area, body-scale epidermal systems for multiple sensing modalities can provide powerful recording capabilities across multiple muscle groups, thereby precisely controlling the actuator terminal. For large-area human-machine interface systems, the assembly of sensors poses a major challenge. Recently, many human-machine interfaces based on multi-channel electromyography have appeared on the market and have been widely used in fields such as VR games and rehabilitation medicine. Commercial HMI interfaces are assembled into a wristband with metal electrodes, elastic bands, circuits, and housings. However, hard metal electrodes and tight elastic bands may cause discomfort when worn, and long-term use may even cause dermatitis, and are difficult to be compatible with arms of different sizes. Flexible human-machine interfaces can be mechanically bent and conform to non-planar and dynamic surfaces of the human body, thereby improving the comfort of the wearer and the quality of the interactive signals obtained. Using micromachining technology to construct conductive electrodes on stretchable substrates is a mainstream method for building flexible human-machine interfaces. An embodiment completes the construction of an 8-channel electromyography electrode pair on PDMS through techniques such as electron beam evaporation deposition, photolithography, and plasma etching. At the same time, by combining microporous silicone adhesives, a close bond between the electrode and the skin is achieved. However, the micromachining process requires expensive clean rooms and equipment, which will lead to high manufacturing costs of human-computer interaction interfaces and is not conducive to large-scale application and promotion. In addition, the method of combining elastomers with conductive electrodes is only applicable to single-modal EMG signals and cannot realize the construction of multimodal human-computer interfaces. In addition, previous studies have shown that multimodal human-computer interaction methods can effectively improve recognition accuracy. Therefore, it is urgent to explore new sensing mechanisms to reduce manufacturing costs and realize the construction of multi-channel and multimodal human-computer interaction interfaces.

[0072] A signal recognition system based on dual-modal sensing and an implementation method according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings. First, a signal recognition system based on dual-modal sensing according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0073] The signal recognition system based on dual-modal sensing provided by the present invention specifically includes:

[0074] Human-computer interaction bracelet, hardware module and server;

[0075] The human-computer interaction bracelet is used to contact the skin of the object to measure a composite signal of the object, wherein the composite signal includes a fused electromyographic signal and a muscle force signal;

[0076] The hardware module is used to extract the composite signal; the server is used to receive the composite signal extracted by the hardware module, perform decoupling processing on the composite signal, and obtain the target recognition result of the object according to the decoupling result;

[0077] The human-computer interaction bracelet includes a hydrogel, a first electrode and a second electrode. The first electrode and the second electrode are fixed to the surface of the hydrogel, and the first electrode and the second electrode are used to contact the skin.

[0078] In some embodiments, the first electrode and the second electrode in the present application are not adjacent to each other, the first electrode and the second electrode are used to measure the electromyographic signal, and the hydrogel is used to measure the muscle force signal. That is, the electromyographic signal in the embodiment of the present application is a signal obtained by the first electrode and the second electrode of the human-computer interaction bracelet. The target recognition result in the embodiment of the present application can be transmitted to the terminal through a wireless connection and displayed.

[0079] The server in the embodiment of the present application can process the decoupling result by means of artificial intelligence to obtain the intention information of the identified object (i.e., the target recognition result). Of course, those skilled in the art can also obtain the target recognition result by the decoupling result in other ways. The decoupling result in the embodiment of the present application is the myoelectric signal and muscle force signal after decoupling.

[0080] The present invention proposes an integrated dual-modal sensor, which uses a digital signal processing decoupling method to achieve a strategy of obtaining EMG and FMG dual-modal signals with a single device. Figure 1 As shown, it includes a zinc electrode 1 (i.e., the first electrode, and the present application does not limit the specific material of the electrode), a zinc electrode 2 (i.e., the second electrode), a hydrogel 3, a differential amplifier 4, and a wavelet change decoupling algorithm 5. The sensor is attached to the epidermal area corresponding to the muscle, and the movement of the muscle will be accompanied by EMG signals and FMG signals. The EMG signal will cause the differential signal between the zinc electrode 1 and the zinc electrode 2 to change, and the muscle force signal will cause the impedance of the hydrogel to change. Therefore, the differential circuit will output a composite signal including the electromyographic signal and the muscle force signal. Finally, the composite signal is decoupled by the wavelet change decoupling algorithm to obtain the electromyographic signal and the muscle force signal. Specifically, the wavelet transform method uses an 8th-order db06 wavelet to decouple the composite signal, wherein the frequency bands of the high-frequency signal (EMG) are D2, D3, D4, and D5, and the frequency bands of the low-frequency signal (FMG) are D6, D7, D8, and A8.

[0081] The integrated dual-mode sensing principle of the present invention is as follows Figure 2As shown. When the muscle moves, the muscle cells release electrical signals, causing the differential signals at two points on the epidermis to change. The movement of the muscle will also cause the epidermis to deform, causing the hydrogel attached to the epidermis to deform, which in turn causes the hydrogel impedance to change. The composite signal of the electromyographic signal and the muscle force signal can be extracted through the differential circuit. The electromyographic signal and the muscle force signal are in different frequency ranges. The electromyographic signal is distributed in 20-500Hz, and the muscle force signal is distributed in 0-10Hz. According to their frequency differences, they can be accurately separated by the wavelet decoupling algorithm.

[0082] Optionally, the signal recognition system based on dual-modal sensing in an embodiment of the present invention includes: the first electrode and the second electrode are fixed to the serpentine wire by soldering, and the serpentine wire is fixed to the surface of the hydrogel by electrochemical treatment.

[0083] In some embodiments, the sensor is attached to the epidermal area corresponding to the muscle, and the signals at both ends of the electrode are measured by a differential circuit, and a composite signal including an electromyographic signal and a muscle force signal is obtained. Based on the difference in the frequency range of the electromyographic signal and the muscle force signal, they can be separated from the composite signal by a wavelet transform decoupling algorithm.

[0084] Optionally, the signal recognition system based on dual-modal sensing in an embodiment of the present invention includes: the hardware module includes a microprocessor unit, an analog front-end unit and a filtering unit; the human-computer interaction bracelet is connected to the analog front-end unit through the filtering unit, and the analog front-end unit is connected to the server through the microprocessor unit.

[0085] In some possible implementations, the filtering unit in the embodiment of the present application is used to filter the composite signal, and the analog front-end unit is used to identify / extract the composite signal.

[0086] The schematic diagram of the multi-channel dual-mode human-computer interaction bracelet of the present invention is as follows: Figure 3 As shown. In this embodiment, it includes a hydrogel 6, a zinc electrode 7, a serpentine wire 8, an eight-channel analog front-end circuit 9, and a signal acquisition and transmission circuit 10. The zinc electrode 7 is fixed to the serpentine wire by soldering, and then the serpentine wire with the zinc electrode array is fixed to the surface of the hydrogel by electrochemical treatment to realize the construction of a four-channel electromyography-muscle force dual-modal sensor. Then two four-channel sensors are fixed to the eight-channel analog front-end circuit 9 by soldering. The eight-channel analog front-end circuit is connected to the signal acquisition and transmission circuit 10 by wires. The eight-channel analog front-end circuit is used to obtain the differential signal of the eight channels, and integrate and package the data and send it to the server via Bluetooth.

[0087] Optionally, in the signal recognition system based on dual-modal sensing in an embodiment of the present invention, the server is used to decouple the composite signal through a wavelet transform algorithm, extract characteristic parameters of the electromyographic signal and the muscle force signal, and identify the characteristic parameters through a trained multi-label neural network model to obtain the target recognition result.

[0088] The human-computer interaction algorithm of the present invention is implemented on the server. First, the server decouples the composite signal through the wavelet transform algorithm to obtain the electromyographic signal and the muscle force signal. Then, the characteristic parameters of the electromyographic signal and the muscle force signal are extracted and sent to the trained multi-label neural network algorithm (i.e., the multi-label neural network model), and the action intention is calculated and transmitted to the execution terminal by wireless means, such as Figure 6 The figure shows the flow chart of gesture recognition algorithm, which includes three parts: signal segmentation, signal feature extraction and neural network algorithm. Figure 7 The electronic skin is attached to the finger extensor muscles, and the electromyographic signals and muscle force signals of different actions (gesture 1, gesture 3, gesture 5, gesture OK, fist) are collected. Specifically, the pressure sensor obtains high-quality muscle deformation signals, and the self-adhesive conductive hydrogel provides a good potential collection interface for the collection system to obtain high-quality electromyographic signals. The electromyographic signals and muscle force signals collected from the finger extensor muscles of the arm for 5 different gestures are respectively shown. 10 gesture signals for each gesture are selected to form a 50*18 training sample, and the five different gestures are marked with 1-5 to form a 50*18 matrix as a training sample input to the multi-label neural network for training (training iteration 1000 times, learning rate 0.02, initial condition setting uses random assignment). According to the comparison results, the weights between each layer are compared to realize the training of the neural network and obtain the weight parameter file of each neuron. Then the test data containing 200 samples is input into the trained multi-label neural network algorithm to test its recognition accuracy. Further, we compare the electromyographic signals obtained with commercial electrodes. Figure 7 The multi-label neural network algorithm further demonstrated the accuracy of output prediction, with the recognition accuracy of integrated dual-modal sensing being 86.5% and the recognition accuracy of commercial electrodes being 64%.

[0089] Furthermore, in order to evaluate the gesture recognition performance of the bracelet, we obtained eight-channel differential signals corresponding to the above-mentioned nine actions, and obtained eight-channel electromyographic signals and muscle force signals by decoupling, as shown in Figure 9. Obviously, since the signals collected from different positions come from different muscle areas, there are significant differences between the electromyographic and muscle force signals of different channels for the same gesture. This difference can greatly improve the information entropy of the interaction signal. In addition, when comparing different gestures within a single channel, significant differences were observed between the electromyographic and muscle force signals due to the different muscle activation areas and movement amplitudes associated with each action. Therefore, by analyzing the eight-channel electromyographic-muscle force signals, high-precision recognition of action intentions can be achieved. In addition, signal repeatability for the same action is also crucial to maintaining this accuracy. To address this issue, we collected eight-channel electromyographic-muscle force signals in multiple consecutive trials involving nine different actions, such as Fig.10 The multi-channel composite signals of multiple gestures shown in Figure 11 are shown in Figure 11. For any given action, all eight channels show good repeatability in their respective myoelectric-force signals. In addition, to compare with the traditional human-computer interaction bracelet, we used 17 commercial electrodes to obtain eight-channel independent myoelectric signals, as shown in Figure 11(c). Obviously, this traditional method not only requires the use of elastic bands for support, but also involves a large number of sensors, making it inconvenient to wear. Commercial human-computer interaction bracelets integrate sensors and signal acquisition systems through innovative structural designs. However, this design usually uses metal electrodes, which have significant differences in modulus compared to the skin and lack self-adhesive properties. Therefore, these commercial bracelets usually rely on high-strength elastic bands to ensure precise contact between the metal electrodes and the skin. However, considering the huge differences in individual arm sizes, too tight fit can hinder blood circulation and cause tissue damage. On the contrary, excessive looseness may cause sensor displacement, resulting in signal artifacts, while changing the detected muscle area, thereby affecting the recognition accuracy during human-computer interaction. The multi-channel dual-modal human-computer interface bracelet provides excellent stretchability and self-adhesiveness, and can be easily applied to a variety of arm sizes. In addition, due to its multi-channel dual-mode information processing capabilities, the bracelet has a recognition accuracy of 95% for 9 different actions (as shown in Figure 11(a)), which is 18% higher than the single-channel EMG-muscle force dual-mode signal. In addition, the bracelet is designed to adapt to different arm sizes (23 cm) and the recognition accuracy reaches 94.2%, as shown in Figure 11(b). In addition, we evaluated the recognition accuracy of eight-channel EMG signals (as shown in Figure 11(a)). Fig.10 c), its recognition accuracy is only 89%. Therefore, the multi-channel dual-modal human-computer interaction bracelet not only surpasses the traditional multi-channel electromyographic model in recognition accuracy, but also provides a more convenient way of assembly and wearing. Its inherent stretchability further enhances compatibility with different arm sizes.

[0090] The base of the human-machine interaction bracelet is composed of hydrogel, which has more than 100% stretchability and can be applied to various parts of the human body, including arms and legs. We fixed the sensor on the upper arm, as shown in Figure 12(a), and collected EMG and FMG data corresponding to four different arm movements: "upward rotation", "downward rotation", "left rotation" and "right rotation". It is obvious that there are significant differences in the EMG and muscle force signals of all 8 channels under different movements. In addition, we use a neural network algorithm to specifically recognize these movements; the results show that the recognition accuracy reaches 94% when worn on the upper arm. It is worth noting that due to its excellent stretchability, the HMI bracelet can also be comfortably worn on the calf and thigh, and the recognition accuracy of these four movements reaches 97% and 95% respectively (as shown in Figure 12(b) and Figure 12(c)).

[0091] Optionally, in the dual-modal sensing-based signal recognition system in an embodiment of the present invention, the hydrogel is modified by a catechol group-rich material and a polymer monomer.

[0092] Next, a signal recognition method based on dual-modal sensing according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0093] A signal recognition method based on dual-modal sensing is provided in an embodiment of the present invention. The signal recognition method based on dual-modal sensing in the embodiment of the present invention can be applied to a terminal, or to a server, or can be software running in a terminal or a server, etc. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The signal recognition method based on dual-modal sensing in the embodiment of the present invention is applied to the signal recognition system based on dual-modal sensing described above, and mainly includes the following steps:

[0094] A composite signal of the object is measured by a human-computer interaction bracelet, wherein the composite signal includes a fused electromyographic signal and a muscle force signal;

[0095] The composite signal is decoupled, and a target recognition result of the object is obtained according to the decoupling result.

[0096] Optionally, the signal recognition method based on dual-modal sensing in the embodiment of the present invention further includes:

[0097] If the average value of the electromyographic signal within the first time period is greater than or equal to the first signal threshold, it is determined that the current signal is a gesture signal; the gesture signal is a composite signal that can be used for gesture recognition;

[0098] Based on the current moment, the gesture signal within the second time duration is obtained as a composite signal.

[0099] Optionally, the signal recognition method based on dual-modal sensing in the embodiment of the present invention further includes:

[0100] The composite signal is decoupled to extract characteristic parameters of the electromyographic and muscle force signals; the characteristic parameters include: peak-to-peak value of the muscle force signal, maximum value of the derivative of the muscle force signal, minimum value of the derivative of the muscle force signal, maximum value of the electromyographic signal, minimum value of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, peak-to-peak value of the electromyographic signal, absolute average value of the electromyographic signal, average value of the electromyographic signal, root mean square of the electromyographic signal, standard deviation of the electromyographic signal, variance of the electromyographic signal, and frequency threshold mean of the electromyographic signal.

[0101] Exemplarily, the characteristic parameters include an average value of the 10 largest values ​​of the electromyographic signal, an average value of the 50 largest values ​​of the electromyographic signal, an average value of the 10 smallest values ​​of the electromyographic signal, and an average value of the 50 smallest values ​​of the electromyographic signal.

[0102] Optionally, the signal recognition method based on dual-modal sensing in the embodiment of the present invention further includes:

[0103] Inputting the characteristic parameters into a trained multi-label neural network model for recognition to obtain the target recognition result;

[0104] The multi-label neural network model is trained by the following steps:

[0105] Attaching the human-computer interaction bracelet to the skin of the subject to obtain gesture sample signals under different actions;

[0106] Extracting sample feature parameters of the gesture sample signal, inputting the sample feature parameters into the multi-label neural network model, and obtaining a sample recognition result;

[0107] According to the difference between the sample recognition result and the true result of the sample, a loss function is established, and the parameters of the multi-label neural network model are updated according to the loss function to obtain the trained multi-label neural network model.

[0108] Optionally, in the signal recognition method based on dual-modal sensing in the embodiment of the present invention, the decoupling process of the composite signal includes:

[0109] Preprocessing the composite signal to obtain a base signal;

[0110] Determine a wavelet basis function and a decomposition layer number, and perform multi-layer decomposition on the base signal according to the wavelet basis function and the decomposition layer number to obtain a multi-layer signal; wherein each layer of decomposition corresponds to a coefficient and a frequency band;

[0111] Based on the coefficients, an inverse wavelet transform is performed to obtain decoupled electromyographic signals and muscle force signals; the electromyographic signals include signals corresponding to frequency bands D2, D3, D4, and D5, and the muscle force signals include signals corresponding to frequency bands D6, D7, D8, and A8.

[0112] In some embodiments, the process of the wavelet transform algorithm includes the following steps:

[0113] 1. Signal preprocessing.

[0114] First, the signal must be acquired. The signal in the embodiment of the present application is a composite signal that represents the fusion of the electromyographic signal and the muscle force signal.

[0115] Perform necessary preprocessing on the signal, including denoising (if the noise is obvious), normalization and other operations. Normalization can map the amplitude of the signal to a specific range, such as normalizing the signal amplitude to the interval [-1,1], which is helpful for subsequent wavelet transform processing.

[0116] 2. Select the wavelet basis function and the number of decomposition levels.

[0117] Select the appropriate wavelet basis function according to the properties of the signal (such as the frequency characteristics of the signal, mutation, etc.). For example, for signals with sharp mutations, such as the QRS complex in the electrocardiogram signal, the Daubechies wavelet may be more appropriate; for smooth signals, a smoother wavelet basis may be selected.

[0118] Determine the number of decomposition layers. This usually requires considering the complexity of the signal and the purpose of the analysis. If you want to analyze the high-frequency details of the signal in detail, you need to design more decomposition layers; if you only care about the general frequency distribution of the signal, you can set fewer layers.

[0119] 3. Wavelet decomposition.

[0120] The signal is decomposed by a filter bank (low-pass filter and high-pass filter) using the selected wavelet basis function and decomposition layer number. Each layer of decomposition divides the signal into low-frequency approximate coefficients and high-frequency detail coefficients. For example, the first decomposition decomposes the original signal into the approximate part A1 and the detail part D1. The approximate part A1 represents the low-frequency component of the signal, and the detail part D1 represents the high-frequency component of the signal. If multiple layers of decomposition are performed, A1 will be further decomposed to obtain A2 and D2, and so on. The frequency of A2 is lower than that of A1, and D2 is the relatively high-frequency part decomposed from A1.

[0121] 4. Coefficient processing (optional).

[0122] The decomposed coefficients (approximate coefficients and detail coefficients) are processed. Common processing includes threshold processing to remove noise.

[0123] 5. Wavelet reconstruction.

[0124] The processed coefficients (or unprocessed coefficients, if the coefficient processing step is not required) are used for inverse wavelet transform. The decomposed approximate coefficients and detail coefficients are reconstructed in the reverse order of decomposition, and finally the signal processed by wavelet transform is obtained.

[0125] The signal recognition system and method provided by the present application are described in detail below through specific embodiments:

[0126] Embodiment 1:

[0127] The above-mentioned adhesive hydrogel achieves self-adhesion and improved mechanical properties by modifying polyacrylamide with tannic acid (which can be other materials rich in catechol groups, such as dopamine) and carrageenan (which can be other polymer monomers, such as vinyl alcohol, cellulose, etc.), and obtains water retention and antifreeze properties by soaking in glycerol solution to improve its stability as an electrical signal interface. The adhesion strength between the zinc electrode and the hydrogel is improved by electrochemical treatment, thereby improving the stability and durability of the sensor.

[0128] Embodiment 2:

[0129] Figure 4This is the schematic diagram of the multi-channel sensor signal acquisition system, including the minimum system circuit of STM32F103C8T6, power module and Bluetooth module. The minimum system of STM32F103C8T6 includes peripheral circuits such as reset circuit, download circuit, crystal oscillator circuit, etc., which are powered by a 3.3V power supply. The reset circuit is a power-on reset circuit composed of R29 and C37, which generates a low level at the moment of power-on, and then sets it high to achieve the reset of STM32F103C8T6. The 12MHz crystal oscillator provides the clock signal for STM32F103C8T6, and the series capacitors C34, C35 and the parallel resistor R27 improve the stability of the clock signal. The program is burned by SWD download method, and it is connected to the SWD downloader through the P1 interface. The power module of the circuit is realized by the stable output 3.3VLDO chip TPS73633. The power indication circuit is composed of R42 and LED to display the power supply status. The system is powered by a 3.7mV lithium battery. The Bluetooth module uses the transparent Bluetooth module RBGMBG22A1 built by Xinchida Company based on TI's CC2640R2F chip, which is powered by 3.3V.

[0130] The schematic diagram of the multi-channel analog front-end circuit is as follows Figure 5 As shown. The eight-channel analog front-end chip ADS1298 is used, which can obtain eight-channel differential signals. R16 / C13, R1 / C1, R18 / C15, R6 / C5, R20 / C17, R10 / C8, R24 / C31, R13 / C11, R2 / C2, R17 / C7, R7 / C6, R19 / C16, R11 / C9, R21 / C18, R14 / C12, R25 / C32 are 16-channel low-pass filters to filter the high-frequency signals output by the sensor. The filtered data is collected by ADS1298 and sent to the STM32F103C8T6 chip through the SPI interface.

[0131] Embodiment 3:

[0132] The human-computer interaction algorithm of the present invention is implemented on the server, and the server uses wavelet transform to decouple the differential signal to obtain the electromyographic signal and the muscle force signal. The characteristic parameters of the electromyographic signal and the muscle force signal are extracted and sent to the trained multi-label neural network algorithm, and the action intention is calculated and transmitted to the execution terminal wirelessly, such as Figure 6As shown. The gesture recognition algorithm includes gesture signal positioning and segmentation, feature parameter extraction and neural network classification algorithm. Myoelectric signals are generated spontaneously by humans and are not affected by external forces, so they are used to identify and locate gesture recognition signals. A myoelectric threshold is set. When the average myoelectric value within 0.1 seconds (i.e. the first duration) exceeds 0.05mV (i.e. the first signal threshold), it is determined to be a gesture signal. The myoelectric and muscle force signal data 0.3s before and after this time (i.e. the second duration) are taken as gesture signals. Then, the characteristic parameters of the electromyographic and force signals are extracted: peak-to-peak value of the force signal, maximum value of the derivative of the force signal, minimum value of the derivative of the force signal, maximum value of the electromyographic signal, minimum value of the electromyographic signal, average of the 10 largest values ​​of the electromyographic signal, average of the 50 largest values ​​of the electromyographic signal, average of the 10 smallest values ​​of the electromyographic signal, average of the 50 smallest values ​​of the electromyographic signal, peak-to-peak value of the electromyographic signal, absolute average value of the electromyographic signal, average value of the electromyographic signal, root mean square value of the electromyographic signal, standard deviation of the electromyographic signal, variance of the electromyographic signal, and frequency threshold mean value of the electromyographic signal. The neural network classification algorithm is divided into an input layer of 18 (or 18*number of channels) neurons, 3 hidden layers of 1024, 512, and 256 neurons, and an output layer of 5 (or 9, adjusted according to the number of actions). Tanh is used as the activation function between hidden layers, and ReLU function is used as the activation function between the last hidden layer and the output layer. The final output test sample is classified into the probability distribution of different gestures, and the highest probability is taken as the prediction result.

[0133] Embodiment 4:

[0134] The single-channel EMG and muscle force dual-modal sensor is attached to the finger extensor muscle area of ​​the arm to obtain EMG signals and muscle force signals under different gestures. Figure 7 As shown in the figure, five different gestures (fist, OK, gesture one, gesture three, gesture five) are displayed, and the electromyographic signals and muscle deformation signals collected from the extensor muscles of the arms are respectively displayed. After signal processing, the data set is input into the neural network algorithm, as shown in the figure. Figure 6 The final conclusion is that the recognition accuracy of the EMG model is 64%, and the recognition accuracy of the EMG-muscle force dual-modal signal is 86.5%, as shown in Figure 8(a) and Figure 8(b).

[0135] Embodiment 5:

[0136] The above bracelet was worn on the arm, and the 8-channel differential signals corresponding to the above 9 actions were obtained, and the 8-channel electromyographic signals and muscle force signals were obtained by decoupling, as shown in Figure 9(a) and Figure 9(b). After the data set was processed, it was input into the neural network algorithm. The results showed that the bracelet's recognition accuracy for the 9 different actions reached 95% (such as Fig.11aAs shown in Figure 2, the recognition accuracy of the wristband can reach 94.2%, which is 18% higher than that of the single-channel EMG-force dual-modal signal. In addition, the wristband is designed to adapt to different arm sizes (23 cm), and the recognition accuracy reaches 94.2%, as shown in Figure 2. Fig.11b In addition, we evaluated the recognition accuracy of eight-channel electromyographic signals obtained using commercial electrodes (see Fig.11c As shown in Figure 12, the performance of the multi-channel dual-mode HMI bracelet is limited to 89%. Therefore, the multi-channel dual-mode HMI bracelet not only surpasses the traditional multi-channel electromyographic model in recognition accuracy, but also provides a more convenient way of assembly and wearing. Its inherent stretchability further enhances compatibility with different arm sizes. At the same time, the bracelet can also be applied to large-size areas of the human body such as the upper arm, calf and thigh, and achieved a recognition accuracy of more than 94%, as shown in Figure 12.

[0137] The embodiments of the present application propose a solution for realizing a single sensor to obtain myoelectric and muscle force dual-modal signals through a digital signal decoupling algorithm, and a solution for constructing a multi-channel dual-modal human-computer interaction bracelet through integrated dual-modal sensing.

[0138] Compared with commercial electrophysiological electrodes and other types of mechanical sensors, the integrated dual-modal sensor proposed in the embodiment of the present application cleverly combines digital signal processing methods to achieve a single device to obtain dual-modal signals of electromyography and muscle force. It has been verified that the dual-modal signal obtained based on the integrated dual-modal sensor is used for human-computer interaction, and the recognition accuracy is much higher than that of the electromyographic signal obtained by commercial electrodes.

[0139] The multi-channel dual-modal human-computer interaction bracelet based on the integrated dual-modal sensing design of the present application embodiment has high motion recognition accuracy and convenient assembly method. Compared with commercial human-computer interaction bracelets and existing cutting-edge flexible human-computer interaction interfaces, the human-computer interaction bracelet based on integrated dual-modal sensing only needs to attach a flexible circuit board with an electrode array to the hydrogel surface to realize the construction of multi-channel dual-modal sensing, without the need for a clean room and expensive industrial design, so it has a lower cost. At the same time, hydrogel is used as the base material, so it has the advantages of self-adhesion and super stretchability. Therefore, it can be worn stably above the arm without relying on elastic bands or other adhesives. Good stretchability makes the bracelet compatible with different arms and can even be worn on human thighs.

[0140] It is understandable that in the related art, 17 commercial electrodes and 8 pressure sensors are used to obtain 8-channel electromyographic and muscle force signals. However, this will significantly increase the difficulty of bracelet design. In order to integrate the sensors together, expensive and complex structural design is required. At the same time, additional circuits need to be added to obtain muscle force signals with pressure sensors, which will increase the difficulty and cost of circuit design. At the same time, the volume and weight of the circuit will increase, reducing the comfort of wearing. At the same time, the multi-channel dual-modal human-computer interaction bracelet constructed in this way often needs to be fixed by elastic bands, which is uncomfortable to wear and difficult to adapt to arms of different sizes.

[0141] It can be seen that the contents of the above system embodiments are all applicable to the present method embodiments, the functions specifically implemented by the present method embodiments are the same as those of the above system embodiments, and the beneficial effects achieved are also the same as those achieved by the above system embodiments.

[0142] On the other hand, an embodiment of the present invention provides a signal recognition device based on dual-modal sensing, comprising:

[0143] at least one processor;

[0144] at least one memory for storing at least one program;

[0145] When the at least one program is executed by the at least one processor, the at least one processor implements the signal recognition method based on dual-modal sensing.

[0146] Similarly, the contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] An embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the above-mentioned signal recognition method based on dual-modal sensing.

[0148] Similarly, the contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0150] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0153] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0154] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0155] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0156] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0157] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A signal recognition system based on dual-modal sensing, characterized in that: include: Human-computer interaction bracelet, hardware module and server; The human-computer interaction bracelet is used to contact the skin of the object to measure a composite signal of the object, wherein the composite signal includes a fused electromyographic signal and a muscle force signal; The hardware module is used to extract the composite signal; the server is used to receive the composite signal extracted by the hardware module, perform decoupling processing on the composite signal, and obtain the target recognition result of the object according to the decoupling result; The human-computer interaction bracelet includes a hydrogel, a first electrode and a second electrode. The first electrode and the second electrode are fixed to the surface of the hydrogel, and the first electrode and the second electrode are used to contact the skin.

2. The signal recognition system based on dual-modal sensing according to claim 1, characterized in that: The first electrode and the second electrode are fixed to the serpentine wire by soldering, and the serpentine wire is fixed to the surface of the hydrogel by electrochemical treatment.

3. The signal recognition system based on dual-modal sensing according to claim 1, characterized in that: The hardware module includes a microprocessor unit, an analog front-end unit and a filtering unit; the human-computer interaction bracelet is connected to the analog front-end unit through the filtering unit, and the analog front-end unit is connected to the server through the microprocessor unit.

4. The signal recognition system based on dual-modal sensing according to claim 1, characterized in that: The server is used to decouple the composite signal through a wavelet transform algorithm, extract characteristic parameters of the electromyographic signal and the muscle force signal, and identify the characteristic parameters through a trained multi-label neural network model to obtain the target recognition result.

5. The signal recognition system based on dual-modal sensing according to claim 1, characterized in that: The hydrogel is modified by a material rich in catechol groups and a polymer monomer.

6. A signal recognition method based on dual-modal sensing, characterized in that: Applied to the signal recognition system based on dual-modal sensing as claimed in claim 1, the method comprises: A composite signal of an object is measured by a human-computer interaction bracelet, wherein the composite signal includes a fused electromyographic signal and a muscle force signal; the composite signal is decoupled, and a target recognition result of the object is obtained according to the decoupling result.

7. The signal recognition method based on dual-modal sensing according to claim 6, characterized in that: The method further comprises: If the average value of the electromyographic signal within the first time period is greater than or equal to the first signal threshold, it is determined that the current signal is a gesture signal; the gesture signal is a composite signal that can be used for gesture recognition; Based on the current moment, the gesture signal within the second time duration is obtained as a composite signal.

8. The signal recognition method based on dual-modal sensing according to claim 6, characterized in that: The method further comprises: The composite signal is decoupled to extract characteristic parameters of the electromyographic and muscle force signals; the characteristic parameters include: peak-to-peak value of the muscle force signal, maximum value of the derivative of the muscle force signal, minimum value of the derivative of the muscle force signal, maximum value of the electromyographic signal, minimum value of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several maximum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, average value of several minimum values ​​of the electromyographic signal, peak-to-peak value of the electromyographic signal, absolute average value of the electromyographic signal, average value of the electromyographic signal, root mean square of the electromyographic signal, standard deviation of the electromyographic signal, variance of the electromyographic signal, and frequency threshold mean of the electromyographic signal.

9. The signal recognition method based on dual-modal sensing according to claim 8, characterized in that: The method further comprises: inputting the characteristic parameters into a trained multi-label neural network model for recognition to obtain the target recognition result; The multi-label neural network model is trained by the following steps: Attaching the human-computer interaction bracelet to the skin of the subject to obtain gesture sample signals under different actions; Extracting sample feature parameters of the gesture sample signal, inputting the sample feature parameters into the multi-label neural network model, and obtaining a sample recognition result; According to the difference between the sample recognition result and the true result of the sample, a loss function is established, and the parameters of the multi-label neural network model are updated according to the loss function to obtain the trained multi-label neural network model.

10. The signal recognition method based on dual-modal sensing according to claim 6, characterized in that: The decoupling process of the composite signal includes: Preprocessing the composite signal to obtain a base signal; Determine a wavelet basis function and a decomposition layer number, and perform multi-layer decomposition on the base signal according to the wavelet basis function and the decomposition layer number to obtain a multi-layer signal; wherein each layer of decomposition corresponds to a coefficient and a frequency band; Based on the coefficients, an inverse wavelet transform is performed to obtain decoupled electromyographic signals and muscle force signals; the electromyographic signals include signals corresponding to the D2 frequency band, the D3 frequency band, the D4 frequency band, and the D5 frequency band, and the muscle force signals include signals corresponding to the D6 frequency band, the D7 frequency band, the D8 frequency band, and the A8 frequency band.

Images