Human-computer interaction method and system based on hydrogel flexible electrode

Abstract

The invention discloses a human-computer interaction method and system based on a hydrogel flexible electrode. The method comprises the following steps: in response to a triggering operation on a first control received on a functional interface, acquiring an acceleration signal and a plurality of paths of electromyographic signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object; inputting the acceleration signal and the electromyographic signal into an intention prediction model to obtain intention information; according to the intention information, based on the first control, determining an output signal of the first terminal so as to enable a target device to execute the target function; the function interface is an interface of the first terminal. The electromyographic signals are acquired through the hydrogel flexible motor, so that the acquisition precision is improved; through fusion of the electromyographic signals and the acceleration signals, the interaction accuracy is improved. The method can be widely applied to the technical field of computers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a human-computer interaction method and system based on hydrogel flexible electrode. BACKGROUND

[0002] High-quality signal input is essential to build an efficient human-computer interaction system, but in practical applications, the high interface impedance between traditional hydrogel and skin tissue leads to the inability to efficiently and accurately transmit weak sEMG signals from the tissue to the electrode to obtain high signal-to-noise ratio signals, and the interface between the electrode and the tissue will also be affected by extreme deformation, large-scale body movement and vibration.

[0003] Traditional myoelectricity collection equipment mostly uses separate hardware devices, which will increase the size of the device and reduce the portability of the device when implementing multi-channel collection, and single myoelectricity signal cannot realize high-precision gesture recognition. Although the fusion of multi-modal signal collection equipment can increase the recognition accuracy, it will increase the cost and size of the equipment, and the algorithm complexity will also increase, thereby affecting the real-time performance. Most of the interactive systems in the research are single-function and lack of portability in practical applications. SUMMARY

[0004] The purpose of the present application is to at least partially solve one of the technical problems existing in the prior art.

[0005] To this end, the purpose of the present application is to provide an efficient human-computer interaction method and system based on hydrogel flexible electrode.

[0006] In order to achieve the above technical purpose, one aspect of an embodiment of the present application provides a human-computer interaction method based on hydrogel flexible electrode, comprising the following steps: in response to a trigger operation of a first control received at a function interface, collecting acceleration signals and multi-channel myoelectricity signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object; inputting the acceleration signals and the myoelectricity signals into an intention prediction model to obtain the intention information; according to the intention information, determining an output signal of a first terminal based on the first control, so that a target device executes the target function; the function interface is the interface of the first terminal. The present application realizes the collection of myoelectricity signals through a hydrogel flexible motor, which improves the collection accuracy; through the fusion of myoelectricity signals and acceleration signals, the interaction accuracy is improved.

[0007] In some embodiments, the first control includes a sign language recognition control, and the determination of the output signal of the first terminal based on the first control according to the intention information so that the target device executes the target function includes:

[0008] According to the intention information, an output signal of the first terminal is determined based on the sign language recognition control, so that the target device performs the target function.

[0009] The first control includes a mechanical control control, and the target device includes a mechanical hand.

[0010] According to the intention information, an output signal of the first terminal is determined based on the sign language recognition control, so that the target device performs the target function.

[0011] In some embodiments, the first control includes a rehabilitation control, and the target device includes a rehabilitation glove.

[0012] According to the intention information, an output signal of the first terminal is determined based on the sign language recognition control, so that the target device performs the target function.

[0013] According to the intention information, an output signal of the first terminal is determined based on the sign language recognition control, so that the target device performs the target function.

[0014] In some embodiments, the hydrogel flexible electrode is prepared by the following steps:

[0015] The acrylamide, crosslinking agent, photoinitiator and sodium chloride are mixed in a predetermined ratio and then added to a deionized water mixed solution to obtain a first mixed solution; wherein the deionized water mixed solution is obtained by mixing honey and deionized water.

[0016] The first mixed solution is injected into a mold and subjected to light irradiation treatment to obtain a hydrogel flexible electrode.

[0017] In some embodiments, the intention prediction model is trained by the following steps:

[0018] Obtain acceleration signal samples and multi-channel electromyography signal samples;

[0019] The acceleration signal samples and the electromyography signal samples are subjected to smooth wavelet packet transform processing, and the processed feature samples are input into the intention prediction model to obtain predicted intention samples.

[0020] According to the predicted intention samples and the true intention, a loss function is determined, and the intention prediction model is trained according to the loss function to obtain a trained intention prediction model.

[0021] In another aspect, the embodiment of the present application provides a human-computer interaction system based on a hydrogel flexible electrode, comprising:

[0022] The acquisition module, the control module, the first terminal and the target device;

[0023] The acquisition module is configured to: in response to a triggering operation of a first control received at a function interface, acquire an acceleration signal and a plurality of electromyography signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object;

[0024] The control module is configured to: input the acceleration signal and the electromyography signal into an intention prediction model to obtain the intention information;

[0025] According to the intention information, an output signal of the first terminal is determined based on the first control, so that the target device executes the target function; the function interface is an interface of the first terminal.

[0026] In some embodiments, the acquisition module comprises:

[0027] An electromyography signal acquisition module and an acceleration acquisition module, and the hydrogel flexible electrode is integrated on the electromyography signal acquisition module.

[0028] In some embodiments, the electromyography signal acquisition module comprises a KS1082 chip, and the acceleration acquisition module comprises an ADXL345 chip;

[0029] The control module is configured to configure an opening channel of the KS1082 chip, and acquire an electromyography signal of the KS1082 chip through an ADC;

[0030] The acceleration signal of the ADXL345 chip is transmitted to the control module through an SDA port and an SCL port.

[0031] In another aspect, the embodiment of the present application provides a human-computer interaction system based on a hydrogel flexible electrode, comprising:

[0032] At least one processor;

[0033] At least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned human-computer interaction method based on a hydrogel flexible electrode.

[0035] In another aspect, an embodiment of the present application provides a storage medium having stored therein a program executable by a processor for implementing the above-mentioned human-computer interaction method based on a hydrogel flexible electrode when executed by the processor.

[0036] The present application has at least the following advantages: the method provided by the present application comprises: in response to a triggering operation of a first control received at a function interface, collecting an acceleration signal and a plurality of electromyography signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object; inputting the acceleration signal and the electromyography signals into an intention prediction model to obtain the intention information; determining an output signal of a first terminal based on the first control according to the intention information, so that a target device executes the target function; and the function interface is an interface of the first terminal. The present application collects electromyography signals through a hydrogel flexible electrode, thereby improving collection accuracy; and the fusion of electromyography signals and acceleration signals improves interaction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing some embodiments of the technical solutions in the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0038] Figure 1 A flowchart of an embodiment of the human-computer interaction method based on a hydrogel flexible electrode provided by the present application;

[0039] Figure 2 A flowchart of an embodiment of the preparation process of the hydrogel flexible electrode provided by the present application;

[0040] Figure 3 An electrode performance comparison chart of an embodiment of the hydrogel flexible electrode provided by the present application;

[0041] Figure 4 A physical schematic diagram of an embodiment of the collection module provided by the present application;

[0042] Figure 5 A flowchart of an embodiment of the signal collection process provided by the present application;

[0043] Figure 6 A principle schematic diagram of an embodiment of the signal collection process provided by the present application;

[0044] Figure 7A contrast diagram of an embodiment of the signal collected by the application;

[0045] Figure 8 A flow diagram of an embodiment of the human-computer interaction process provided by the application;

[0046] Figure 9 An interface diagram of an embodiment of the human-computer interaction process provided by the application;

[0047] Figure 10 A monitoring diagram of an embodiment of the sign language recognition provided by the application;

[0048] Figure 11 An application diagram of an embodiment of the rehabilitation training provided by the application. DETAILED DESCRIPTION

[0049] Embodiments of the application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numbers represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the application and cannot be understood as a limitation of the application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] Multimodal sensor: a multimodal sensor refers to a device that can detect two or more variables at the same time and convert them into signals that can be received and processed.

[0051] Human-computer interaction: physiological signals of the human body are collected through a signal collection device, AI algorithms are used to recognize the signals, and corresponding output devices are controlled to complete corresponding tasks.

[0052] With the rapid development of flexible electronic technology, flexible wearable human-computer interaction devices gradually replace traditional rigid interface devices due to their ability to collect human bioelectric signals in real time and efficiently, and play an increasingly important role in health monitoring, active rehabilitation, human-computer interaction and other fields. Building a flexible human-computer interaction system with high accuracy, rich functions and portability has become an important development direction for future human-computer interaction technology. At present, intelligent data gloves based on strain sensors are a common wearable human-computer interaction interface and have been widely used in sign language recognition, mechanical control and other fields. However, the wearing method of intelligent data gloves limits the free movement of the user's hands, to some extent, reducing the user's wearing comfort and convenience. Therefore, it is necessary to explore new human-computer interaction interface materials that are more efficient, comfortable and suitable for long-term wear. Surface electromyography (sEMG) signals are weak electrical signals generated on the surface of the skin by human muscle activity, containing rich muscle movement information. By wearing sEMG sensing devices on the human arm, accurate and diversified human-computer interaction tasks can be completed without hindering the natural movement of the hands. In recent years, hydrogel materials with three-dimensional network structure have been widely used as electrode materials for flexible electronic devices due to their flexibility, self-adhesion and biocompatibility comparable to human skin. Using hydrogel as a sEMG electrode to collect sEMG signals can provide an ideal input signal for a human-computer interaction system. In an embodiment, by adding MXene material to the hydrogel, a hydrogel electrode is made to collect arm electromyography signals and complete the application of sign language recognition.

[0053] High-quality signal input is crucial for building efficient human-computer interaction systems. However, in practical applications, the high interfacial impedance between conventional hydrogels and skin tissue prevents efficient and accurate transmission of weak sEMG signals from the tissue to the electrodes, achieving a high signal-to-noise ratio. Furthermore, the electrode-tissue interface is subject to extreme deformation, extensive body movement, and vibration, which can affect signal quality. Numerous studies have optimized the electrical properties of hydrogels by adding conductive polymers, carbon nanotubes, and MXene. However, the addition of these fillers often compromises the inherent flexibility and mechanical properties of the hydrogels, preventing them from conforming well to the complex human body surface. Furthermore, the biosafety and antibacterial properties of the electrodes also hinder the practical requirements of long-term, continuous wear. Furthermore, conventional EMG acquisition devices often utilize separate, discrete hardware components. Multi-channel acquisition increases device size and reduces portability, while a single EMG signal cannot achieve high-precision gesture recognition. While integrating multimodal signal acquisition devices can improve recognition accuracy, it also increases device cost and size, and increases algorithm complexity, compromising real-time performance. Consequently, most of these interactive systems suffer from limited functionality and lack portability in practical applications, significantly impacting network and other environmental factors. Therefore, it is necessary to develop an integrated flexible interface device that can sense high-quality bioelectric signals while taking into account high accuracy, comfort and safety, so as to build a real-time, multifunctional mobile human-computer interaction system.

[0054] The human-computer interaction method and system based on hydrogel flexible electrodes proposed in an embodiment of the present invention will be described in detail below with reference to the accompanying drawings. First, the human-computer interaction method based on hydrogel flexible electrodes proposed in an embodiment of the present invention will be described with reference to the accompanying drawings.

[0055] Reference Figure 1 , an embodiment of the present invention provides a human-computer interaction method based on hydrogel flexible electrodes. The human-computer interaction method based on hydrogel flexible electrodes in the embodiment of the present invention can be applied to a terminal, can be applied to a server, and can also be software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The human-computer interaction method based on hydrogel flexible electrodes in the embodiment of the present invention mainly includes the following steps:

[0056] S100: in response to a triggering operation of a first control received at a function interface, collecting an acceleration signal and a plurality of electromyography signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object;

[0057] S200: inputting the acceleration signal and the electromyography signal into an intention prediction model to obtain the intention information;

[0058] S300: determining an output signal of a first terminal based on the first control according to the intention information, so that a target device executes the target function; the function interface is an interface of the first terminal.

[0059] In some possible implementations, the target object can be a person being tested, and the acceleration signal and the plurality of electromyography signals of the target object are collected at an arm of the target object through the hydrogel flexible electrode. The first control can be a control representing the target function. The intention prediction model can be any feasible artificial intelligence model. The target device can be a device connected to the first terminal, or a certain device of the first terminal. When the first control is not a sign language control, the target device can be a voice output device of the first terminal, or a display device of the first terminal.

[0060] In some embodiments, the first control includes a sign language recognition control, and the determining of the output signal of the first terminal based on the first control according to the intention information, so that the target device executes the target function, includes:

[0061] determining a display instruction output by the first terminal based on the sign language recognition control according to the intention information, so that the target device outputs voice information or text information; the voice information and the text information correspond to the intention information;

[0062] The first control includes a mechanical control control, and the target device includes a mechanical hand; the determining of the output signal of the first terminal based on the mechanical control control according to the intention information, so that the target device executes the target function, includes:

[0063] determining a motion instruction output by the first terminal based on the mechanical control control according to the intention information, so that the mechanical hand moves according to the motion instruction.

[0064] In some embodiments, the first control includes a rehabilitation control, and the target device includes a rehabilitation glove; the determining of the output signal of the first terminal based on the first control according to the intention information, so that the target device executes the target function, includes:

[0065] determining a motion instruction output by the first terminal based on the rehabilitation control according to the intention information.

[0066] According to the action instruction, action programming is performed to control the rehabilitation glove to perform a rehabilitation action.

[0067] In some embodiments, the hydrogel flexible electrode is prepared by the following steps:

[0068] The acrylamide, crosslinking agent, photoinitiator and sodium chloride are mixed in a preset ratio and then added to a deionized water mixed solution to obtain a first mixed solution; the deionized water mixed solution is obtained by mixing honey and deionized water;

[0069] The first mixed solution is injected into a mold and subjected to light irradiation treatment to obtain a hydrogel flexible electrode.

[0070] In some embodiments, the intention prediction model is trained by the following steps:

[0071] Obtain acceleration signal samples and multi-channel electromyography signal samples;

[0072] Perform smooth wavelet packet transform processing on the acceleration signal samples and the electromyography signal samples, and input the processed feature samples into the intention prediction model to obtain predicted intention samples;

[0073] According to the predicted intention samples and the real intention, a loss function is determined, and the intention prediction model is trained according to the loss function to obtain a trained intention prediction model.

[0074] In another aspect, an embodiment of the present application proposes a human-computer interaction system based on a hydrogel flexible electrode, comprising:

[0075] The acquisition module, the control module, the first terminal and the target device;

[0076] The acquisition module is configured to: in response to a trigger operation on a first control received at a function interface, acquire acceleration signals and multi-channel electromyography signals of a target object through a hydrogel flexible electrode; the first control is a target function realized based on intention information of the target object;

[0077] The control module is configured to: input the acceleration signals and the electromyography signals into an intention prediction model to obtain the intention information;

[0078] According to the intention information, based on the first control, an output signal of the first terminal is determined to enable the target device to execute the target function; the function interface is an interface of the first terminal.

[0079] In some embodiments, the acquisition module comprises:

[0080] An electromyography signal acquisition module and an acceleration acquisition module, the hydrogel flexible electrode is integrated on the electromyography signal acquisition module.

[0081] In some embodiments, the electromyography signal acquisition module comprises a KS1082 chip, and the acceleration acquisition module comprises an ADXL345 chip.

[0082] The control module is configured to open a channel of the KS1082 chip, and collect the electromyography signal of the KS1082 chip through an ADC.

[0083] The acceleration signal of the ADXL345 chip is transmitted to the control module through an SDA port and an SCL port.

[0084] In the following, a specific embodiment is used to introduce the human-computer interaction method provided in the application in detail.

[0085] The hydrogel flexible electrode has excellent flexibility, adhesion, conductivity, biocompatibility and antibacterial property, is suitable for application of a skin electrode, and is simple to prepare and low in cost. Figure 2 is a preparation method of the hydrogel electrode, honey and sodium chloride are added into a polyacrylamide hydrogel network to prepare the high-performance hydrogel electrode, Figure 3 (a) to (h) show that the hydrogel has excellent adhesion and can be stably adhered to a human body surface, ensures stable signal collection, has excellent flexibility, ensures that the hydrogel can be fully attached to the human body surface, and the hydrogel also has excellent electrical conductivity, as a hydrogel electrode has low interface impedance, and can collect high signal-to-noise ratio surface electromyography signals.

[0086] The hydrogel electrode is integrated on a flexible signal acquisition device, the flexible signal acquisition device adopts flexible PI as a substrate, Figure 4 shows the appearance of the acquisition device, including a front surface and a back surface, adopts a wristband shape design wearable on a human arm, Figure 5 shows a signal acquisition system, the hydrogel is integrated on the acquisition device, and the STM32 in the acquisition circuit controls the acquisition chip to synchronously collect double-channel surface electromyography signals and acceleration signals. Figure 6 shows a schematic diagram of the circuit of the acquisition device. To test the performance of the acquisition device, 9 kinds of sign language signal graphs are collected, as shown in Figure 7 , the signal graph shows double-channel surface electromyography signals and acceleration signals of 9 kinds of sign language.

[0087] The application develops a mobile multifunctional human-computer interaction system, which realizes data receiving, processing and transfer by using a mobile APP program, the APP can be wirelessly connected to the collection equipment to realize data transmission, and the APP is internally provided with efficient data preprocessing algorithms and neural network models, which can realize real-time processing of signals from the transmission equipment and complete gesture classification through an AI model, Figure 8 The working flow chart of the mobile human-computer interaction system is shown. The APP integrates multiple human-computer interaction functions, Figure 9 The function page of the APP is shown, which can realize sign language recognition, motion monitoring, mechanical control and active rehabilitation human-computer interaction applications at the same time, and the system is not limited by the network and can realize stable and convenient human-computer interaction applications in complex scenes. Figure 10 The (a) to (c) in the figure show that the system can realize real-time sign language recognition application, can display the waveforms of the double-channel surface electromyography signals and acceleration signals in real time, and can realize sign language recognition in an outdoor environment, greatly facilitating the daily communication of the deaf-mute group, Figure 11 The (a) to (c) in the figure show that the mobile human-computer interaction system can recognize gesture actions in real time, and can send instructions to the output equipment through the APP to control the machine to complete the interactive application of picking and placing dangerous objects and assisting stroke disabled patients to complete active rehabilitation. It has great application prospect. DETAILED DESCRIPTION

[0089] Figure 2 is a preparation schematic diagram of the hydrogel flexible electrode. The specific method is as follows: first, prepare a hydrogel with 0% honey content, add a mixture of 6g acrylamide, 6mg crosslinking agent, 0.2g photoinitiator and 2M NaCl into 40ml deionized water. Then, the mixed solution is injected into a mold. Finally, after irradiation under UV light (365nm) for 45 minutes, the mixture is polymerized into a hydrogel. To prepare hydrogels with different honey contents, only the deionized water is replaced with an equal volume (40ml) of a mixed solution of different proportions of honey and deionized water, and the volume ratio of honey is 10%, 20%, 30% and 40% respectively.

[0090] Figure 3 (a) in the figure is a test of the shear stress of the hydrogel on pigskin to evaluate the adhesion performance of the hydrogel, and the hydrogel can realize a shear stress of up to 4kPa on pigskin, which is sufficient to adhere to the skin stably. Figure 3 (b) in the figure is the modulus change of the hydrogel electrode with different honey contents tested by the tensile experiment, and it is found that the addition of honey can reduce the modulus of the hydrogel, and the hydrogel electrode with 30% honey content has a modulus of 2.2kPa, which has sufficient flexibility to ensure conformal contact with the skin. Figure 3 (c) in the figure is the conductivity of the hydrogel and the commercial electrode, and the hydrogel has excellent conductivity due to the addition of high-concentration NaCl, which is much higher than that of the commercial electrode,Figure 3 In (d) and (e), the conductivity and impedance of the hydrogel and commercial electrodes at different frequencies were tested. The hydrogel has better electrical properties than the commercial electrodes. The above performance ensures that the interface impedance is low enough when collecting physiological electrical signals, and the biocompatibility of the hydrogel obtained by biocompatibility test is as follows. Figure 3 As shown in (f), the cell biocompatibility experiment shows that the hydrogel has no cytotoxicity and has excellent biocompatibility, ensuring the biosafe application of electrode materials. Figure 3 (g) The antibacterial properties of the hydrogel were tested through antibacterial experiments, and it was found that the hydrogel had a significant inhibitory effect on the growth of bacteria, which can ensure that skin inflammation and allergies will not occur during long-term use. Figure 3 As shown in (e), the hydrogel electrode has lower skin interface impedance than the commercial electrode. Figure 3 Middle (h) shows that the hydrogel can collect electromyographic signals with a signal-to-noise ratio of up to 21dB, which is higher than that of commercial electrodes. Specific implementation 2:

[0092] In order to make data collection and equipment wearing more comfortable, the signal acquisition equipment is designed with FPCB board and made into a wristband shape suitable for arm wear. Figure 4 The design diagram of the wristband is shown. The entire wristband weighs 8 grams and uses a flexible circuit board design to ensure good and comfortable adhesion to the skin. It is also very small and has excellent portability. Figure 5 (a) shows that the hydrogel electrode is integrated with the device by attaching the metal contacts of the collection wristband, ensuring the high integration of the entire collection system. The collection circuit layout is as follows Figure 5 In part (b), the circuit size is 4cm × 4cm. The circuit layout in the figure can be divided into four modules: Module 1 is the MCU control module, Module 2 is the surface electromyography signal acquisition module, Module 3 is the acceleration signal acquisition module, and Module 4 is the Bluetooth module. Figure 5The signal acquisition process of the signal acquisition circuit is shown in (c). The sEMG signal (i.e., the electromyography signal) is a muscle electrical activity signal recorded by the electrode attached to the skin surface, which reflects the electrical physiological changes generated by the motor unit when the user contracts the hand muscles. When the brain sends a movement instruction, it will stimulate muscle movement, and the potential on the surface of the muscle will change. The electrode senses this change and collects it through the acquisition circuit, so that the sEMG signal can be captured. The three-axis acceleration signal is the mechanical signal in the X, Y and Z directions when the user's hand moves. The circuit uses the ECG acquisition chip KS1082 to collect double-channel electromyography signals, and uses the three-axis acceleration sensor ADXL345 to collect acceleration signals. KS1082 configures the amplification multiple and opens the channel by SPI of STM32. The double-channel surface electromyography signal is collected by the ADC of STM32, and the three-axis acceleration signal is collected by ADXL345. The ADXL345 acceleration signal sensor has a movable mass, a fixed electrode and a spring structure. When there is an external acceleration, the inertial force causes the mass to displace, resulting in a change in capacitance between the fixed electrode. This differential capacitance change is proportional to the acceleration. The capacitance change is converted to a voltage signal by the internal capacitance-voltage conversion structure of the chip, and is converted to a digital signal by the ADC. The digital signal is transmitted by I2C communication between ADXL345 and STM32 through SDA and SCL signal lines, and is stored in the memory of STM32. All signals are processed by STM32 and packaged into data frames, which are sent to Bluetooth through the serial port to complete the wireless real-time acquisition and transmission of data. Figure 6 The schematic diagram of the entire acquisition circuit is shown, which includes a power module, a main control module, a surface electromyography signal acquisition module, an acceleration signal sensing module and a Bluetooth module. In the surface electromyography signal acquisition module, P3 is used to connect the hydrogel flexible electrode, and the electrode is attached to the human skin to sense the double-channel electromyography signal.

[0093] To test the signal acquisition performance of the entire acquisition circuit, the double-channel surface electromyography signals and acceleration signals of 9 kinds of sign language were collected, as shown in Figure 7 The collected different sign language signals have high signal quality, and the electromyography signals and acceleration signals corresponding to each sign language have large differences. These differences can be used to construct high-performance signal preprocessing algorithms and AI models to complete the classification of gesture signals. DETAILED DESCRIPTION

[0095] Figure 8The workflow diagram of the entire system is shown. First, a dataset is constructed by performing different gestures. The dataset is then preprocessed using smoothed wavelet packet transform (SWPT) and input into the AI ​​model for training. After the model is trained, it can be deployed by a mobile application. When the user performs a gesture, the dual-channel sEMG and acceleration signals are transmitted to the mobile application via Bluetooth. The SWPT preprocessing algorithm and CBAM-LSTM-CNN neural network model deployed in the app process and recognize the dual-channel electromyographic signal and acceleration signal data of the input gesture action, and realize the completion of multiple functional applications including sign language recognition function, motion monitoring function, active rehabilitation function and mechanical control function through different function pages. Figure 8 The three controls in the lower part of the middle interface are the sign language recognition control, mechanical control control and rehabilitation control in this application. Figure 9 The mobile APP functional interface of the mobile human-computer interaction system is demonstrated. The APP can establish a stable Bluetooth connection with the external acquisition device to realize data reception and transmission, and integrates three human-computer interaction functions including sign language recognition function, motion monitoring function, active rehabilitation function and mechanical control function, etc. A variety of human-computer exchange applications. When the APP is connected to the input device and output device, the real-time operation of the entire system can be completed. After the APP recognizes the dual-channel sEMG signal and acceleration signal transmitted by the acquisition input device, the gesture signal is pre-processed by the SWPT preprocessing algorithm, and then the CBAM-LSTM-CNN neural network model is called to recognize the data. After the gesture signal is recognized, the gesture photo will be displayed on the corresponding function page to show the corresponding gesture action, and the control command signal will be sent through Bluetooth to complete various human-computer interaction applications. Figure 10 (a) shows that the mobile interactive system can accurately and in real time complete the high-accuracy sign language recognition function. When the user wears the acquisition device and performs sign language movements, the sign language recognition function page can display the corresponding sign language pictures and texts and play the voice of the sign language. Figure 10 (b) shows that the human-computer interaction system can complete stable arm motion data monitoring in outdoor scenarios. The motion monitoring function page realizes the waveforms of dual-channel electromyographic signals and acceleration signals in real time. Figure 10 (c) shows that the mobile human-computer interaction system can stably implement sign language recognition applications in outdoor environments, and shows that the system can accurately recognize sign language in complex environments.

[0096] Figure 11 (a) in the middle is the gesture action signal collected by the acquisition device when the human-computer interaction system controls the robotic arm in real time. The actions are grasping, lifting and placing. Figure 11The middle(b) shows that the mobile human-computer interaction system can control the robot arm to complete the task of grabbing dangerous objects in real time. When the user performs the actions of grabbing, lifting and placing, the mobile APP program receives the bimodal data and recognizes the gesture actions made by the user. The APP displays the gesture photo of the action instruction made by the user and sends the action instruction signal through Bluetooth. The robot arm has a motion program burned in it. After receiving the instruction signal, the robot arm executes the corresponding action. The user does not need to operate the mobile phone again. The robot arm can complete the corresponding action task according to the instruction sent by the APP, and then complete the whole grabbing and placing task.

[0097] Figure 11 The middle(c) shows the expansion performance of the mobile human-computer interaction system, which can complete the control of various mechanical devices, such as active rehabilitation application. Stroke patients need rehabilitation training, but the efficiency of rehabilitation training is low due to physical reasons. Therefore, rehabilitation gloves can be used to assist users to complete rehabilitation training efficiently. The rehabilitation gloves have a hand opening and fist closing action program burned in them. When the user wears the bimodal acquisition wristband and the automatic rehabilitation assisting glove, the user performs the hand opening and fist closing action at will. The mobile APP can complete the recognition of the intention in real time by receiving the intention signal collected by the acquisition wristband. After the recognition is completed, the APP sends the instruction signal corresponding to the gesture. After receiving the instruction signal, the rehabilitation glove completes the rehabilitation action corresponding to the instruction signal, thereby helping the patient to complete efficient rehabilitation training. The rehabilitation action can be determined by the user. After determining the action, the data set corresponding to the action is constructed and the action programming of the data glove is completed, thereby completing the active rehabilitation application of various actions. Moreover, the mobile system does not depend on external network environment. The user can complete efficient and accurate active rehabilitation application in various complex environments through the mobile application.

[0098] Therefore, the mobile human-computer interaction system has high real-time performance. The integration of various interactive applications ensures that the system can play an important role in various application scenarios and is not affected by external network environment. The system can still realize accurate interactive application in complex environment. Moreover, the system has excellent expansion performance and can control various output devices.

[0099] The application provides an implementation method of a conformal biocompatible flexible hydrogel electrode. A high-performance hydrogel skin electrode is prepared by adding 30% volume ratio of honey and 2M sodium chloride in the hydrogel. The shear stress of the hydrogel skin electrode with the skin is up to 4kPa, the modulus is 2kPa, the conductivity is up to 1.6Sm-1, the electromyographic signal with a signal-to-noise ratio of up to 20dB can be collected, the hydrogel skin electrode has excellent biocompatibility and antibacterial performance, and the biological safety of the electrode in long-term application is ensured.

[0100] The application provides a multifunctional mobile man-machine interaction system, which is connected with a double-mode signal acquisition device through a Bluetooth connection signal, realizes wireless transmission of signals, deploys a data SWPT preprocessing algorithm and a CBAM-LSTM-CNN neural network model in an APP, and can realize gesture recognition application with low delay and high accuracy. The APP adopts a Fragment layout to integrate various man-machine interaction application pages, including sign language recognition, motion monitoring and mechanical control. The connection acquisition device can complete real-time sign language recognition, motion monitoring, mechanical arm control and active rehabilitation and various man-machine interaction applications. The device relies on Bluetooth to complete data transmission, is less affected by external environment, can complete stable and accurate man-machine interaction application in a complex environment, has excellent expansion performance, and can realize control of various output devices.

[0101] The application provides a double-mode signal real-time acquisition wristband device. The acquisition wristband device is designed with an FPCB and has excellent flexibility, can conform to the surface of human skin to realize comfortable wearing, has a small volume, only 16 cm2 of the area of the circuit part, and a total mass of 8 g, and is made into the shape of a wristband to realize convenient carrying of the device. The circuit integrates a KS1082 electromyographic signal acquisition chip and an ADXL345 acceleration sensor, can realize synchronous acquisition of surface double-channel sEMG signals and three-axis acceleration signals by integrating a hydrogel flexible electrode and attaching the electrode to a human body, transmits the acquired signals to an STM32 microprocessor to complete signal processing and data frame encapsulation, and sends the data frame to a Bluetooth module through a serial port to realize wireless data transmission with external devices.

[0102] Compared with reported hydrogel electrodes, the hydrogel electrode has more excellent flexibility, adhesion, conductivity, can stably conform to the surface of a human body and maintain low interface impedance, can acquire surface electromyographic signals with high signal-to-noise ratio, and simultaneously meets excellent biocompatibility and antibacterial performance, and guarantees biological safety in long-term use. Compared with reported man-machine interaction systems, the system has excellent portability and real-time performance. The acquisition device in the system can be comfortably and conveniently worn on the arm to complete data acquisition and transmission, data is only transmitted to the APP application end of a mobile phone through wireless Bluetooth, the APP is built-in with a high-performance SWPT data preprocessing algorithm and a CBAM-LSTM-CNN neural network model, excellent real-time performance can be realized on the premise of guaranteeing high recognition accuracy, the processing delay is less than 200 ms, various man-machine interaction functions are integrated, various needs of users in different scenes can be met, the device is not affected by external network environment, stable and accurate interaction application can be completed in a complex environment, and the system has high practicability and expandability.

[0103] An embodiment can realize a hydrogel electrode by adding MXene and tannic acid in the hydrogel, but the MXene is easy to be oxidized, the conductivity of the prepared hydrogel electrode is low, the high signal-to-noise ratio electromyographic signal cannot be sensed, the preparation method is complex, and the cost is high. In terms of system implementation, the algorithm and model can be deployed by using the cloud server method, but the deployment in the cloud server will cause the delay of data transmission and processing, and the cloud server is limited by network factors and cannot complete stable application in a complex environment, and the cost of the cloud server is also high, which cannot meet the demand of multi-functional human-computer interaction application.

[0104] It can be seen that the contents in the method embodiments are applicable to the system embodiments, the system embodiments specifically realize the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0105] On the other hand, the embodiment of the present application provides a human-computer interaction device based on a hydrogel flexible electrode, comprising:

[0106] At least one processor;

[0107] At least one memory for storing at least one program;

[0108] When the at least one program is executed by the at least one processor, the at least one processor realizes the human-computer interaction method based on the hydrogel flexible electrode.

[0109] Similarly, the contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically realize the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0110] The embodiment of the present application further provides a computer readable storage medium, wherein a program executable by a processor is stored, and the program executable by the processor is used to execute the human-computer interaction method based on the hydrogel flexible electrode when executed by the processor.

[0111] Similarly, the contents in the method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically realize the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0112] In some alternative embodiments, the function / operations mentioned in the block diagrams can not occur in the order mentioned in the operational illustrations. For example, depending on the involved function / operation, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in reverse order, depending upon the functionality / operations involved. Furthermore, embodiments presented and described in the flowcharts are only examples of implementing the present application. Alternative embodiments are possible where some of the steps are omitted, wherein additional steps are added, or wherein some of the steps are performed in a different order. Additionally, the disclosed methods are not limited to the order of operations presented in the flowcharts.

[0113] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine skill of engineers familiar with the attributes, functions, and internal relationships of the various functional modules disclosed herein. Accordingly, the present application is not limited to the details of the implementation as set forth in the following description. Further, the disclosed subject matter is intended to be illustrative, and not limiting, of the scope of the present application, as defined by the appended claims, the full scope of which is to be determined by the full breadth of equivalents, of which the claims attached hereto arc entitled to.

[0114] The functions described can be implemented in software, firmware, or hardware. The functions can be created in or transferred to a computer-readable storage medium (or media) without unduly compromising the security of the information belonging to the users and the providers. The orders of the operations described are merely examples and the actual implementation can be different from those described. The actual implementation is within the routine skill of engineers familiar with the attributes, functions, and internal relationships of the various functional modules disclosed herein. Accordingly, the present application is not limited to the details of the implementation as set forth in the following description. Further, the disclosed subject matter is intended to be illustrative, and not limiting, of the scope of the present application, as defined by the appended claims, the full scope of which is to be determined by the full breadth of equivalents, of which the claims attached hereto arc entitled to.

[0115] The logic and / or steps represented in a flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, a program execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can retrieve and execute a program from a program execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, a program execution system, apparatus, or device.

[0116] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0117] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0119] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be taken as limiting the scope of the application. The scope of the application is defined by the claims and their equivalents.

[0120] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the described embodiment, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A human-computer interaction method based on hydrogel flexible electrodes, characterized in that: The following steps are involved: In response to a trigger operation on the first control received on the functional interface, collecting acceleration signals and multi-channel electromyographic signals of the target object through the hydrogel flexible electrode; The first control is a target function implemented based on the intention information of the target object; Inputting the acceleration signal and the electromyographic signal into an intention prediction model to obtain the intention information; According to the intention information and based on the first control, an output signal of the first terminal is determined so that the target device performs the target function; the functional interface is the interface of the first terminal.

2. The human-computer interaction method based on hydrogel flexible electrodes according to claim 1, characterized in that: The first control includes a sign language recognition control, and determining, according to the intention information and based on the first control, an output signal of the first terminal so that the target device performs the target function includes: According to the intention information, based on the sign language recognition control, determining that the first terminal outputs a display instruction so that the target device emits a voice message or a text message; the voice message and the text message correspond to the intention information; The first control includes a mechanical control control, and the target device includes a robot; determining, according to the intention information and based on the mechanical control control, an output signal of the first terminal so that the target device performs the target function, includes: According to the intention information, based on the mechanical control component, it is determined that the first terminal outputs a motion instruction so that the robot arm moves according to the motion instruction.

3. The human-computer interaction method based on hydrogel flexible electrodes according to claim 2, characterized in that: The first control includes a rehabilitation control, and the target device includes a rehabilitation glove; The determining, according to the intention information and based on the first control, an output signal of the first terminal so that the target device performs the target function includes: According to the intention information and based on the rehabilitation control, determining that the first terminal outputs an action instruction to the rehabilitation glove; Action programming is performed according to the action instructions to control the rehabilitation gloves to perform rehabilitation actions.

4. The human-computer interaction method based on hydrogel flexible electrodes according to claim 1, characterized in that: The hydrogel flexible electrode is prepared by the following steps: acrylamide, a cross-linking agent, a photoinitiator, and sodium chloride are mixed in a predetermined ratio and added to a deionized water mixed solution to obtain a first mixed solution; wherein the deionized water mixed solution is obtained by mixing honey and deionized water; The first mixed solution is injected into a mold and subjected to light irradiation treatment to obtain a hydrogel flexible electrode.

5. The human-computer interaction method based on hydrogel flexible electrodes according to claim 1, characterized in that: The intention prediction model is trained through the following steps: Obtain acceleration signal samples and multi-channel electromyographic signal samples; Performing smoothed wavelet packet transform processing on the acceleration signal samples and the electromyographic signal samples, and inputting the processed feature samples into the intention prediction model to obtain predicted intention samples; A loss function is determined based on the predicted intent sample and the true intent, and the intent prediction model is trained based on the loss function to obtain a trained intent prediction model.

6. A human-computer interaction system based on hydrogel flexible electrodes, characterized in that: include: An acquisition module, a control module, a first terminal, and a target device; The acquisition module is used to: in response to a trigger operation on the first control received on the functional interface, acquire acceleration signals and multi-channel electromyographic signals of the target object through the hydrogel flexible electrode; The first control is a target function implemented based on the intention information of the target object; The control module is used to: input the acceleration signal and the electromyographic signal into the intention prediction model to obtain the intention information; According to the intention information and based on the first control, an output signal of the first terminal is determined so that the target device performs the target function; the functional interface is the interface of the first terminal.

7. The human-computer interaction system based on hydrogel flexible electrodes according to claim 6, characterized in that: The acquisition module includes: An electromyographic signal acquisition module and an acceleration acquisition module, wherein the hydrogel flexible electrode is integrated on the electromyographic signal acquisition module.

8. The human-computer interaction system based on hydrogel flexible electrodes according to claim 7, characterized in that: The electromyographic signal acquisition module includes a KS1082 chip, and the acceleration acquisition module includes an ADXL345 chip; The control module is used to configure the opening channel of the KS1082 chip and collect the electromyographic signal of the KS1082 chip through the ADC; The acceleration signal of the ADXL345 chip is transmitted to the control module through the SDA port and the SCL port.

9. A human-computer interaction device based on hydrogel flexible electrodes, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the human-computer interaction method based on the hydrogel flexible electrode according to any one of claims 1 to 5.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the human-computer interaction method based on hydrogel flexible electrodes as claimed in any one of claims 1 to 5 when executed by the processor.

Images