Preparation method of ionic conductive hydrogel multi-mode sensor and sign language recognition system

Abstract

The invention discloses a preparation method of an ionic conductive hydrogel multi-mode sensor and a sign language recognition system. The preparation method comprises the following steps: preparing a prepolymer solution containing an anionic monomer and a first photoinitiator, and carrying out ultraviolet polymerization, drying, ball milling and sieving to obtain polyanion electrolyte powder; dissolving the powder in water, selectively adding an acrylamide monomer, a cross-linking agent, a nano reinforcing filler, an ionic conductive agent and a water-retaining agent to form a first mixed solution and a second mixed solution, and respectively adding a cationic monomer and a second photoinitiator to prepare first ink adaptive to strain sensing and pressure sensing and second ink adaptive to myoelectricity sensing; two inks are printed to the flexible substrate in a 3D mode and subjected to ultraviolet curing, a strain sensor, a pressure sensor and an electromyographic sensor are manufactured, and the multi-mode sensor is jointly formed by the strain sensor, the pressure sensor and the electromyographic sensor. The sign language recognition system comprises a wearable module integrated with the sensor, a hardware module and a software module, sign language gestures can be recognized in real time with high precision, and communication of people with hearing impairment is assisted.

Description

Technical Field

[0001] This invention relates to the field of flexible electronics and intelligent sensing technology, specifically to a method for fabricating an ion-conductive hydrogel multimodal sensor and a sign language recognition system. Background Technology

[0002] Sign language is the primary means of communication for deaf and mute individuals. Developing efficient and accurate sign language recognition technology is of great significance for eliminating communication barriers and promoting social integration among people with hearing impairments. Data from the World Health Organization shows that more than 1.5 billion people worldwide suffer from hearing loss, and this number is projected to grow to approximately 2.5 billion by 2050. However, professional sign language interpreters are both expensive and scarce. Therefore, developing automated sign language recognition systems has become a key technological approach to solving this social problem.

[0003] In recent years, with the rapid development of artificial intelligence and flexible electronics technology, two mainstream technical routes have emerged in the field of sign language recognition: computer vision-based methods and wearable sensor-based methods. Computer vision-based methods rely on cameras to capture gesture images or video sequences, using deep learning algorithms for feature extraction and classification. While offering advantages such as being contactless and convenient, their recognition performance is highly susceptible to factors like image quality, ambient lighting changes, background interference, and gesture occlusion. They lack robustness in complex real-world scenarios and struggle to meet the core requirements of real-time performance and high accuracy in everyday applications. Wearable sensor-based methods, on the other hand, directly collect physiological or motion signals through worn devices. They offer significant advantages such as compact size, continuous and accurate motion capture, and minimal susceptibility to environmental interference, and are considered a more promising technology for practical application. Currently, mainstream sensing mechanisms include strain sensing, pressure sensing, surface electromyography (sEMG) sensing, and inertial measurement unit (IMU) sensing.

[0004] Despite some progress in wearable sensing solutions, several challenges remain in practical applications, severely hindering their large-scale adoption in sign language recognition systems: First, there is the challenge of material and integration compatibility. Most existing wearable systems use heterogeneous materials to construct sensor units, such as metal strain gauges, carbon-based materials, liquid metals, and commercially available Ag / AgCl electrodes. These materials have inherent differences in mechanical properties (such as modulus and ductility), interface stability, and fabrication processes. This not only makes it difficult for integrated systems to balance structural consistency, wearability, and long-term signal stability, but also increases manufacturing costs due to the complex modular fabrication and assembly processes, hindering the practical application of the technology.

[0005] Second, the limited sensor dimensions lead to information gaps. Natural sign language is a complex sequence of movements that includes multi-dimensional information such as finger posture, joint angles, contact force, and muscle activity. However, most existing studies focus on acquiring only a single type of signal (such as detecting only finger bending). Single-modal sensors cannot fully capture the key features of sign language, easily leading to the omission of semantic information, greatly limiting the range of recognizable words and the accuracy of system recognition, making it difficult to meet the needs of complex sign language expressions.

[0006] Third, the shortcomings of core material performance. Hydrogel materials are considered ideal candidate materials for next-generation integrated flexible sensors due to their excellent flexibility, biocompatibility and tunable conductivity. However, traditional hydrogels have two major drawbacks: First, high mechanical hysteresis and signal drift. During cyclic loading and unloading, irreversible chain slippage and energy dissipation in the polymer network will generate significant residual strain, resulting in waveform distortion, baseline drift and poor response repeatability of the sensing signal, which seriously affects signal fidelity and recognition accuracy. Second, poor environmental stability (easy to dehydrate). The continuous evaporation of water in the hydrogel will cause the material to dry and harden, the conductivity to decrease and the mechanical properties to degrade. It usually fails after a few hours or days and cannot meet the requirements of long-term wear.

[0007] In summary, existing technologies struggle to simultaneously address three major challenges: material integration compatibility, multi-dimensional signal acquisition, and stable core material performance, thus hindering the practical application of sign language recognition systems. There is an urgent need in this field for novel materials and integration solutions. These materials must possess low hysteresis, high resilience, excellent dehydration resistance, and adjustable electromechanical properties, enabling them to be fabricated into various sensors such as strain, pressure, and sEMG electrodes and seamlessly integrated to synchronously and with high fidelity acquire multimodal gesture information. Therefore, developing a method for fabricating an ion-conductive hydrogel multimodal sensor and a sign language recognition system adapted to the needs of sign language recognition has become a pressing technical problem to be solved. Summary of the Invention

[0008] To address the technical problems existing in the prior art, the first objective of this invention is to provide a method for preparing an ion-conductive hydrogel multimodal sensor. This method involves preparing polyanionic electrolyte powder, preparing two mixed solutions and corresponding inks, and then 3D printing and curing them to form strain, pressure, and electromyography sensors, which together constitute a multimodal sensor.

[0009] The second objective of this invention is to provide a sign language recognition system that integrates the aforementioned sensors into a data glove and a flexible armband, combined with data acquisition and transmission hardware and dual-branch model software, to achieve high-precision real-time recognition of sign language gestures, thereby assisting people with hearing impairments in communication.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A method for fabricating an ion-conductive hydrogel multimodal sensor includes the following steps: A prepolymer solution containing anionic monomer and a first photoinitiator was prepared, and polymerized by ultraviolet irradiation to form a prehydrogel. The prehydrogel was then dried, ball-milled, and sieved to obtain polyanionic electrolyte powder. The polyanionic electrolyte powder is dissolved in water to obtain a polyanionic electrolyte aqueous solution. Acrylamide monomers, crosslinking agents, nano-reinforcing fillers, ionic conductive agents, and water-retaining agents are added to the solution, and the mixture is stirred and dispersed uniformly to form a first mixture. A cationic monomer and a second photoinitiator are added to the first mixture, and the reaction yields a homogeneous, printable first ionic conductive hydrogel ink. This first ionic conductive hydrogel ink is used for strain sensing or pressure sensing applications. The polyanionic electrolyte powder is dissolved in water to obtain a polyanionic electrolyte aqueous solution. Acrylamide monomers, nano-reinforcing fillers, ionic conductive agents and water-retaining agents are added to the solution and mixed evenly to form a second mixture. Cationic monomers and a second photoinitiator are added to the second mixture and reacted to obtain a homogeneous printable second ionic conductive hydrogel ink. The second ionic conductive hydrogel ink is used for electromyography. The first and second ion-conductive hydrogel inks were printed onto a flexible substrate using 3D printing technology and cured by ultraviolet light. Strain sensor, pressure sensor and electromyography sensor were prepared accordingly, which together constitute an ion-conductive hydrogel multimodal sensor.

[0011] According to one example, when preparing the prepolymer solution, deionized water is used as the solvent to thoroughly stir and dissolve the anionic monomer and the first photoinitiator. The concentration of the anionic monomer in the prepolymer solution is 0.8-1.2M, and the amount of the first photoinitiator added is 0.03-0.07% of the total molar amount of the prepolymer solution. During ultraviolet irradiation polymerization, the prepolymer solution is injected into the reaction chamber, the irradiation wavelength is 405nm, and the polymerization time is 10 hours. The pre-hydrogel is dried by heating in an oven at 60-70℃ for 20-28 hours, ball milling is performed at a speed of 300-350rpm for 2-4 times, and sieving is done using a 300-mesh sieve.

[0012] According to one example, the anionic monomer is sodium styrene sulfonate; the polyanionic electrolyte is poly(4-styrene sulfonate); the cationic monomer is [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt; the first photoinitiator is α-ketoglutaric acid; and the second photoinitiator is lithium phenyl-2,4,6-trimethylbenzoyl phosphate.

[0013] According to one example, the acrylamide monomer is acrylamide, the crosslinking agent is N,N'-methylenebisacrylamide, and the nano-reinforcing filler is a TpPa-1 type covalent organic framework nanosheet. The surface of the TpPa-1 type covalent organic framework nanosheet includes hydroxyl, carbonyl, and imine groups, which can bond with the polyacrylamide network formed by acrylamide polymerization through hydrogen bonding to inhibit irreversible slippage of polymer chains under external forces and reduce the mechanical hysteresis of the ion-conductive hydrogel.

[0014] According to one example, the ionic conductive agent is lithium chloride, and the water-retaining agent is glycerol; when mixing and dispersing to form the first mixture or the second mixture, the stirring and ultrasonic dispersion are carried out simultaneously. First, the raw materials are initially mixed by stirring, and then the raw material agglomerates are broken by ultrasonic dispersion; after adding the cationic monomer and the second photoinitiator to the first mixture or the second mixture, the mixture is heated and stirred for 1-3 hours under a water bath at 65-75°C to allow the system to fully react and form a homogeneous first ionic conductive hydrogel ink or second ionic conductive hydrogel ink.

[0015] According to one example, the 3D printing technology is direct-write 3D printing; during the ultraviolet light curing, the irradiation wavelength is 405nm, the light intensity is 10-20mW / cm², and the curing time is 1.5-2.5 hours.

[0016] According to one example, the flexible substrate is a flexible printed circuit board, and the flexible printed circuit board is provided with a pair of copper electrodes corresponding to a preset sensing area. The first ion-conductive hydrogel ink or the second ion-conductive hydrogel ink is printed between the two copper electrodes respectively.

[0017] A sign language recognition system, comprising: A wearable module comprising a data glove and a flexible armband, and a sensor obtained by the fabrication method of an ion-conductive hydrogel multimodal sensor as described in any one of claims 1 to 7, the sensor comprising a plurality of strain sensors, a plurality of pressure sensors and a plurality of electromyography sensors, the plurality of strain sensors and the plurality of pressure sensors being integrated on the data glove, and the plurality of electromyography sensors being integrated on a multichannel differential surface electromyography electrode array of the flexible armband; The hardware module includes a multi-channel data acquisition circuit and a Wi-Fi module. The multi-channel data acquisition circuit is used to convert the analog signals acquired by the wearable module into digital signals, and the Wi-Fi module wirelessly transmits the digital signals to the cloud server through a real-time bidirectional communication protocol. The software module includes a bidirectional long short-term memory network model deployed on the cloud server and a user interaction terminal; the bidirectional long short-term memory network model extracts and classifies features of digital signals and outputs sign language recognition results; the user interaction terminal receives the recognition results and displays them in the form of text or synthesized speech.

[0018] According to one example, the plurality of strain sensors are arranged at the corresponding finger joints, palm, and wrist of the data glove to monitor hand joint flexion and palm deformation; the plurality of pressure sensors are arranged at the corresponding fingertips of the data glove to monitor fingertip contact pressure; the electromyography sensor is a multi-channel differential surface electromyography electrode array, fitted to the forearm skin, to collect muscle electrical signals during hand movement; the multi-channel data acquisition circuit includes an ADC chip and a gain amplifier to improve signal conversion accuracy and signal strength.

[0019] According to one example, in the bidirectional long short-term memory network model, one branch is used to process the signals collected by the strain sensor and the pressure sensor, and the other branch is used to process the signals collected by the electromyography sensor. The features extracted by the two branches are fused and the gesture recognition result is output through a classifier. The wearable module simultaneously collects strain signals, pressure signals and surface electromyography signals. The collected raw signals are filtered, normalized and preprocessed by segmentation before being input into the bidirectional long short-term memory network model.

[0020] The present invention has the following advantages: This invention designs and prepares an ionic hydrogel (PIG) ​​based on a polyamplifier / polyacrylamide biorthogonal crosslinking network. Its core lies in using covalent organic framework (COF) nanosheets as multifunctional physical crosslinking points to form strong hydrogen bonds with the polyanionic electrolyte (PNaSS) and polyacrylamide (PAAm) networks, effectively suppressing irreversible polymer chain slippage and achieving low mechanical hysteresis (90.25% recovery rate) and high elasticity. By simply adjusting the concentrations of the covalent crosslinking agent (MBAA) and monomer (AAM), a PIG suitable for high tensile strain sensing can be directionally optimized within the same material system. 0.075-5 (550% elongation at break) or low modulus sEMG sensing PIG 0-5 (Modulus 0.09MPa) formulation. From the molecular design level, COF nanosheets, a rigid nano-reinforcement, are introduced to solve the energy dissipation problem of dynamic networks from a mechanistic perspective, resulting in better low hysteresis performance; at the same time, a clear and universal performance regulation strategy (adjusting crosslink density or polymer chain fraction) is provided, which can customize materials for different sensing needs, making the materials more versatile and designable.

[0021] Leveraging the shear-thinning properties of PIG ink, direct-write 3D printing (DIW) technology unifies the fabrication of strain / pressure sensors and sEMG electrodes on a single printing platform. Customized sensor patterns can be directly printed on flexible printed circuit boards (FPCBs), achieving precise alignment and integrated integration with conductive circuitry. This results in the manufacture of a high-density sensing glove containing 12 strain sensors and 5 pressure sensors. As an additive manufacturing process, 3D printing offers advantages such as mold-free operation, rapid prototyping, and high design freedom, easily enabling the fabrication of complex structures and customized sensors with higher integration and efficiency. In contrast, existing multi-step composite molding processes combining electrospinning and multi-layer assembly are complex and difficult to control in terms of alignment accuracy, making them more suitable for fabricating sheet-like sensing units rather than integrated systems.

[0022] A multimodal data acquisition system with 54 sensing channels (34 strain / pressure channels + 20 sEMG channels) was constructed. A dual-branch Bi-LSTM neural network model was employed, with different network structures (e.g., 128 units and 512 units) designed for low-frequency strain / pressure signals and high-frequency sEMG signals for feature extraction. Subsequent deep fusion at the feature layer resulted in a 99.65% recognition accuracy for 24 Chinese sign language gestures. This system represents a dual upgrade in terms of information dimension and algorithm architecture: firstly, it features more sensing channels and more comprehensive information, expanding from 7 pressure channels to 54 strain / pressure / sEMG channels, capturing more detailed sign language features; secondly, the algorithm is more advanced. Bi-LSTM, as a recurrent neural network designed specifically for time-series signals, better understands the spatiotemporal dynamics of gesture movements, naturally exhibiting superior recognition accuracy and robustness compared to traditional BPNN models.

[0023] This invention provides a single hydrogel material platform (PIG) ​​that combines tunable mechanical properties with stable ionic conductivity. Through a biorthogonal crosslinking network design (synergistic use of a polyampholyte network and a covalently crosslinked polyacrylamide framework), the single material can be optimized by adjusting the crosslinking density or polymer chain fraction to create formulations suitable for strain / pressure sensing (high ductility, low hysteresis) and sEMG sensing (low modulus, high conductivity), respectively. 0.075-5 With PIG 0-5This approach fundamentally solves the problems of mechanical / electrical performance mismatch, interface instability, and complex fabrication processes caused by the mismatch of intrinsic material properties in multi-sensor systems, laying the material foundation for highly integrated wearable systems. Simultaneously, a molecular-level design strategy significantly improves the material's response reliability and environmental stability: COF nanosheets are introduced as physical cross-linking points, their rigid porous structure forming strong hydrogen bonds with polymer chains, achieving a recovery rate of up to 90.25%, overcoming the mechanical hysteresis problem of traditional hydrogels under cyclic loading; glycerol (hydrogen-bonded water locking) and LiCl (hygroscopic) are introduced to construct an efficient dehydration prevention mechanism, enabling the material to maintain stable electromechanical properties for 14 days, solving the performance degradation problem of traditional hydrogels caused by water loss.

[0024] To address the limitations of existing solutions, which suffer from single sensing modalities or sparse nodes and are unable to capture key features of complex sign language, this invention constructs a multimodal, high spatial resolution sensing system. First, it enables multimodal perception by simultaneously integrating 12 strain sensors (monitoring finger joints, wrist joints, and palm flexion), 5 pressure sensors (monitoring fingertip pressure), and a 10-channel differential sEMG electrode array (monitoring forearm muscle activity) on a single material platform. This achieves simultaneous, high-fidelity acquisition of four types of gesture features: finger flexion, wrist movement, fingertip touch, and palm interaction. Second, it enables information fusion and intelligent recognition by deeply fusing and extracting features from heterogeneous multimodal time-series data using a Bi-LSTM model, achieving a classification accuracy of 99.65% for 24 Chinese sign language gestures. Attached Figure Description

[0025] Figure 1 This is a flowchart of a method for preparing an ion-conductive hydrogel multimodal sensor according to the present invention.

[0026] Figure 2 These are characterization figures of the performance and printing properties of the ion-conducting hydrogel multimodal sensor of the present invention. Figure a shows the Raman spectrum of the ion-conducting hydrogel, characterizing its molecular functional group (-SO3). - Figure a) shows the distribution of -C=N, -C=O, and -NH2; Figure b) shows the infrared spectrum of the ion-conducting hydrogel, further verifying the molecular functional group structure; Figure c) shows the viscosity-shear rate rheological curves of the ion-conducting hydrogel at different acrylamide (AAM) concentrations; Figure d) shows the shear modulus-oscillatory strain rheological curves of the ion-conducting hydrogel at different AAM concentrations; Figure e) shows the microscopic morphology comparison of the ion-conducting hydrogel ink before and after curing (including two scales: 100 μm and 200 μm); Figure f) shows the optical photographs of actual printed samples of the ion-conducting hydrogel (including spiral and linear structures); Figure g) shows the printing pressure (P) and printing speed (V) at different times. pThe printing performance partition diagram below, combined with the typical morphologies on the right (i) poor shape fidelity, (ii) printable, and (iii) unprintable; Figure h shows the curves of printing line width (d) variation under different printing pressures and printing speeds.

[0027] Figure 3 These are the mechanical property characterization figures of the ion-conducting hydrogel multimodal sensor of the present invention. Figure a shows the tensile stress-strain curves of the ion-conducting hydrogel with different crosslinking agent (MBAA) contents; Figure b shows the tensile stress-strain curves of the ion-conducting hydrogel with different acrylamide (AAM) concentrations; Figure c shows the compressive stress-strain curves of the ion-conducting hydrogel with different crosslinking agent (MBAA) contents; Figure d shows the compressive stress-strain curves of the ion-conducting hydrogel with different acrylamide (AAM) concentrations; Figure e shows the loading-unloading stress-strain curves of the preferred formulation (AAM 5 mol / L, MBAA 0.075 wt%) under different tensile strains (ε=1-5), used for hysteresis analysis; Figure f shows the loading-unloading stress-strain curves of the preferred formulation (AAM 5 mol / L, MBAA 0.075 wt%) under different tensile strains (ε=1-5), used for hysteresis analysis; Figure g shows the cyclic loading-unloading stress-strain curves of the preferred formulation under different compressive strains (ε=0.2-0.6) for hysteresis analysis; Figure h shows the cyclic loading-unloading stress-strain curves of the preferred formulation under tensile strain of 200% (cycles 1st-100th) for fatigue testing; Figure i shows the actual tensile process and recovery of the preferred formulation (tensile strain 550%); Figure j shows the actual compression process and recovery of the preferred formulation (compressive strain 60%).

[0028] Figure 4 These are hysteresis analysis diagrams of the ion-conducting hydrogel of the present invention under different formulations. The hysteresis characteristics are quantified by the area ratio of the unloading curve to the loading curve. Figure a shows the tensile loading-unloading stress-strain curves of the ion-conducting hydrogel under different crosslinking agent (MBAA) contents; Figure b shows the tensile loading-unloading stress-strain curves of the ion-conducting hydrogel under different acrylamide (AAM) concentrations; Figure c shows the compression loading-unloading stress-strain curves of the ion-conducting hydrogel under different crosslinking agent (MBAA) contents; and Figure d shows the compression loading-unloading stress-strain curves of the ion-conducting hydrogel under different acrylamide (AAM) concentrations.

[0029] Figure 5These are quantitative analysis graphs of the recovery rate and energy loss of the ion-conducting hydrogel under different strains. Graph a shows the tensile strain-recovery rate curves of the ion-conducting hydrogel under different crosslinking agent (MBAA) contents (the recovery rate is calculated by the area ratio of the unloading curve to the loading curve); Graph b shows the tensile strain-recovery rate curves of the ion-conducting hydrogel under different acrylamide (AAM) concentrations; Graph c shows the compressive strain-recovery rate curves of the ion-conducting hydrogel under different crosslinking agent (MBAA) contents; Graph d shows the compressive strain-recovery rate curves of the ion-conducting hydrogel under different acrylamide (AAM) concentrations. Figure e shows the tensile strain-hysteresis loop area curves of ion-conducting hydrogels with different crosslinking agent (MBAA) contents (energy loss is calculated by the area difference between the loading curve and the unloading curve); Figure f shows the tensile strain-hysteresis loop area curves of ion-conducting hydrogels with different acrylamide (AAM) concentrations; Figure g shows the compressive strain-hysteresis loop area curves of ion-conducting hydrogels with different crosslinking agent (MBAA) contents; Figure h shows the compressive strain-hysteresis loop area curves of ion-conducting hydrogels with different acrylamide (AAM) concentrations.

[0030] Figure 6 These are front and back images of the data glove assembled with the ion-conductive hydrogel multimodal sensor prepared in this invention and a wearable module.

[0031] Figure 7 This is a physical image of the wearable module of the present invention when worn.

[0032] Figure 8 This is a structural diagram of the FPCB with a serpentine structure for the data glove of the wearable module of this invention.

[0033] Figure 9 This is a schematic diagram of the fabrication process of the multi-channel differential surface electromyography electrode array of the present invention.

[0034] Figure 10 This is a schematic diagram of the sign language recognition system of the present invention.

[0035] Figure 11 This is a schematic diagram of the workflow of the sign language recognition system of the present invention.

[0036] Figure 12 This is a diagram of the WeChat mini-program interface of the user interaction terminal of this invention.

[0037] Figure 13 This is a flowchart of the sign language recognition method of the present invention.

[0038] Figure 14 This is a schematic diagram of the model architecture of the sign language recognition method of the present invention.

[0039] Figure 15Figure 1 shows the loss value and accuracy curves during the training process of the model of this invention. Figure 2a shows the loss value change curves of the training set and the validation set, and Figure 3b shows the accuracy change curves of the training set and the validation set. These figures visually demonstrate the stable state in which the loss value of the model gradually converges and the accuracy continuously improves during the training process.

[0040] Figure 16 The diagrams show the clustering graph, confusion matrix, and actual demonstration diagram of this invention. Graph d is the t-SNE clustering graph, which intuitively presents the feature distribution and category discrimination of 24 Chinese sign language gestures. Graph e is the confusion matrix, showing that the system achieves a classification accuracy of 99.65% for the 24 sign language gestures, and the confusion of each category is clearly distinguishable. Graph f is the actual demonstration diagram, showing the system's recognition and translation effects of the sign languages ​​South, China, University, and Technology in a real-world scenario. Detailed Implementation

[0041] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0042] This application primarily addresses the key technical problems existing in sign language recognition systems, such as the limited performance of sensing materials, poor integration compatibility, and limited signal acquisition dimensions. Specifically, current recognition systems based on flexible sensors mostly rely on heterogeneous materials to construct discrete sensor units, leading to problems such as poor material compatibility and complex fabrication processes in integrated systems. Meanwhile, traditional hydrogel materials suffer from high mechanical hysteresis, easy water evaporation, and insufficient electrical stability, severely limiting their application in long-term, high-precision gesture monitoring. Furthermore, most existing sensors can only capture single-modal signals, making it difficult to simultaneously achieve the collaborative perception and fusion recognition of multi-dimensional gesture features such as finger joint movements, fingertip pressure, palm interaction, and forearm muscle activity, thus restricting the accuracy and real-time performance of the entire system in parsing complex sign language semantics.

[0043] This application belongs to the interdisciplinary field of flexible electronics and intelligent sensing, specifically involving functional materials, additive manufacturing, and AI-assisted human-computer interaction systems. Its sub-technologies mainly include: first, flexible sensing materials, particularly ion-conductive hydrogels with tunable electromechanical properties and their applications in wearable devices; second, multimodal sensing system integration, covering the design of collaborative sensing gloves and electrode arrays based on strain, pressure, and surface electromyography (sEMG) signals; and third, AI-driven signal processing and classification, including multimodal data fusion and gesture recognition algorithms based on bidirectional long short-term memory networks (Bi-LSTM). This application, through material innovation and system-level integration, promotes the practical application of high-performance, low-hysteresis, and customizable hydrogel sensors in the field of barrier-free human-computer interaction.

[0044] Before detailing the embodiments of this application, some terms used in the embodiments of this application will be explained first, so that those skilled in the art can understand them.

[0045] PIG (Printable Ion-conductive Hydrogel) is a composite hydrogel that combines adjustable mechanical properties, stable ionic conductivity, low hysteresis, and environmental stability. It achieves multi-sensor adaptation through the synergistic effect of biorthogonal cross-linking networks and functional additives.

[0046] The polyampholyte network is a reversible ionic cross-linked network formed by the electrostatic interaction between the cationic monomer [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt (DMAEA-Q) and the polyanionic electrolyte poly(4-styrene sulfonate) (PNaSS), providing the material with dynamic recoverability and high conductivity.

[0047] Covalent organic framework nanosheets (COF nanosheets) refer to TpPa-1 type covalent organic framework nanosheets with hydroxyl, carbonyl and imine groups on the surface. They act as nano-reinforcing fillers and form strong hydrogen bonds with polymer networks to inhibit irreversible slippage of polymer chains and reduce mechanical hysteresis.

[0048] PNaSS (poly(sodium 4-styrenesulfonate)) refers to poly(sodium 4-styrenesulfonate), a polyanionic electrolyte powder obtained by ultraviolet polymerization, drying, ball milling and sieving of sodium styrenesulfonate (NaSS). It is the core component for constructing polyampholyte networks.

[0049] Lap (Lithium Phenyl-2,4,6-Trimethylbenzoylphosphinate) refers to lithium phenyl-2,4,6-trimethylbenzoyl phosphate, the second photoinitiator in this application, used to initiate the ultraviolet photopolymerization reaction of ion-conductive hydrogel ink.

[0050] Direct Ink Writing (DIW) is an additive manufacturing technology based on the shear-thinning properties of materials. It can accurately print homogeneous printable PIG ink onto a pre-defined area of ​​a flexible substrate, achieving integrated molding of the sensor and the substrate, and is suitable for the preparation of complex sensing patterns.

[0051] The biorthogonal crosslinking network refers to the hydrogel structure of this application, which is composed of an ionic crosslinking network formed by polyampholyte and a covalent crosslinking network formed by the polymerization of acrylamide (AAM) and N,N'-methylenebisacrylamide (MBAA), thus possessing both dynamic recoverability and structural stability.

[0052] Multimodal sensors refer to integrated sensing units that combine strain sensors, pressure sensors, and surface electromyography (sEMG) sensors. They are fabricated using the same hydrogel material platform and can simultaneously collect multi-dimensional gesture features such as hand joint flexion, fingertip pressure, and forearm muscle electrical signals.

[0053] Surface electromyography (sEMG) is a weak electrical signal generated during muscle activity, collected by electrodes on the skin surface. It can reflect the muscle intention of hand movements and is a key signal mode that supplements the motor features in sign language recognition.

[0054] The multichannel differential surface electromyography electrode array consists of 20 PIGs 0-5 The 10-channel differential electrode structure, composed of Ag electrodes, is fitted to the skin of the forearm for high-fidelity acquisition of sEMG signals from multiple muscle groups, reducing motion artifacts and environmental interference.

[0055] The real-time bidirectional communication protocol is a communication protocol used for data transmission between the Wi-Fi module and the cloud server. The WebSocket protocol is preferred. It supports real-time wireless transmission of sensor digital signals and ensures the low latency requirement for sign language recognition.

[0056] The Bidirectional Long Short-Term Memory (Bi-LSTM) network is the multimodal data fusion model adopted in this application. It contains two independent branches that process low-frequency strain / pressure signals and high-frequency sEMG signals respectively. Through feature layer splicing, it achieves deep fusion of multi-dimensional information and improves gesture recognition accuracy.

[0057] Min-Max Normalization is a data preprocessing method that linearly scales the original sensor signal to the [0,1] interval. The formula is Xnorm=(X-Xmin) / (Xmax-Xmin). It is used to eliminate the dimensional differences between signals from different channels and optimize the model training effect.

[0058] The sliding window method is a method that segments the temporal sensing signal with a step size of 20ms to extract temporal feature fragments of gesture actions and provide structured input data for the model.

[0059] The present invention will be further described in detail below with reference to embodiments, but the embodiments of the present invention are not limited thereto. Unless otherwise specified, the reagents, methods and equipment used in the present invention are conventional reagents, methods and equipment in this technical field. The test methods in the following embodiments that do not specify specific experimental conditions are generally performed under conventional experimental conditions. Unless otherwise specified, the reagents and raw materials used in the present invention are all commercially available. Among them, lithium chloride (LiCl), sodium p-styrenesulfonate (NaSS, 90% wt) and glycerol (C3H8O3, 99% wt) were purchased from Sigma-Aldrich. Lithium phenyl (2,4,6-trimethylbenzoyl) phosphate, lithium phenyl-2,4,6-trimethylbenzoyl phosphate (Lap, 98%) and dimethylaminoethyl acrylate quaternary ammonium salt (DMAEA-Q, 80% wt) were purchased from J&K Chemicals Ltd. Acrylamide (AAM, 99 wt%) and phenyl-1,4-diamine (1:1) in the form of 2,4,6-trihydroxybenzene-1,3,5-tricarboxyaldehyde (TpPa-1, 98 wt%) were purchased from J&K Scientific Co. Ltd. α-ketoglutaric acid (α-keto) and N,N'-methylenebisacrylamide (MBAA) were supplied by Sinopharm Chemical Reagent Co. Ltd. All reagents were analytical grade and used directly, and all experiments were conducted using deionized water (DI, 18.3 MΩ).

[0060] Reference Figure 1 This application provides a method for fabricating an ion-conductive hydrogel multimodal sensor, comprising the following steps: A prepolymer solution containing anionic monomer and a first photoinitiator was prepared, and polymerized by ultraviolet irradiation to form a prehydrogel. The prehydrogel was then dried, ball-milled, and sieved to obtain polyanionic electrolyte powder. Polyanionic electrolyte powder is dissolved in water to obtain an aqueous solution of polyanionic electrolyte. Acrylamide monomers, crosslinking agents, nano-reinforcing fillers, ionic conductive agents, and water-retaining agents are added to this solution, and the mixture is stirred and dispersed uniformly to form a first mixture. Cationic monomers and a second photoinitiator are added to the first mixture, and the reaction yields a homogeneous, printable first ionic conductive hydrogel ink. This first ionic conductive hydrogel ink is used for strain sensing or pressure sensing applications. Polyanionic electrolyte powder is dissolved in water to obtain a polyanionic electrolyte aqueous solution. Acrylamide monomers, nano-reinforcing fillers, ionic conductive agents and water-retaining agents are added to the solution and mixed evenly to form a second mixture. Cationic monomers and a second photoinitiator are added to the second mixture and reacted to obtain a homogeneous and printable second ionic conductive hydrogel ink. The second ionic conductive hydrogel ink is used for electromyography. The first and second ion-conductive hydrogel inks were printed onto a flexible substrate using 3D printing technology and cured by ultraviolet light. The resulting strain sensor, pressure sensor and electromyography sensor were fabricated, which together constitute an ion-conductive hydrogel multimodal sensor.

[0061] In one embodiment of this application, the polyanionic electrolyte powder is poly(4-styrene sulfonate), the anionic monomer is sodium styrene sulfonate, and the first photoinitiator is α-ketoglutarate. First, a homogeneous prepolymer solution is prepared in a 70°C water bath using deionized water as the solvent. This solution contains 1M sodium styrene sulfonate (NaSS, 90wt%) and 0.05mol% α-ketoglutarate (α-keto). The prepolymer solution is then rapidly injected into a reaction chamber consisting of a pair of transparent glass plates fitted with silicone gaskets, measuring 10cm × 10cm × 0.1cm. Finally, the mixture is irradiated with ultraviolet light at room temperature (405 nm wavelength, 10 W cm⁻²) for 10 hours to form a prehydrogel. After drying in a 65°C oven for 24 hours, the prehydrogel is ball-milled three times at 320 rpm, ultimately obtaining homogeneous PNaSS powder through a 300-mesh sieve.

[0062] Using polyanionic electrolytes as the base material, two types of ion-conducting hydrogel inks adapted to different sensing requirements were prepared by selectively adding crosslinking agents. Specifically, both inks used PNaSS aqueous solution as the base, with the following raw material configuration: acrylamide monomer as acrylamide, N,N'-methylenebisacrylamide as the crosslinking agent, TpPa-1 type covalent organic framework nanosheets as the nano-reinforcing filler, lithium chloride as the ion-conducting agent, glycerol as the water-retaining agent, [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt as the cationic monomer, and lithium phenyl-2,4,6-trimethylbenzoyl phosphate as the second photoinitiator. The only difference between the two types of inks is the addition of a crosslinking agent, thus forming a formulation adapted to different sensing modes. The first type of ion-conducting hydrogel ink (PIG)... 0.075-5 It incorporates MBAA to adapt to strain and pressure sensing, while the second ion-conductive hydrogel ink (PIG) 0-5 (Without adding MBAA, it is compatible with surface electromyography.)

[0063] The first ion-conductive hydrogel ink was prepared by dissolving sodium poly(4-styrene sulfonate) (PNaSS) powder in water to prepare an aqueous solution with a concentration of 0.5688 mol / L. Acrylamide (AAM, 5 mol / L), N,N'-methylenebisacrylamide (MBAA, 0.075 wt%, i.e., 0.0042 mol / L), TpPa-1 type covalent organic framework nanosheets (0.1 wt%, i.e., 0.00113 mol / L), lithium chloride (LiCl, 1 mol / L), and glycerol (30% of the total volume) were added sequentially to this aqueous solution. The raw materials were first initially mixed by stirring, and then the aggregates were broken up by ultrasonic dispersion to form the first mixture. [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt (DMAEA-Q, 0.6 mol / L) and lithium phenyl-2,4,6-trimethylbenzoyl phosphate (Lap, 0.2% of total volume, with its molar fraction maintained at 0.1 mol% relative to the total ionic monomer concentration (1.15 mol / L)) were added to the first mixture. The mixture was then heated and stirred in a 70°C water bath to allow the system to react fully, ultimately forming a homogeneous first ionic conductive hydrogel ink PIG. 0.075-5 .

[0064] The second ion-conducting hydrogel ink was prepared by dissolving sodium poly(4-styrene sulfonate) (PNaSS) powder in water to prepare an aqueous solution with a concentration of 0.5688 mol / L. Acrylamide (AAM, 5 mol / L), TpPa-1 type covalent organic framework nanosheets (0.1 wt%, i.e., 0.00113 mol / L), lithium chloride (LiCl, 1 mol / L), and glycerol (30% of the total volume) were added sequentially to this aqueous solution. The raw materials were first initially mixed by stirring, and then the aggregates were broken up by ultrasonic dispersion to form a second mixture. [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt (DMAEA-Q, 0.6 mol / L) and lithium phenyl-2,4,6-trimethylbenzoyl phosphate (Lap, 0.2% of total volume, with its molar fraction maintained at 0.1 mol% relative to the total ionic monomer concentration (1.15 mol / L)) were added to the second mixture. The mixture was then heated and stirred in a 70°C water bath to allow the system to react fully, ultimately forming a homogeneous second ionic conductive hydrogel ink PIG. 0-5 .

[0065] Finally, using direct-write 3D printing technology, the first and second ion-conductive hydrogel inks were printed onto the corresponding preset sensing areas of the flexible substrate. These were then cured by ultraviolet light irradiation at a wavelength of 405 nm for 2 hours, ultimately producing strain sensors, pressure sensors, and electromyography sensors, which together constitute an ion-conductive hydrogel multimodal sensor. The nozzle diameter was 0.1 mm, the nozzle height from the bottom surface (i.e., the printing height) was 0.1 mm, the printing pressure was 200 kPa, and the printing speed was 5 mm / s. The flexible substrate was a flexible printed circuit board (PCB), with a pair of copper electrodes corresponding to each preset sensing area. The first or second ion-conductive hydrogel ink was printed between the two copper electrodes in the corresponding area, and the sensing signals were acquired and transmitted through the conductive pathway formed after ink curing.

[0066] Reference Figure 2 ah, among which, Figure 2 a and Figure 2 b verified the successful preparation of the gel. The effect of AAM concentration (0–7 mol / L) on the rheological properties of PIG ink was systematically studied. Figure 2 c and Figure 2d) It was found that the ink without AAM was in a liquid state, and the loss modulus (G″) was always higher than the storage modulus (G′) throughout the entire shear strain range, indicating insufficient mechanical stability. The ink containing 5 mol / L AAM exhibited elastic dominance (G′>G″), and when the concentration increased to 7 mol / L, the viscosity and modulus significantly increased, which was attributed to the strong hydrogen bonding between AAM and COF. The viscoelasticity of this ink was strain-dependent: it remained elastically dominant when the oscillating strain was below 0.63%, and a viscoelastic transition occurred after exceeding the threshold, turning into a quasi-liquid state at 100% strain. Regarding the optimization of printing parameters, the nozzle height (H=0.1mm) and inner diameter (D=0.1mm) were fixed, and the air pressure (P) and nozzle speed (Vp) were adjusted. Figure 2 e and Figure 2 f) can print clear patterns with minimal difference in shape before and after curing. Printability is assessed by measuring the filament diameter using an electron microscope. Figure 2 g and Figure 2 h): Low pressure and high speed (e.g., 50 kPa, 5 mm / s) cause the filament bundle to break, while high pressure and low speed (e.g., 300 kPa, 1 mm / s) cause the filament to expand. Quantitative analysis shows that the filament bundle diameter is negatively correlated with Vp under constant pressure, and increases with the increase of P value under constant speed.

[0067] Reference Figure 3 aj, where, Figure 3 a and Figure 3 b indicates that with increasing AAM or MBAA content, the Young's modulus, tensile strength, and tensile work of the gel increase, while the elongation at break decreases. Figure 3 c and Figure 3 d shows that its Young's modulus, compressive strength, and compressive work are all improved simultaneously. Figure 3 e and Figure 3 f represents the tensile / compressive hysteresis performance of the preferred sample. Figure 3 g and Figure 3 h represents the result of its fatigue resistance test. Figure 3 i and Figure 3 j represents a practical application demonstration. The preferred samples for strain and compression sensing are selected based on hysteresis screening (combined with...). Figure 4 and Figure 5 analyze). Figure 4 ad represents the loading-unloading stress-strain curve of a non-preferred sample, which needs to be compared with... Figure 3 The optimal sample data of e and 3f were analyzed together; for intuitive comparison, the specific analytical values ​​of the two sets of data were integrated and plotted on [the graph]. Figure 5 . Figure 5 The results showed that, within the range of 100%–500% tensile strain and 20%–60% compressive strain, the recovery rate of all formulations decreased with increasing hysteresis loop area, with PIG showing the highest recovery rate. 0.075-5(0.075 wt% MBAA, 5 mol / L AAM) showed the highest recovery rate under tensile load. Figure 5 The results show that the absolute area of ​​the hysteresis loop increases with increasing strain and AAM or MBAA content.

[0068] In summary, a polyampholyte network is constructed using [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt as the cationic monomer and poly(sodium 4-styrene sulfonate) as the polyanionic electrolyte. This network forms a reversible ionic crosslinking network through electrostatic interactions, providing dynamically recoverable mechanical properties and high ionic conductivity. A covalently crosslinked polyacrylamide network, formed by UV-initiated polymerization of acrylamide as an acrylamide monomer and N,N'-methylenebisacrylamide as a crosslinking agent, provides a stable framework structure and mechanical strength for the entire hydrogel. The TpPa-1 type covalent organic framework nanosheets, with their hydroxyl, carbonyl, and imine groups on their surface, can bond with the polyacrylamide network formed by acrylamide polymerization through hydrogen bonding, inhibiting irreversible slippage of polymer chains under external forces and reducing the mechanical hysteresis of the ion-conductive hydrogel. Lithium chloride provides freely moving lithium and chloride ions, serving as the main carrier for ion conduction and imparting high conductivity to the hydrogel. Glycerol effectively locks in moisture by forming strong hydrogen bonds with water molecules, inhibiting the evaporation of moisture from the hydrogel in the air and significantly improving its environmental stability.

[0069] By adjusting the concentrations of crosslinking agent MBAA and monomer AAM, formulations suitable for different sensing modes can be optimized. The first ion-conductive hydrogel ink, used in strain and pressure sensors, contains 0.075 wt% MBAA and 5 mol / L AAM. This formulation of hydrogel exhibits high elongation at break (550%), moderate compressive strain tolerance (90%), low Young's modulus (0.09 MPa), and high conductivity (0.22 S m). -1 This method is particularly suitable for detecting a wide range of joint flexion and pressure stimulation. A second ion-conductive hydrogel ink was used for the surface electromyography sensor, with an MBAA content of 0 wt% and an AAM concentration of 5 mol / L. The prepared hydrogel has an extremely low modulus (≈90 kPa) and high conductivity (0.22 S m). -1 It possesses excellent skin adhesion, effectively reducing electrode-skin interface impedance, making it suitable for acquiring weak electromyographic signals. The preferred sample for the electromyographic sensor is based on the gel's conductivity and modulus; lower modulus and higher conductivity are better, so MBAA of 0 was chosen. However, to achieve the desired 3D printing effect, based on the previous 3D printing analysis, AAM is 5 mol / L instead of 0.

[0070] Performance testing of ion-conducting hydrogels Tensile Testing: All mechanical property tests were conducted on a universal testing machine (model HZ-1004A, made in China) equipped with a 5kN load cell. Monotonic tensile test specimens were prepared in a dumbbell shape with a gauge length of 12mm, a width of 2mm, and a thickness of 1mm. The tensile rate was set to 50 mm·min⁻¹ (strain rate 7% s⁻¹). The actual dimensions of the specimens were precisely measured using vernier calipers. The specimen ends were fully clamped to prevent shoulder elongation, and sandpaper was applied between the specimen and the clamps to prevent slippage. No slippage was observed during the entire test, and all specimens fractured in the linear mid-section region. Young's modulus (E) was calculated from the slope of the initial linear region (strain ≤10%) of the stress-strain curve. The stress value was calculated by dividing the measured load by the cross-sectional area of ​​the specimen. The strain value was determined by measuring the displacement relative to the initial gauge length. The tensile work (Wt) absorbed by the specimen was calculated by integrating the area under the stress-strain curve using the following formula. σ and ε represent stress and strain, respectively, and εb represents the strain value at specimen fracture. Each specimen must be tested at least three times, and the average value and standard deviation are calculated; the standard deviation is presented in the form of error bars.

[0071] Compression Test: In the monotonic compression test, the specimen should be prepared as a cylinder with a diameter of 10 mm and a height of 3 mm. A monotonic compression test is conducted along the height of the cylinder at a compression rate of 3 mm / min⁻¹ (strain rate 1.7% s⁻¹; range length 3 mm). Actual test dimensions are precisely measured using vernier calipers. No slippage was observed throughout the test. The compressive modulus (Ec) is calculated from the slope of the initial linear segment of the stress-strain curve (strain ≤ 10%). The compressive work (Wc) absorbed by the specimen at different strains is obtained by integrating the stress-strain curve using the following formula: σ and ε represent stress and strain, respectively. Each group of samples must be tested at least three times, and the average value and standard deviation must be calculated; the standard deviation is presented in the form of error bars.

[0072] Reference Figure 10 and Figure 11This application also provides a sign language recognition system, including a wearable module, a hardware module, and a software module. The wearable module achieves multimodal signal acquisition through an integrated design. The wearable module includes a data glove and a flexible armband, as well as sensors obtained using the aforementioned method for fabricating ion-conductive hydrogel multimodal sensors. The sensors include multiple strain sensors, multiple pressure sensors, and multiple electromyography (EMG) sensors. The strain and pressure sensors are integrated on the data glove, while the EMG sensors are integrated on a multi-channel differential surface electromyography (SSME) electrode array on the flexible armband. Specifically, the strain sensors are positioned at the corresponding finger joints, palm, and wrist of the data glove to monitor hand joint flexion and palm deformation; the pressure sensors are positioned at the corresponding fingertips of the data glove to monitor fingertip contact pressure; and the EMG sensors are multi-channel differential surface electromyography (SSME) electrode arrays, fitted to the forearm skin, to collect muscle electrical signals during hand movements.

[0073] Reference Figure 6 and Figure 7 In this embodiment, each data glove integrates 12 strain sensors and 5 pressure sensors on its flexible printed circuit board. Specifically, one strain sensor is placed at the proximal interphalangeal joint and metacarpophalangeal joint of each finger to monitor joint flexion, and one pressure sensor is placed at each fingertip to monitor fingertip contact or pressure. One strain sensor is also placed at the palm and wrist to monitor palm movement and wrist flexion.

[0074] Reference Figure 6 The data glove structure, built with strain and pressure sensors, places a PIG hydrogel between copper electrodes on two flexible printed circuit boards, encapsulated externally with an elastic encapsulation material such as Ecoflex. Each glove contains 12 strain sensors and 5 pressure sensors: one strain sensor at the proximal interphalangeal joint and one at the interphalangeal joint of each finger (totaling 10), one strain sensor at the palm and wrist, and one pressure sensor at the fingertip of each finger (totaling 5). When the sensors are stretched or compressed, the hydrogel deforms, altering the internal ion migration paths and causing a change in resistance (ΔR / R0). By measuring the change in resistance, the magnitude of the strain (ε) or pressure (P) can be deduced. (See reference...) Figure 8 The serpentine structure within the FPCB ensures that the FPCB can deform along with the hydrogel to accommodate varying degrees of bending. This sensor exhibits high linearity and a strain sensitivity coefficient of 2.3 (with a pressure sensitivity of -0.013 kPa over a strain range of 0-550%). -1 (within the range of 0-50 kPa).

[0075] Each flexible armband integrates a 10-channel differential sEMG electrode array, worn on the forearm, to collect surface electromyography (EMG) signals from relevant muscle groups during hand gestures. The surface EMG sensor employs a sandwich structure for the 10-channel sEMG electrode array, with one side of the PIG hydrogel in direct contact with the skin and the other side in contact with a flexible substrate deposited with silver conductive traces, forming a PIG-Ag electrode.

[0076] The fabrication process of sEMG electrode array is as follows: Figure 9 As shown, firstly, a flexible PU film is prepared as the substrate to construct the conductive circuit. After alignment with the screen printing template, silver paste is applied and dried to form the silver conductive circuit. Subsequently, an insulating layer is treated and the electrodes are exposed. A thermoplastic polyurethane film is used as a protective insulating layer. A hot pressing process is used to pressurize and heat the insulating layer to shape it and precisely expose the silver electrode connection points. Next, functional materials are printed, and after positioning the silver electrode connection points, PIG hydrogel ink is deposited and initially shaped. Finally, the process proceeds to curing and molding. The PIG material is completely cross-linked and cured by irradiation with ultraviolet light at a wavelength of 405nm for 2 hours, ultimately obtaining the finished sEMG electrode array.

[0077] Performance testing of a 10-channel differential surface electromyography (sEMG) electrode array Adhesion test method: The 180-degree peel test method specified in ASTM F2256 was used to evaluate the interfacial toughness of the pigskin adhesive at the pigskin contact surface. Adhesion strength was determined by calculating the average force under stable peel conditions and dividing by the adhesion width. Each specimen was 50 mm long, 15 mm wide, and 2 mm thick. To prevent stretching of the hydrogel samples during adhesion testing, cyanoacrylate adhesive was used to bond them to polyethylene terephthalate (PET) film. Each sample was subjected to a 5 Newton load for 60 seconds before testing. All tests were performed using a universal testing machine at a constant tensile speed of 100 mm / min. The PIG hydrogel exhibited low modulus and high adhesion (adhesion strength to pigskin reached 73.46 Nm). -1 This ensures dynamic fit with the skin and significantly reduces motion artifacts.

[0078] Mechanical interference resistance test: When subjects performed a gripping task at 50% of their maximum voluntary contraction force (MVC), the skin at the electrode site was subjected to periodic stretching and compression interference. The sEMG signal was then measured. Compared to traditional Ag / AgCl (silver / silver chloride) electrodes, the electrode of this application exhibited a higher signal-to-noise ratio under mechanical interference, with a static signal-to-noise ratio of 23.99 dB and a dynamic signal-to-noise ratio of 18.15 dB, demonstrating excellent interference resistance.

[0079] In one embodiment, the hardware module includes a multi-channel data acquisition circuit and a Wi-Fi module. The multi-channel data acquisition circuit converts the analog signals acquired by the wearable module into digital signals, and the Wi-Fi module wirelessly transmits the digital signals to a cloud server via a real-time bidirectional communication protocol. The multi-channel data acquisition circuit includes an ADC chip and a gain amplifier to improve signal conversion accuracy and signal strength. Preferably, the ADC chip is an ADS1115, and the gain amplifier is a PGA280. The analog signals acquired by the sensor are digitized by the multi-channel data acquisition circuit and then wirelessly transmitted to the cloud server via the ESP32 Wi-Fi module using the WebSocket protocol.

[0080] In one embodiment, the software module includes a bidirectional long short-term memory (LSTM) network model deployed on a cloud server and a user interaction terminal. The bidirectional LTM network model extracts and classifies features from digital signals and outputs sign language recognition results. The user interaction terminal receives the recognition results and displays them in text or synthesized speech format. Specifically, in the bidirectional LTM network model, one branch processes signals acquired by strain and pressure sensors, while the other branch processes signals acquired by electromyography (EMG) sensors. The features extracted by the two branches are fused and then output as gesture recognition results through a classifier. The wearable module simultaneously acquires strain, pressure, and surface EMG signals. The raw signals are filtered, normalized, and preprocessed in segments before being input into the bidirectional LTM network model.

[0081] Reference Figure 12The user interaction terminal for the software module utilizes a WeChat mini-program as its user interface. It receives recognition results from the cloud and displays them in real-time in both text and synthesized speech, built using WeChat developer tools. The mini-program interface includes, but is not limited to, core functional areas such as signal connection status display, real-time display of recognition results, historical record query, and speech synthesis on / off switch. Recognition results are presented simultaneously in both text and synthesized speech modes. The text information intuitively displays the semantic content corresponding to the sign language, while the synthesized speech is played back instantly via audio playback. The interface also provides real-time feedback on signal transmission status and recognition success rate, ensuring users clearly understand the interaction progress. Specifically, users first click "connect to server" in the lower right corner to connect to the server, then click "start collecting" in the lower left corner to send a collection command to the server. Upon receiving the command, the server sends a collection command to the ESP32. The ESP32 then begins collecting sensor data and immediately sends the collected data to the server. The server receives the data, processes it, inputs it into the Bi-LSTM model, and sends the recognition results to the WeChat mini-program. After receiving the recognition result, the mini-program replaces "the text corresponding to the sign language" with the displayed text and plays the corresponding audio. The current recognition result will be displayed until the next result arrives, and the corresponding audio can be played again by clicking the play button. Reference Figure 13 Based on the aforementioned sign language recognition system, the core of the sign language recognition method of this invention lies in utilizing a bidirectional long short-term memory network to fuse and classify multimodal time-series data. A schematic diagram of the specific model architecture is shown below. Figure 14 This model employs a dual-branch architecture, capable of independently processing strain / pressure and surface electromyography (sEMG) signal flow. Each branch contains a three-layer stacked bidirectional LSTM network followed by two fully connected layers, with the outputs integrated through a feature fusion layer. Specifically, the strain / pressure sensing branch uses a 128-hidden-unit bidirectional LSTM to capture low-frequency temporal features, while the sEMG branch uses a higher-specification 512-hidden-unit network to extract fine-grained temporal dynamics. After nonlinear temporal feature extraction, the fused multimodal features are concatenated and input into a fully connected network for classification. The model is trained using the AdamW optimizer with a batch size of 50, a learning rate of 0.001, and a dropout rate of 0.3.

[0082] The identification method includes the following steps: (1) Data collection The system simultaneously acquires 15 strain signals, 15 pressure signals, and 10 sEMG signals. The sampling rate for strain and pressure signals is 200Hz, and the sampling rate for sEMG signals is 1000Hz. Each gesture lasts for 2 seconds, generating a multimodal time-series sample consisting of two types of data: strain and pressure data, containing 400 time points with a 40-dimensional data dimension (including 6 virtual channels), and sEMG data, containing 2000 time points with a 20-dimensional data dimension.

[0083] (2) Data preprocessing The acquired raw signals were sequentially filtered, normalized, and segmented to construct a training dataset. The filtering stage employed a Butterworth filter to extract signals within the 10-500Hz frequency range, while simultaneously removing power frequency interference using band-stop filtering. The normalization stage used a minimum-maximum normalization method to linearly scale the data from each channel to the 0-1 range, with the formula: Xnorm = (X - Xmin) / (Xmax - Xmin). The segmentation stage used a sliding window method, dividing the signal into segments with a step size of 20 milliseconds.

[0084] (3) Model building and training A bidirectional long short-term memory network multimodal fusion deep learning model is constructed. This model adapts to the heterogeneity of strain pressure signals and surface electromyography signals through a bidirectional architecture, and can independently process two different signal streams, ensuring the pertinence and effectiveness of feature extraction.

[0085] The strain-stress branch has an input dimension of 400×40 and employs a three-layer bidirectional long short-term memory network for feature extraction. This network has 128 hidden units to accurately capture the low-frequency spatiotemporal features of hand movements. The surface electromyography (EMG) signal branch has an input dimension of 2000×20 and also employs a three-layer bidirectional ESM network with 512 hidden units to capture the high-frequency, fine-grained spatiotemporal dynamic features of muscle activity. Both branches contain three stacked bidirectional ESM networks, followed by two fully connected layers. The features output from each branch are then transferred to a feature fusion layer for integration.

[0086] For feature fusion and classification, the high-dimensional feature vectors extracted from the two branches are first concatenated, and then the concatenated features are input into a fully connected network to complete deep fusion. Finally, the classification probabilities corresponding to 24 Chinese sign language gestures are output through a Softmax classifier.

[0087] Model training parameter settings: The dataset was divided into training, validation, and test sets in an 8:1:1 ratio for training the classification of 24 gestures. The model used the AdamW optimizer, with a batch size of 50, a learning rate of 0.001, and a dropout rate of 0.3. The changes in the loss function and classification accuracy during training are shown below. Figure 15 As shown, no signs of underfitting or significant overfitting were observed in the trend of change, and the model training state is stable and reliable.

[0088] (4) Real-time recognition The trained bidirectional long short-term memory network model is deployed on a cloud server. Sensor data collected in real time by the system is uploaded to the cloud wirelessly. After being processed by the model, the recognition results are instantly returned to the WeChat mini-program on the mobile phone, thus realizing real-time translation and voice broadcast of sign language. The overall latency of the entire recognition process is less than 0.1 seconds.

[0089] Test results show that the system achieved a classification accuracy of 99.65% in the task of recognizing 24 Chinese sign language gestures. The relevant clustering diagram and confusion matrix are shown below. Figure 16 As shown, the model's classification performance and the confusion in recognizing each gesture category are presented intuitively.

[0090] It should be noted that the above hardware design is merely a specific example illustrating the specific structure and appearance of the fabrication method of the ion-conductive hydrogel multimodal sensor and the sign language recognition system provided by this invention in practical applications, the installation positions of each module, and the specific hardware selection in each module. The system's software settings are matched to the hardware design. The specific settings of the hardware and software designs can be adjusted according to actual needs; this embodiment does not limit this.

[0091] This invention proposes a method for fabricating an ion-conductive hydrogel multimodal sensor and a sign language recognition system, which can be used for daily sign language communication scenarios for hearing-impaired individuals. By capturing strain, pressure signals, and forearm electromyography signals of hand movements in real time through a wearable module, the system quickly identifies and converts these signals into text and synthesized speech via a cloud-based model. This breaks down communication barriers between hearing-impaired and hearing individuals, allowing sign language communication to proceed without the need for professional interpreters. It is suitable for various everyday scenarios such as family communication, communication at public service windows, and daily social interactions, significantly improving the communication convenience for hearing-impaired individuals.

[0092] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method of preparing an ionically conductive hydrogel multi-modal sensor, characterized in that, The method comprises the following steps: a prepolymer solution containing an anionic monomer and a first photoinitiator is prepared, ultraviolet irradiation is performed to polymerize the prepolymer solution to form a prehydrogel, and the prehydrogel is sequentially subjected to drying, ball milling and sieving to obtain a polyanion electrolyte powder; the polyanion electrolyte powder is dissolved in water to obtain a polyanion electrolyte aqueous solution, acrylamide monomers, a crosslinking agent, a nano-enhanced filler, an ion conductive agent and a water-retaining agent are added to the polyanion electrolyte aqueous solution, and the mixture is uniformly dispersed to form a first mixed solution; cationic monomers and a second photoinitiator are added to the first mixed solution, and a homogeneous printable first ion conductive hydrogel ink is obtained by reaction, wherein the first ion conductive hydrogel ink is used for strain sensing or pressure sensing; and the polyanion electrolyte powder is dissolved in water to obtain a polyanion electrolyte aqueous solution, acrylamide monomers, a nano-enhanced filler, an ion conductive agent and a water-retaining agent are added to the polyanion electrolyte aqueous solution, and the mixture is uniformly dispersed to form a second mixed solution; cationic monomers and a second photoinitiator are added to the second mixed solution, and a homogeneous printable second ion conductive hydrogel ink is obtained by reaction, wherein the second ion conductive hydrogel ink is used for electromyographic sensing; the first ion conductive hydrogel ink and the second ion conductive hydrogel ink are printed on a flexible substrate by using a 3D printing technology, and are cured by ultraviolet irradiation, thereby obtaining a strain sensor, a pressure sensor and an electromyographic sensor, which together constitute an ion conductive hydrogel multi-modal sensor.

2. The method of claim 1, wherein the ionically conductive hydrogel multi-modal sensor is prepared by the steps of: In the preparation of the prepolymer solution, deionized water is used as a solvent, the anionic monomer and the first photoinitiator are fully stirred and dissolved, the concentration of the anionic monomer in the prepolymer solution is 0.8-1.2 M, and the addition amount of the first photoinitiator is 0.03-0.07% of the total molar amount of the prepolymer solution; In the ultraviolet irradiation polymerization, the prepolymer solution is injected into a reaction chamber, the irradiation wavelength is 405 nm, and the polymerization time is 10 hours; The prehydrogel is dried in an oven at 60-70°C for 20-28 hours, the ball milling treatment is performed at a speed of 300-350 rpm and for 2-4 times, and a 300-mesh sieve is used for sieving.

3. The method of claim 1, wherein the ionically conductive hydrogel multi-modal sensor is prepared by the steps of: The anionic monomer is sodium styrenesulfonate; The polyanion electrolyte is poly(4-sodium styrenesulfonate); The cationic monomer is [2-(dimethylamino)ethyl acrylate]-quaternary ammonium salt; The first photoinitiator is α-ketoglutaric acid; The second photoinitiator is phenyl-2,4,6-trimethylbenzoyl phosphate lithium.

4. The method of claim 1, wherein the ionically conductive hydrogel multi-modal sensor is prepared by, The acrylamide monomer is acrylamide, the crosslinking agent is N,N'-methylenebisacrylamide, and the nano-enhanced filler is a TpPa-1 type covalent organic framework nanosheet.

5. The method of claim 1, wherein the ionically conductive hydrogel multi-modal sensor is prepared by, The ion conductive agent is lithium chloride, and the water-retaining agent is glycerol; In the uniform mixing and dispersion to form the first mixed solution or the second mixed solution, stirring and ultrasonic dispersion are simultaneously performed, the raw materials are initially mixed by stirring, and the raw material agglomerates are broken by ultrasonic dispersion. After adding cationic monomers and a second photoinitiator into the first mixed solution or the second mixed solution, the system is heated and stirred at 65-75 DEG C for 1-3 hours to form a homogeneous first ionically conductive hydrogel ink or a second ionically conductive hydrogel ink.

6. The method of claim 1, wherein the ionically conductive hydrogel poly-modal sensor is prepared by the steps of: The 3D printing technology is direct writing 3D printing. During ultraviolet light irradiation curing, the wavelength of irradiation is 405 nm, the light intensity is 10-20 mW / cm2, and the curing time is 1.5-2.5 hours.

7. The method of claim 1, wherein the ionically conductive hydrogel poly-modal sensor is prepared by the steps of: The flexible substrate is a flexible printed circuit board, which is provided with a pair of copper electrodes corresponding to a preset sensing area, and the first ionically conductive hydrogel ink or the second ionically conductive hydrogel ink is printed between the two copper electrodes.

8. A sign language recognition system characterized by, Comprise: A wearable module comprising a data glove and a flexible arm band, and a sensor obtained by the preparation method of the ionically conductive hydrogel multi-modal sensor according to any one of claims 1 to 7, the sensor comprising a plurality of strain sensors, a plurality of pressure sensors, and a plurality of electromyography sensors, the plurality of strain sensors and the plurality of pressure sensors being integrally arranged on the data glove, and the plurality of electromyography sensors being integrally arranged on a multi-channel differential surface electromyography electrode array of the flexible arm band; A hardware module comprising a multi-channel data acquisition circuit and a Wi-Fi module, the multi-channel data acquisition circuit being used to convert analog signals collected by the wearable module into digital signals, and the Wi-Fi module being used to wirelessly transmit the digital signals to a cloud server through a real-time bidirectional communication protocol; A software module comprising a double-branch bidirectional long short-term memory network model deployed on the cloud server, and a user interactive terminal; the double-branch bidirectional long short-term memory network model is used to extract features and classify the digital signals, and output a sign language gesture recognition result; and the user interactive terminal is used to receive the recognition result and display it in the form of text or synthesized speech.

9. The sign language recognition system of claim 8, wherein, The plurality of strain sensors are arranged at finger joints, a palm, and a wrist corresponding to the data glove, and are used to monitor bending of hand joints and deformation of a palm; The plurality of pressure sensors are arranged at fingertips corresponding to the data glove, and are used to monitor fingertip contact pressure; The electromyography sensors are a multi-channel differential surface electromyography electrode array, and are attached to the skin of a forearm, and are used to collect muscle electrical signals during hand movement; The multi-channel data acquisition circuit comprises an ADC chip and a gain amplifier, so as to improve signal conversion accuracy and signal strength.

10. The sign language recognition system of claim 8, wherein, In the double-branch bidirectional long short-term memory network model, one branch is used to process signals collected by the strain sensors and the pressure sensors, and the other branch is used to process signals collected by the electromyography sensors, features extracted by the two branches are fused, and a gesture recognition result is output through a classifier; The wearable module synchronously collects strain signals, pressure signals, and surface electromyography signals, the original signals collected are preprocessed through filtering, normalization, and segmentation, and then are input into the double-branch bidirectional long short-term memory network model.

Images