A handwritten letter signal recognition system and method based on flexible sensor

Abstract

The present invention discloses a handwritten letter signal recognition system and method based on a flexible sensor. The flexible sensor is arranged at the finger joint. During the handwriting process, the flexible sensor senses the voltage change caused by the strain of bending at the finger joint and sends it to a semiconductor feature analyzer; the semiconductor feature analyzer obtains the real-time resistance signal of the flexible sensor and transmits it to a signal processor; the signal processor has a built-in multi-stream convolutional neural network, receives the signal of the semiconductor feature analyzer, and then obtains the classification result of the handwritten letter corresponding to the output handwritten strain signal through multi-stream convolutional neural network analysis and processing. The conductive material of the flexible sensor of the present invention is a liquid metal material, which has very high sensitivity and conductivity, ensuring the feasibility and accuracy of letter recognition. The recognition method can effectively judge the letter category according to the transmitted resistance change, which is more intuitive and accurate.

Description

Technical Field

[0001] The invention relates to a signal recognition system, and in particular to a handwritten letter signal recognition system and method based on a flexible sensor. Background Art

[0002] In recent years, with the coordinated progress of computer science, optoelectronics, materials science, mechanics and other fields, artificial intelligence technology has developed rapidly at an unprecedented speed, and smart wearable devices that convert external stimuli such as force, strain, temperature, and humidity into electrical signals have been booming. Traditional silicon-based semiconductor and metal-based rigid strain sensors have the advantages of high resolution, small error, and can measure dynamic and static forces. However, their inherent rigidity and brittleness lead to poor flexibility, non-stretchability, and extremely small strain range (<5%), which makes it difficult to meet the requirements of high stretch and high sensitivity at the same time, seriously affecting their measurement stability, measurement accuracy and measurement range. The normal physiological activities of the human body will withstand 30% strain in the skin and about 100% strain in the joints. Traditional rigid materials will fail at 1-3% strain, which limits its development and application. As one of the indispensable and important components in smart wearable devices, flexible strain sensors have naturally become a hot spot in scientific research today. Although the materials of traditional strain sensors such as silver nanowires, carbon nanotubes, graphene, etc. can greatly improve their sensitivity, the inherent brittleness of these materials reduces the mechanical strength of the sensor, which limits the practical use of the sensor and affects the durability and reliability of the sensor. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention provides a handwritten letter signal recognition system and method based on a flexible sensor, designs a conductive hydrogel composed of liquid metal and a flexible hydrogel matrix, prepares the liquid metal conductive hydrogel into a flexible sensor, and applies it to signal collection, and then combines it with an artificial intelligence algorithm to provide a handwritten letter signal recognition system and method based on a flexible sensor.

[0004] The technical solution adopted by the present invention is:

[0005] 1. A handwritten letter signal recognition system based on flexible sensor:

[0006] The handwritten letter signal recognition system includes a handwritten strain signal acquisition device and a signal processor, and the handwritten strain signal acquisition device and the signal processor are electrically connected; the handwritten strain signal acquisition device includes a flexible sensor and a 4200 semiconductor feature analyzer, the flexible sensor is electrically connected to the 4200 semiconductor feature analyzer, and the 4200 semiconductor feature analyzer is electrically connected to the signal processor.

[0007] The flexible sensor is arranged at the finger joints. The flexible sensor is used to sense the voltage changes caused by the strain under the bending of the finger joints during the process of handwriting letters, and send it to the 4200 semiconductor characteristic analyzer; the 4200 semiconductor characteristic analyzer is used to receive the voltage changes from the flexible sensor and derive the real-time resistance signal of the flexible sensor, and transmit it to the signal processor.

[0008] The signal processor has a built-in multi-stream convolutional neural network for receiving the signal from the 4200 semiconductor feature analyzer, and then obtains the classification result of the handwritten letters corresponding to the output handwritten strain signal through multi-stream convolutional neural network analysis and processing.

[0009] 2. A handwritten letter signal recognition method based on flexible sensor:

[0010] The handwritten letter signal recognition method comprises the following steps:

[0011] 1) Preparation of flexible sensors.

[0012] 2) The flexible sensor is arranged at the finger joint and electrically connected to the 4200 semiconductor characteristic analyzer.

[0013] 3) Use a pen to write several letters, and each letter is written repeatedly for the same number of times. The bending strain change of the flexible sensor during handwriting is received by a 4200 semiconductor characteristic analyzer, and the strain resistance signal when each letter is handwritten is calculated as a handwriting strain signal, thereby obtaining several handwriting strain signals corresponding to each handwritten letter. The several handwriting strain signals corresponding to each handwritten letter constitute a signal data set.

[0014] 4) The signal data set is standardized and several handwritten strain signals are selected as training sets; the rest are used as test sets.

[0015] 5) Building a multi-stream convolutional neural network and embedding it in a signal processor; inputting the training set into the multi-stream convolutional neural network in the signal processor for training;

[0016] The test set is input into the trained multi-stream convolutional neural network for verification, and the verification classification results of the handwritten letters corresponding to each handwriting pressure signal in the test set are obtained.

[0017] 6) Repeat step 3) to reacquire several handwritten strain signals to form a target signal data set, input the target signal data set into the trained multi-stream convolutional neural network for processing, and obtain the classification results of the handwritten letters corresponding to each handwritten pressure strain in the target signal data set.

[0018] In the step 1), preparing the flexible sensor comprises the following steps:

[0019] 1.1) Add acrylamide and acrylic acid into deionized water to dissolve, and then add zirconium oxychloride octahydrate and a photoinitiator 2-hydroxy-4'-(2-hydroxyethoxy)-2-methylpropiophenone in sequence to obtain a mixed solution.

[0020] 1.2) The mixed solution was placed on a magnetic stirrer and stirred for 30 minutes until the small solid particles in the mixed solution were fully dissolved and turned into a colorless transparent liquid, and then argon gas was introduced for 30 minutes to exhaust the dissolved oxygen in the colorless transparent solution to obtain a transparent solution.

[0021] 1.3) The transparent solution was injected into a pre-prepared hydrogel mold with a thickness of 0.5 mm, and placed under an ultraviolet lamp for photopolymerization for 5 minutes to obtain a semi-polymerized poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel that was not swollen and balanced; half of the semi-polymerized poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel was immersed in a large amount of deionized water for 7 days until swelling equilibrium was reached, and the mass of the hydrogel remained unchanged, thereby obtaining a poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel substrate.

[0022] 1.4) placing a poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel substrate in a glass dish, using a 3D printer to print a liquid metal conductive circuit with uniform line width on the poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel substrate, and bonding and electrically connecting one end of two copper electrodes at both ends of the liquid metal conductive circuit; transferring the poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel substrate, the liquid metal conductive circuit and the two copper electrodes in the glass dish to another pre-prepared hydrogel mold with a thickness of 1 mm, and coating the other half of the semi-polymerized poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel evenly on the copper electrode and the liquid metal conductive circuit until the semi-polymerized poly(acrylamide-co-acrylic acid) / zirconium ion hydrogel completely wraps the liquid metal conductive circuit and the two copper electrodes, thereby obtaining a semi-polymerized conductive hydrogel strain sensor.

[0023] 1.5) Placing the semi-polymerized conductive hydrogel strain sensor under ultraviolet light for photopolymerization to obtain a liquid metal / hydrogel strain sensor, i.e., a flexible sensor.

[0024] In the step 1.1), 0.11-3.73 g of acrylamide, 0.54-6.81 g of acrylic acid, 0.48-4.83 g of zirconium oxychloride octahydrate and 0.03-0.24 g of a photoinitiator are added by mass.

[0025] Calculated by mole fraction (or concentration), it mainly includes 10-50 mol % of acrylamide, 10-50 mol % of acrylic acid, 0.1-1.0 mol / L of zirconium oxychloride octahydrate and 1.0 mol % of a photoinitiator.

[0026] In the step 1.4), the other ends of the two copper electrodes protrude the flexible sensor.

[0027] In the step 2), the other protruding ends of the two electrodes are electrically connected to the red and black wires of the 4200 semiconductor characteristic analyzer respectively.

[0028] In step 4), the following operations are performed for each handwritten strain signal in the signal data set:

[0029] The handwriting strain signal is standardized, the original data of the handwriting strain signal is scaled to between 0 and 1, and the distribution of the original data of the handwriting strain signal is retained.

[0030] In the step 5), the multi-stream convolutional neural network is mainly composed of a multi-stream decomposition stage and a fusion stage performed sequentially.

[0031] The multi-stream decomposition stage includes four identical decomposition parts arranged in sequence, each of which includes a 32-convolution kernel convolution layer, a 64-convolution kernel convolution layer, a 128-convolution kernel convolution layer and a 128-convolution kernel local connection layer; the fusion stage includes a 512-convolution kernel fully connected layer, a 256-convolution kernel fully connected layer, a 128-convolution kernel fully connected layer and a 64-convolution kernel fully connected layer.

[0032] The input of the multi-stream convolutional neural network is evenly divided into four input parts of equal length, which correspond to the four identical decomposition parts arranged in sequence in the multi-stream decomposition stage. The four input parts are input into the four decomposition parts respectively; for each input part and the corresponding decomposition part, the following operations are performed:

[0033] The input part is sequentially input into the 32-convolution kernel convolution layer, 64-convolution kernel convolution layer, 128-convolution kernel convolution layer and 128-convolution kernel local connection layer of the decomposition part for processing. A linear rectification function and a BN algorithm are connected in sequence between the 32-convolution kernel convolution layer and the 64-convolution kernel convolution layer, between the 64-convolution kernel convolution layer and the 128-convolution kernel convolution layer, between the 128-convolution kernel convolution layer and the 128-convolution kernel local connection layer, and after the 128-convolution kernel local connection layer.

[0034] The outputs of the 32-convolution kernel convolution layer, the 64-convolution kernel convolution layer, the 128-convolution kernel convolution layer, and the 128-convolution kernel local connection layer are first input into a linear rectification function for processing. The output of the linear rectification function is then processed by the BN algorithm as the input of the 64-convolution kernel convolution layer, the 128-convolution kernel convolution layer, the 128-convolution kernel local connection layer and the final output of the decomposition part.

[0035] There is a linear rectification function connected between the 512-convolution kernel fully connected layer and the 256-convolution kernel fully connected layer, between the 256-convolution kernel fully connected layer and the 128-convolution kernel fully connected layer, and between the 128-convolution kernel fully connected layer and the 64-convolution kernel fully connected layer. A SoftMax classifier is connected after the 64-convolution kernel fully connected layer.

[0036] The final output of each decomposition part is feature fused in the fusion stage, and the fusion results are processed by 512 convolution kernel fully connected layer, 256 convolution kernel fully connected layer, 128 convolution kernel fully connected layer and 64 convolution kernel fully connected layer in turn. The outputs of 512 convolution kernel fully connected layer, 256 convolution kernel fully connected layer and 128 convolution kernel fully connected layer are first input into a linear rectification function for processing, and then used as the input of 256 convolution kernel fully connected layer, 128 convolution kernel fully connected layer and 64 convolution kernel fully connected layer respectively. The output of 64 convolution kernel fully connected layer is finally processed by SoftMax classifier to output the classification result.

[0037] The beneficial effects of the present invention are:

[0038] 1. Among many flexible substrate materials, hydrogel materials have the advantages of good water absorption, flexibility, elasticity and biocompatibility; liquid metal, as a conductive material that is liquid at room temperature, is one of the softest and most easily deformable conductors. Unlike traditional metal materials, liquid metal has the advantages of mechanical flexibility, high reliability, fast response, low hysteresis, high conductivity and low toxicity.

[0039] 2. The handwritten letter signal recognition method is to judge the written letters from the change of resistance. Therefore, the letters can be recognized more intuitively and accurately.

[0040] 3. The flexible sensor is an original invention. Due to the special material (liquid metal), it has very high sensitivity and conductivity, thus ensuring the feasibility and accuracy of letter recognition. Based on the high sensitivity and high accuracy of recognition, the written letters can be effectively obtained based on the changes in the transmitted resistance, allowing it to be widely used in the field of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The resistance change diagram for different letters;

[0042] Figure 2 The resistance change diagram of continuous handwriting of different letters;

[0043] Figure 3 Schematic diagram of the multi-stream convolutional neural network structure;

[0044] Figure 4 is the recognition accuracy of different letters. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The specific embodiments are as follows:

[0047] In a specific implementation, the handwritten letters are set to 26 English letters; 26 letters are handwritten by holding a pen, and each letter is handwritten 40 times; the current of the 4200 semiconductor characteristic analyzer is set to 0.01A, and the 4200 semiconductor characteristic analyzer obtains resistance changes by receiving voltage changes of the flexible sensor, and then obtains 40 handwritten strain signals corresponding to each handwritten letter, and the 40 handwritten strain signals corresponding to each handwritten letter constitute a signal data set;

[0048] like Figure 1 As shown in , the resistance change diagrams of different English letters are obtained, such as "A", "B", "C", "D", "E", "N", "O", "R" and "S"; Figure 2 As shown, this is the resistance change diagram when "S", "E", "N", "S", "O", and "R" are written continuously.

[0049] The sensitivity of the flexible sensor can be obtained according to the resistance change, and the sensitivity formula GF is as follows:

[0050]

[0051] Among them, β represents the fitting function, which is used to describe the contact resistance between the copper electrode and the liquid metal conductive circuit; ε represents the strain of the flexible sensor; ΔR represents the resistance change value of the flexible sensor; ΔL represents the change value of the path length of the liquid metal conductive circuit; R0 represents the initial resistance of the flexible sensor; L0 represents the conductive path length of the liquid metal conductive circuit;

[0052] The signal data set is standardized, and 60% of the handwritten strain signals are selected as the training set, and the remaining 40% are selected as the test set. The test set is input into the trained multi-stream convolutional neural network for verification, and the verification classification results of the handwritten letters corresponding to each handwritten strain signal in the test set are obtained. Figure 3 The figure shows the structure of the multi-stream convolutional neural network. Several handwritten strain signals can be re-acquired to form a target signal data set, and the target signal data set is input into the trained multi-stream convolutional neural network for processing to obtain the classification results of the handwritten letters corresponding to each handwritten strain signal in the target signal data set.

[0053] The sensitivity of the system of the present invention is 1.03, and the final recognition accuracy acc is 60.75%. Figure 4 The following table shows the recognition accuracy of a single letter.

Claims

1. A handwritten letter signal recognition method of a handwritten letter signal recognition system based on a flexible sensor, the handwritten letter signal recognition system comprising a handwritten strain signal acquisition device and a signal processor, the handwritten strain signal acquisition device and the signal processor are electrically connected; the handwritten strain signal acquisition device comprises a flexible sensor and a 4200 semiconductor feature analyzer, the flexible sensor is electrically connected to the 4200 semiconductor feature analyzer, and the 4200 semiconductor feature analyzer is electrically connected to the signal processor; The invention comprises a flexible sensor arranged at the finger joint, wherein the flexible sensor senses the voltage change caused by the strain of bending at the finger joint during the process of handwriting letters, and sends the voltage change to a 4200 semiconductor characteristic analyzer; the 4200 semiconductor characteristic analyzer receives the voltage change from the flexible sensor and obtains the real-time resistance signal of the flexible sensor, and transmits the signal to a signal processor; the signal processor has a built-in multi-stream convolutional neural network, receives the signal of the 4200 semiconductor characteristic analyzer, and then obtains the classification result of the handwritten letter corresponding to the output handwriting strain signal through the multi-stream convolutional neural network analysis and processing, and is characterized in that: The following steps are involved: 1) Preparation of flexible sensors; 2) placing the flexible sensor at the finger joint and electrically connecting the flexible sensor to a 4200 semiconductor characteristic analyzer; 3) Use a hand to write several letters with a pen, and each letter is written repeatedly for several identical times. The bending strain change of the flexible sensor during handwriting is received by a 4200 semiconductor characteristic analyzer, and the strain resistance signal when each letter is handwritten is calculated as a handwriting strain signal, thereby obtaining several handwriting strain signals corresponding to each handwritten letter. The several handwriting strain signals corresponding to each handwritten letter constitute a signal data set; 4) Standardize the signal data set and select several handwritten strain signals as training sets; 5) Build a multi-stream convolutional neural network and embed it in the signal processor; input the training set into the multi-stream convolutional neural network in the signal processor for training; 6) Repeat step 3) to reacquire several handwritten strain signals to form a target signal data set, input the target signal data set into the trained multi-stream convolutional neural network for processing, and obtain the classification results of the handwritten letters corresponding to each handwritten strain signal in the target signal data set; The multi-stream convolutional neural network is mainly composed of a multi-stream decomposition stage and a fusion stage that are performed sequentially; The multi-stream decomposition stage includes four identical decomposition parts arranged in sequence, each of which includes a 32-convolution kernel convolution layer, a 64-convolution kernel convolution layer, a 128-convolution kernel convolution layer and a 128-convolution kernel local connection layer; the fusion stage includes a 512-convolution kernel fully connected layer, a 256-convolution kernel fully connected layer, a 128-convolution kernel fully connected layer and a 64-convolution kernel fully connected layer.

2. The handwritten letter signal recognition method of the handwritten letter signal recognition system based on a flexible sensor according to claim 1, characterized in that: In the step 1), preparing the flexible sensor comprises the following steps: 1.1) adding acrylamide and acrylic acid into deionized water to dissolve, and then sequentially adding zirconium oxychloride octahydrate and a photoinitiator 2-hydroxy-4'-(2-hydroxyethoxy)-2-methylpropiophenone to obtain a mixed solution; 1.2) Stir the mixed solution on a magnetic stirrer until it becomes a colorless transparent liquid, then introduce argon gas to obtain a transparent solution; 1.3) The transparent solution was injected into a pre-prepared hydrogel mold and placed under a UV lamp for photopolymerization for 5 min to obtain semi-polymerized poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel; half of the semi-polymerized poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel was immersed in deionized water for 7 days to obtain poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel substrate; 1.4) Poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel substrate was placed in a glass dish and printed on poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel substrate, and bond and electrically connect one end of two copper electrodes to the two ends of the liquid metal conductive circuit; co -acrylic acid) / zirconium ion hydrogel substrate, liquid metal conductive circuit and two copper electrodes were simultaneously transferred to another pre-prepared hydrogel mold, and the other half of the semi-polymerized poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel is coated on the copper electrode and liquid metal conductive circuit until the semi-polymerized poly(acrylamide- co -acrylic acid) / zirconium ion hydrogel completely wraps the liquid metal conductive circuit and two copper electrodes to obtain a semi-polymerized conductive hydrogel strain sensor; 1.5) The semi-polymerized conductive hydrogel strain sensor is placed under an ultraviolet lamp for photopolymerization to obtain a liquid metal / hydrogel strain sensor, i.e., a flexible sensor.

3. The handwritten letter signal recognition method of the handwritten letter signal recognition system based on a flexible sensor according to claim 2, characterized in that: In the step 1.1), 0.11-3.73 g of acrylamide, 0.54-6.81 g of acrylic acid, 0.48-4.83 g of zirconium oxychloride octahydrate and 0.03-0.24 g of a photoinitiator are added by mass.

4. The handwritten letter signal recognition method of the handwritten letter signal recognition system based on a flexible sensor according to claim 2, characterized in that: In the step 1.4), the other ends of the two copper electrodes protrude the flexible sensor; In the step 2), the other protruding ends of the two electrodes are electrically connected to the red and black wires of the 4200 semiconductor characteristic analyzer respectively.

5. The handwritten letter signal recognition method of the handwritten letter signal recognition system based on a flexible sensor according to claim 1, characterized in that: In step 4), the following operations are performed for each handwritten strain signal in the signal data set: The handwriting strain signal is standardized, the original data of the handwriting strain signal is scaled to between 0 and 1, and the distribution of the original data of the handwriting strain signal is retained.

6. The handwritten letter signal recognition method of the handwritten letter signal recognition system based on a flexible sensor according to claim 1, characterized in that: In the step 5), the input of the multi-stream convolutional neural network is evenly divided into four input parts of equal length, and corresponds to the four identical decomposition parts arranged in sequence in the multi-stream decomposition stage, and the four input parts are respectively input into the four decomposition parts; for each input part and the corresponding decomposition part, the following operations are performed: The input part is sequentially input into the 32-convolution kernel convolution layer, 64-convolution kernel convolution layer, 128-convolution kernel convolution layer and 128-convolution kernel local connection layer of the decomposition part for processing. A linear rectification function and a BN algorithm are sequentially connected between the 32-convolution kernel convolution layer and the 64-convolution kernel convolution layer, between the 64-convolution kernel convolution layer and the 128-convolution kernel convolution layer, between the 128-convolution kernel convolution layer and the 128-convolution kernel local connection layer, and after the 128-convolution kernel local connection layer. The outputs of the 32-convolution kernel convolution layer, the 64-convolution kernel convolution layer, the 128-convolution kernel convolution layer, and the 128-convolution kernel local connection layer are first input into a linear rectification function for processing. The output of the linear rectification function is then processed by the BN algorithm as the input of the 64-convolution kernel convolution layer, the 128-convolution kernel convolution layer, the 128-convolution kernel local connection layer, and the final output of the decomposition part; There is a linear rectification function connected between the 512-convolution kernel fully connected layer and the 256-convolution kernel fully connected layer, between the 256-convolution kernel fully connected layer and the 128-convolution kernel fully connected layer, and between the 128-convolution kernel fully connected layer and the 64-convolution kernel fully connected layer. A SoftMax classifier is connected after the 64-convolution kernel fully connected layer. The final output of each decomposition part is feature fused in the fusion stage, and the fusion results are processed by a 512-convolution kernel fully connected layer, a 256-convolution kernel fully connected layer, a 128-convolution kernel fully connected layer and a 64-convolution kernel fully connected layer in turn. The outputs of the 512-convolution kernel fully connected layer, the 256-convolution kernel fully connected layer and the 128-convolution kernel fully connected layer are first input into a linear rectification function for processing, and then used as the input of the 256-convolution kernel fully connected layer, the 128-convolution kernel fully connected layer and the 64-convolution kernel fully connected layer respectively. The output of the 64-convolution kernel fully connected layer is finally processed by the SoftMax classifier to output the classification result.

Images