A virtual keyboard and keyboard input method based on sensor information fusion

Abstract

The invention discloses a virtual keyboard system, which comprises a PC host computer, a data acquisition board, a six-axis posture sensor, and a curvature sensor; the PC host computer is installed with PC-side software for receiving data information sent back by the data acquisition board and processing the data information; the data acquisition board is used to collect data from the six-axis posture sensors and curvature sensors of two hands, and transmit the data to the PC host computer after interweaving the data; the six-axis posture sensors are installed at the finger tips of both hands, a total of 10, for measuring the bending angle and movement speed of the fingers; there are two curvature sensors, the first curvature sensor is installed at the side of the index finger close to the base of the thumb, for measuring the offset when the index finger strikes the corresponding two columns of keys; the second curvature sensor is installed at the outer side of the wrist on the little finger side, for measuring the degree of wrist swing offset.

Description

Technical Field

[0001] The invention belongs to the technical field of virtual keyboards, and relates to a virtual keyboard and a keyboard input method based on sensor information fusion. Background Art

[0002] As a traditional computer input method, keyboards are widely used in daily life. Traditional keyboards are large in size, not easy to carry, and have certain requirements for the use environment, which limits the scope of keyboard use. At present, the physical keyboards on the market are mainly divided into three categories: mechanical keyboards, membrane keyboards and electrostatic capacitance keyboards. The membrane keyboard is mainly composed of the positive circuit at the top, the negative circuit at the bottom and the insulating material in the middle. When pressed, the upper and lower membranes are in contact and the circuit is turned on. The mechanical keyboard uses a similar metal contact switch. The working principle is to make the contact point turn on or off. It has the characteristics of simple process, high noise and easy maintenance. The electrostatic capacitance keyboard is similar to the membrane keyboard. It uses the change of capacitance to judge the opening and closing of the key. After pressing the key, the air capacitor capacity in the conductive glue changes, thereby realizing the trigger. The whole process can realize the tapping action without physical contact points. For traditional keyboards, although their advantages are different, they all have the disadvantages of large size, inconvenient to carry, and certain requirements for the use environment, which limits the scope of keyboard use. Mechanical keyboards, in particular, although they feel better, they also produce the most noise when used, which can easily cause inconvenience to others. Capacitive keyboards, although they make less noise and are silent, are often more expensive and less cost-effective due to their process characteristics. Summary of the invention

[0003] In order to solve the shortcomings of the prior art, the purpose of the present invention is to provide a virtual keyboard based on sensor information fusion. The virtual keyboard designed by the present invention can simulate the typing process of the user and realize the keyboard input function. The present invention is suitable for occasions where it is inconvenient to use a traditional keyboard, such as realizing fast input on mobile devices such as mobile phones.

[0004] The virtual keyboard proposed by the present invention takes the standard typing gesture as a model, collects the motion data of the finger by installing a cheap six-axis posture sensor at a specific position on the hand, completes the capture and recognition of the finger position and movement, and is assisted by a curvature sensor installed on the outside of the index finger and the outside of the wrist, and respectively senses the horizontal position changes of the index finger and the wrist by detecting the bending degree, so as to improve the positioning accuracy of the finger, and adopts a small sample learning method to obtain the relationship between the letter and the finger posture. Before use, the user uses a conventional keyboard to complete the input of a paragraph of text, and the computer collects the six-axis posture sensor, curvature sensor and the letter data tapped at the fingertips, and establishes the inline relationship between the sensor information and the letter through an algorithm. During use, the user can move the finger according to his own input habits, and the computer analyzes the fingertip gyroscope posture and curvature sensor information, and then infers the blind typing letters to replace the traditional keyboard input.

[0005] The present invention provides a virtual keyboard system, which consists of a PC host computer, a data acquisition board, a six-axis attitude sensor and a curvature sensor.

[0006] The PC host computer is installed with PC software for receiving data information returned by the data acquisition board and processing the data information;

[0007] The data acquisition board collects the six-axis attitude sensor data on the left or right hand fingers and the curvature sensor data on the index finger and wrist, and transmits the data to the PC host computer after interleaving;

[0008] The six-axis attitude sensors are installed at the fingertips of both hands, a total of 10, for measuring the bending angle and movement speed of the fingers;

[0009] There are two curvature sensors. The first curvature sensor is installed on the side of the index finger near the base of the thumb to measure the displacement when the index finger taps the corresponding two columns of keys; the second curvature sensor is installed on the outside of the wrist on the little finger side to measure the degree of wrist swing displacement.

[0010] The six-axis attitude sensor integrates a gyroscope, an accelerometer and a microprocessor, and can quickly solve the real-time motion attitude of each finger corresponding to the sensor, and output the attitude angle and acceleration data of the finger in the three directions of X, Y and Z; the direction forward along the tip of the index finger is defined as the Y-axis direction, the direction parallel to the palm and perpendicular to the Y-axis is defined as the X-axis direction, and the direction perpendicular to the XY plane and upward is the Z-axis direction.

[0011] The first curvature sensor is a sensor covered with a variable resistor film. When the index finger taps different columns of keys corresponding to the fingering, the index finger will rotate along the joint at the root of the index finger, and the angle between the index finger and the base of the thumb will change, causing the first curvature sensor to bend. Different bending angles will change the resistance value of the first curvature sensor. The change in resistance value is used to quantify the horizontal displacement of the index finger, that is, the horizontal rotation angle of the index finger in the X direction.

[0012] The second curvature sensor is a sensor covered with a variable resistance film. When the outer side of the wrist swings, the angle between the palm and the arm will change, causing the second curvature sensor to bend. Different bending angles will change the resistance value of the second curvature sensor to detect the degree of wrist swing deviation.

[0013] The horizontal rotation angle signal of the index finger in the X direction obtained by the first curvature sensor and the front and rear position signal of the finger in the Y axis obtained by the gyroscope are combined to form a feature vector group to determine the key value of the index finger button.

[0014] When the above virtual keyboard system performs input, the input method includes:

[0015] Step 1: During input, the gyroscope sensor in the six-axis attitude sensor collects the bending angle and movement speed of the finger and sends it to the data acquisition board for processing;

[0016] Step 2: The curvature sensor uses the analog-to-digital converter (ADC) interface of the data acquisition board to measure the resistance change caused by bending, and transmits the resistance change to the data acquisition board through the ADC interface. After filtering algorithm, the degree of bending is read.

[0017] Step 3: The data acquisition board takes out the data stored in the structure, reads the filtered ADC voltage data, combines the two data and adds the sensor identification (Sensor0, Sensor1...Sensor9), and then sends it to the PC host computer through the serial peripheral interface SPI;

[0018] Step 4: After receiving the data, the PC host computer classifies and summarizes it according to the sensor identification in the data, completes the recognition of each finger movement posture and positioning, determines the key value tapped by the user, compares it with the pre-prepared dictionary library, and outputs the characters.

[0019] Fingering training is required before the virtual keyboard is officially used. During the training, a paragraph of text is input using a conventional computer keyboard. The text contains 26 letters. When each letter is typed, the angle data of different fingers measured by the six-axis attitude sensor and the curvature data measured by the curvature sensor are recorded. The angle data and curvature data are used as the standard and threshold for subsequent recognition and judgment. The computer records the angle of each finger measured by the six-axis attitude sensor and the curvature of the curvature sensor in the time interval between the two letters before and after the typing.

[0020] In order to suppress the drift of the gyroscope, a state machine is used to represent the keyboard information during the training and recognition process. The six-axis attitude sensor information of the five fingers of the left or right hand and the first curvature sensor information are grouped together to form a state machine based on the current keystroke B. i For state S i , next time B i+1 Tap to state S i+1 During training, record key value B i , B i+1 And the change of state S Δ =S i -S i+1 , generate a dictionary library related to the actual input characters, key values, and state changes, and the dictionary library can be shown in Table 1. When applied, according to the current key value B i Find S corresponding to all possible successor keys Δ The minimum value is the key value entered subsequently.

[0021] Table 1 Character, key value, and state change dictionary

[0022] Start keystroke value The state difference between the two key values ​​from the start to the end End keystroke value A <![CDATA[State difference S of the tapping intervals between two consecutive taps on A Δ > A A <![CDATA[State difference S between two knocks from knock A to knock B Δ > B A <![CDATA[State difference S between two tapping intervals from tapping A to tapping C Δ > C A <![CDATA[State difference S between two tapping intervals from tapping A to tapping D Δ > D … … … Z <![CDATA[State difference S between two knocks from knocking Z to knocking Y Δ > Y Z <![CDATA[State difference S of the tapping interval between two consecutive taps on Z Δ > Z

[0023] The six-axis attitude sensor can also determine keystrokes through the measured acceleration data. When the acceleration of the finger is greater than the threshold T (generally set to 0.6g, 1g is the acceleration of gravity), the finger's movement direction is detected and the acceleration data of the sensor is identified. After the acceleration of the finger decreases rapidly to near 0, a reverse acceleration that increases rapidly is generated. It is determined that the finger has made a keystroke when the acceleration decreases to 0. The acceleration waveform is, for example, Figure 4 As shown, the spatial tilt angles of the five fingers and the curvature of the index finger are recorded at the same time at the moment of tapping.

[0024] In step 1, the data acquisition board receives the data sent by the five six-axis attitude sensors in the order of priority from high to low, i.e., index finger, middle finger, ring finger, pinky finger, and thumb. If the received data register is not empty, the interrupt function is entered. Each piece of data of the six-axis attitude sensor is a fixed-format data frame containing the same head and tail. The interrupt function first looks for the beginning of the data frame, and then reads the acceleration data and angle data in the data frame, and stores them in the acceleration data and angle data structures, respectively. Each of the data structures has three variables, which store the data of the three axes of X, Y, and Z, respectively.

[0025] In step 2, the filtering utilizes the sliding average method, taking N sample values ​​for averaging, and setting N temporary storage areas in the storage area; each time a new data is collected, the data is stored in the Nth position in the temporary storage area, and other data are moved forward by one position, so that the N data are always the latest updated data, reducing the ups and downs of the original sequence.

[0026] The main function of the data acquisition board is a loop function, which will cyclically detect whether the six-axis attitude sensor has data sent to the data acquisition board. If it is detected that data is received, the operation of step three is performed.

[0027] The character output operation process also includes error correction processing, which uses an error correction algorithm to correct each word tapped by the user, collects sensor data generated by the user tapping the word, and quantifies the error probability from two dimensions: time interval and tapping intensity. First, the word will be searched in the dictionary library. If it exists, it means that no error has occurred. If the word does not exist, it is determined to be a tapping error. At this time, the tapping intensity data of each letter of the word will be sorted, and the tapping interval time of each letter will be sorted. Then, the possible error points in the word are judged. For example, if the tapping intensity of a letter is too low and the time interval is decisive, there may be repeated tapping. These error points are corrected by replacing, deleting, and adding characters, and the words generated by the error correction are searched in the dictionary library. Finally, they are sorted according to the frequency of occurrence of each word in the dictionary library, and the words with the most occurrences are output.

[0028] The beneficial effects of the present invention include: fingering learning will be conducted before using the virtual keyboard to ensure that the user's habits are met during use without additional learning costs; the left and right hands can be placed separately when using the virtual keyboard, and no bracket is required to support the placement of the keyboard, reducing the space requirements for using the keyboard. The structure of separating the left and right hands is more conducive to users adjusting the arm posture and reducing fatigue. The virtual keyboard can be made into a form such as gloves, which is easy to fold and carry. The virtual keyboard has no mechanical tapping process, does not generate noise, and is quieter to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a front view of the installation of the virtual keyboard sensor of the present invention.

[0030] Figure 2 It is the standard typing gesture diagram used by the present invention.

[0031] Figure 3 This is a side view of the installation of the first curvature sensor on the index finger side of the virtual keyboard of the present invention.

[0032] Figure 4 It is a knock acceleration waveform diagram of the six-axis attitude sensor of the present invention.

[0033] Figure 5 This is a flow chart of dictionary generation in the training phase of the present invention.

[0034] Figure 6 It is a structural diagram of the virtual keyboard system of the present invention.

[0035] Figure 7 This is a diagram of the installation position of the second curvature sensor on the wrist side of the virtual keyboard of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0037] Figure 1 This is a schematic diagram of the installation position of the sensor of the virtual keyboard on the hand. In the figure, the Y-axis direction is along the tip of the index finger, the X-axis direction is parallel to the palm and perpendicular to the Y-axis, and the Z-axis direction is perpendicular to the XY plane. Sensors 1 to 5 in the figure are six-axis attitude sensors, which integrate high-precision gyroscopes, accelerometers and microprocessors, and can quickly solve the current real-time motion attitude of each finger, and output attitude angles and acceleration data in the three directions of X, Y and Z.

[0038] When a finger taps a key, the finger will swing up and down around the middle joint of the finger. The radius from the top of the finger to the middle joint of the finger is the largest, so the swing distance is the longest and the amplitude of the movement is the most obvious. Therefore, the six-axis attitude sensor is placed at the top of the finger. If the sensor is placed on the side of the top of the finger, it will block the movement of the adjacent fingers and interfere with the tapping. Therefore, the sensor is installed at the nail position of the top of the finger. When the finger is tapping, there will be a tapping action from falling to lifting. Through the six-axis attitude sensor, the tapping action can be quantified as the acceleration data of the sensor.

[0039] This virtual keyboard is mainly used to tap the 26 letter keys. Figure 2 This is a standard keyboard typing gesture. As can be seen from the figure, each finger has its corresponding key area when typing, and the letters typed by each finger can be divided into three rows: upper, middle and lower. Only the index finger of the left and right hand needs to take into account the typing of two columns of letters at the same time. In order to determine the exact position of the index finger's typing, it is necessary to add a detection of the left and right offset of the index finger.

[0040] When the finger taps the three rows of the column area to which it belongs, the finger rotates with the middle joint of the finger as the center, causing the finger to bend. The farther the point is from the middle joint of the finger, the greater the movement amplitude, so the six-axis attitude sensor needs to be pressed Figure 1 Installed at the top of each fingernail, so that the acceleration and angle change of each tap are maximized.

[0041] When a key is pressed, the changes in the keys in the upper, middle and lower rows can be quantified as the changes in the angles produced by the rotation of the sensor along the X-axis (e.g. Figure 3 ). Each row of keys has its corresponding angle change range. According to the actual angle change of the key, the row of the key can be determined. At the same time, the sensor on each finger has its own logo, so by reading the sensor logo, it can be determined which finger the sensor is on, and then according to the angle change of the key. Figure 2 The area that each finger is responsible for determines the column interval of the key being tapped. In this way, all fingers except the index finger determine the exact position of the row and column when tapping, and then determine the letter corresponding to the key tapped by the finger.

[0042] At the same time, when the finger taps each row, the acceleration change of the sensor at the top of the finger corresponds to a different direction. When tapping the top row of keys, the finger remains straight, and the sensor is in a horizontal position. The tapping action mainly causes the change of the Z-axis acceleration. When tapping the bottom row, the sensor moves vertically downward along the Y-axis, and the tapping action mainly causes the Y-axis acceleration to change. When tapping the middle row, the sensor remains tilted, and the tapping will cause the acceleration of both the Z and Y axes to change. At this time, the acceleration changes in these two directions need to be considered to judge the tapping. Combining these three situations, the square and root operation of the Z-axis and Y-axis accelerations can be directly performed to convert the two-dimensional changes into one-dimensional changes. Reduce the amount of data to be processed and process data quickly and in real time. Figure 4 The combined waveform of the knock acceleration shows that at the waveform indicated by the arrow, there are huge troughs and peaks. The acceleration decreases downward first, and then changes direction and increases in the opposite direction when it reaches the inflection point. This feature can provide a basis for the judgment of knock.

[0043] Figure 1The sensor 6 in the figure is a bending sensor. It is a thin film composed of a variable resistor covering the surface of the sensor. Different bending angles will change the resistance value of the sensor, so that it can respond to the bending. The degree of bending can be determined based on the change in the resistance value of the sensor. Figure 2 In the typing gesture, the index finger of each hand has two columns of key areas, so a curvature sensor is added to measure the offset when the index finger hits the two columns of keys. Thereby improving the accuracy of the index finger tapping the keys. When the index finger moves in these two columns, it rotates along the joint at the base of the finger, forming two states: closed and open. There are two angle changes, one is the change in the angle formed by the inner side of the index finger and the middle finger, and the other is the change in the angle between the index finger and the base of the thumb. The angle between the index finger and the middle finger is too small. If the curvature sensor is placed here, the curvature sensor will be folded in half, causing damage to the sensor. Therefore, the curvature sensor is placed on the outside of the index finger. Figure 3 , close to the index finger, when closed, the curvature sensor remains vertical, when open, the sensor bends outward, causing resistance changes, so that the degree of deviation caused by the index finger tapping the two columns of letters can be quantified as the change in the value of the curvature sensor, and the index finger is in which specific column according to the degree of deviation. In addition to the standard typing gesture where each finger controls the corresponding column, some people only use the index finger to type, and rotate the wrist to offset the index finger to complete the tapping of different columns of letters. Figure 1 Sensor 7 is a curvature sensor and is located on the outside of the wrist. Figure 7 The degree of wrist swing deviation can be converted into the sensor value, which is then used to determine which column the index finger is in, and then the tapping position is determined based on the gyroscope data.

[0044] This virtual keyboard needs to be fingered before use. During the learning process, the user wears this device and uses a regular computer keyboard to input a paragraph of text, which contains all 26 English letters. In this process, this virtual keyboard uses this tapping as the standard, records the angle data of different fingers measured by the six-axis attitude sensor and the curvature data measured by the curvature sensor when each letter is tapped, and uses these data as the standard and threshold for subsequent recognition and judgment. At the same time, the acceleration data of the six-axis attitude sensor is used to sense the tapping of the finger. The finger tapping process will have a rapid drop, pause and then lift action. Therefore, when the acceleration of the finger is greater than the threshold T (T can be set to 0.6g, 1g is the gravity acceleration), the direction of movement of the finger is detected and the acceleration data of the sensor is identified. After the acceleration of the finger is rapidly reduced to near the value of 0, a reverse and rapidly increasing acceleration is generated. It is determined that the finger has made a tap when the acceleration is reduced to 0. At the same time, the spatial inclination angle of the five fingers and the curvature of the index finger at the moment of the tapping are recorded.

[0045] In order to suppress the influence of the drift of the gyroscope, a state machine will be used to represent the key information during the learning and subsequent recognition process. The six-axis attitude sensor data at the top of the five fingers of the left hand (or right hand) and the data of the first curvature sensor on the outside of the index finger saved during the tapping are combined into a set of data and recorded as a state. Figure 5 The current tapping state is saved as S i , the next time you tap, save the tapping status as S i+1 , and change the letters and states of the two taps to S Δ =S i -S i+1 Correspondingly, after all letter combinations have completed training, the corresponding state changes S Δ The key values ​​of the two consecutive taps that caused this state change are saved as a dictionary library (as shown in Table 1). When in use, the sensor data at the moment of the last key tap is a state, and the state of this tap is used to calculate the state change caused by this tap. At the same time, the key value output by the last tap is read, and all possible subsequent key combinations of this key value are traversed in the dictionary, and the key value with the state change closest to this state change is found. The key value corresponding to it is the current input key value.

[0046] Figure 6 This is the system structure diagram of one hand of this virtual keyboard. The sensor senses the finger posture and motion data in real time and sends it to the data acquisition board MCU. The six-axis posture sensor sends the data to the data acquisition board through the serial port. The data acquisition board has five serial ports, corresponding to the six-axis posture sensor at the top of each finger. According to the priority order of index finger-middle finger-ring finger-little finger-thumb from high to low, the data sent by the five six-axis posture sensors are received in sequence. Each data of the six-axis posture sensor is a fixed format data frame containing the same head and tail. When the data acquisition board receives the data register that is not empty and enters the interrupt function, it will first find the beginning of the data frame, then read the acceleration data and angle data contained in the data frame, and then store them in the corresponding acceleration data and angle data structures respectively. Each structure has three variables, which store the data of the three axes of X, Y and Z respectively.

[0047] The curvature sensor transmits data through the ADC interface of the data acquisition board, and uses the ADC to sense the change in resistance of the curvature sensor itself caused by bending, thereby realizing the reading of the degree of bending. The directly sampled curvature sensor data contains a lot of noise, which is not conducive to data processing. In order to remove the influence of these noises, filtering is required. This virtual keyboard adopts a sliding average filtering method, which takes N sample values ​​for averaging and opens up a temporary storage area for N data in the storage area. Each new data collected is stored in the Nth position in the temporary storage area, and the other data is moved forward by one position, so that these N data are always the latest updated data. This reduces the up and down fluctuations of the original sequence.

[0048] The main function of the data acquisition board consists of a loop function. Once it is detected that the attitude sensor has data to send to the data acquisition board, the data saved in the structure is taken out, and the filtered ADC voltage data is read once. The two types of data are combined and the sensor identifier (Sensor0, Sensor1...Sensor9) is added, and then sent to the PC host computer through SPI.

[0049] After receiving the data, the host computer will classify and summarize it according to the sensor identification in the data, so as to complete the recognition of the movement posture and positioning of each finger, and determine the key value tapped by the user. At the same time, in order to improve the accuracy of the output of this virtual keyboard, the key value judged for the first time must be corrected. The error correction algorithm will correct each word tapped by the user, collect the sensor data generated by the user tapping the word, quantify the error probability from the two dimensions of time interval and tapping intensity, and then correct the possible errors in the word by replacing, deleting, adding characters, etc. The words generated by the error correction are searched in the dictionary library, and finally sorted according to the frequency of occurrence of each word in the dictionary library, and the words with the most occurrences are output.

[0050] Suppose the input word is X. First, word X will be searched in the word library to find out whether there is the same word in the word library. If there is, it is judged that the input is correct and word X is directly output. If word X is not found in the word library, error correction will be entered. First, the tapping intensity data and tapping time data of word X will be taken out, and then each letter of word X will be sorted according to the tapping intensity and the time interval between two letters. The letter with the lowest tapping intensity will be found, and the new word formed after deleting this letter will be put back into the word library for search; find the position with the longest time interval, put the 26 letters in turn to recombine, and put the generated new word into the word library for search; delete letters at the position with the shortest time interval, combine into a new word and put it into the word library for search; then find the finger used to tap each letter, replace the letter with other letters that may be tapped by this finger, and put the combined new word into the word library for search; all words that can be found in the word library are put into the word buffer area, and finally sort according to the frequency of each word in the word buffer area in the word library, and output the word S with the highest frequency as the word after error correction.

[0051] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A virtual keyboard system, characterized in that: The virtual keyboard system includes a PC host computer, a data acquisition board, a six-axis attitude sensor, and a curvature sensor; The PC host computer is installed with PC software for receiving data information returned by the data acquisition board and processing the data information; The data acquisition board is used to collect data from the six-axis posture sensors and curvature sensors of both hands, and transmit the data to the PC host computer after interleaving; The six-axis attitude sensors are installed at the fingertips of both hands, a total of 10, for measuring the bending angle and movement speed of the fingers; There are two curvature sensors. The first curvature sensor is installed on the side of the index finger near the base of the thumb to measure the displacement when the index finger taps the corresponding two columns of keys. The second curvature sensor is installed on the outside of the wrist on the little finger side to measure the degree of wrist swing displacement. Among them, the data acquisition board receives the data sent by the five six-axis attitude sensors in the order of priority from index finger-middle finger-ring finger-little finger-thumb from high to low. Once the received data register is not empty, the interrupt function is entered. The interrupt function first looks for the beginning of the data frame sent by the six-axis attitude sensor, and then reads the acceleration data and angle data in the data frame, and stores them in the acceleration data and angle data structures respectively.

2. The virtual keyboard system according to claim 1, wherein: The six-axis posture sensor integrates a gyroscope, an accelerometer and a microprocessor, and can quickly solve the real-time motion posture of each finger corresponding to the sensor, and output the posture angle and acceleration data in the three directions of X, Y and Z.

3. The virtual keyboard system according to claim 2, wherein: The direction along the tip of the index finger is defined as the Y-axis direction, the direction parallel to the palm and perpendicular to the Y-axis is defined as the X-axis direction, and the direction perpendicular to the XY plane is defined as the Z-axis direction.

4. The virtual keyboard system according to claim 1, wherein: The first curvature sensor is a sensor covered with a variable resistor film. When the index finger taps different columns of keys corresponding to the fingering, the index finger will rotate along the joint at the base of the index finger, and the angle between the index finger and the base of the thumb will change, causing the first curvature sensor to bend. Different bending angles will change the resistance value of the first curvature sensor. The change in resistance value is used to quantify the horizontal offset of the index finger.

5. The virtual keyboard system according to claim 1, wherein: The second curvature sensor is a sensor covered with a variable resistance film. When the outer side of the wrist swings, the angle between the palm and the arm will change, causing the second curvature sensor to bend. Different bending angles will change the resistance value of the second curvature sensor to detect the degree of wrist swing deviation.

6. A method for inputting using the virtual keyboard system according to any one of claims 1 to 5, characterized in that: The input method comprises the following steps: Step 1: During input, the gyroscope sensor in the six-axis attitude sensor collects the bending angle and movement speed of the finger and sends them to the data acquisition board for processing; the data acquisition board receives the data sent by the five six-axis attitude sensors in the order of priority from index finger-middle finger-ring finger-little finger-thumb from high to low. Once the received data register is not empty, the interrupt function is entered. The interrupt function first finds the beginning of the data frame sent by the six-axis attitude sensor, and then reads the acceleration data and angle data in the data frame, and stores them in the acceleration data and angle data structures respectively; Step 2: The curvature sensor uses an analog-to-digital converter (ADC) interface to measure the resistance change caused by bending, and transmits the resistance change to a data acquisition board through the ADC interface. After filtering by a moving average algorithm, the data is sent to a PC host computer to read the degree of bending. Step 3: The data acquisition board takes out the data stored in the structure, reads the filtered ADC voltage data, combines the two data and adds the sensor identification, and then sends it to the PC host computer through the serial peripheral interface SPI; Step 4: After receiving the data, the PC host computer classifies and summarizes it according to the sensor identification in the data, completes the recognition of each finger movement posture and positioning, determines the key value tapped by the user, compares it with the pre-prepared dictionary library, and outputs the characters.

7. The method according to claim 6, characterized in that Before input, the process also includes a finger learning step. During learning, the virtual keyboard records through sensors the angle data of different fingers measured by the six-axis posture sensor and the curvature data measured by the curvature sensor when each letter is typed when the fingers input 26 letters on a conventional computer keyboard. The angle data and curvature data are used as the standard and threshold for subsequent recognition for judgment.

8. The method according to claim 7, characterized in that The key tapping is determined by the acceleration data measured by the accelerometer integrated in the six-axis attitude sensor. When the acceleration of the finger is greater than the threshold T, the finger's movement direction is detected and the acceleration data of the six-axis attitude sensor is identified. After the acceleration of the finger decreases rapidly to near 0, a rapidly increasing acceleration in the opposite direction is generated. It is determined that the finger has tapped once when the acceleration decreases to 0. At the same time, the spatial inclination angles of the five fingers and the curvature of the index finger at the moment of tapping are recorded.

9. The method according to claim 6, characterized in that Each of the data structures has three variables, which store data of the three axes X, Y, and Z respectively.

10. The method according to claim 6, characterized in that In step 2, the filtering utilizes the sliding average method, taking N sample values ​​for averaging, and setting N temporary storage areas in the storage area; each time a new data is collected, the data is stored in the Nth position in the temporary storage area, and other data are moved forward by one position, so that the N data are always the latest updated data, reducing the ups and downs of the original sequence.

11. The method according to claim 6, characterized in that The main function of the data acquisition board is a loop function, which will cyclically detect whether the six-axis attitude sensor has data sent to the data acquisition board. If it is detected that the data is received, the operation of step three is performed.

12. The method according to claim 6, characterized in that The operation process of outputting characters also includes error correction processing, which uses an error correction algorithm to correct each word typed by the user, collects sensor data generated by the user typing the word, quantifies the error probability from the two dimensions of time interval and tapping intensity, and then corrects the possible error points in the word by replacing, deleting, and adding characters. The words generated by the error correction are searched in the dictionary library, and finally sorted according to the frequency of occurrence of each word in the dictionary library, and the words that appear the most times are output.

13. The method according to claim 6, characterized in that In step 4, the generation of the dictionary library includes the following steps: during training, the current tapping state is saved as S i , the next time you tap, save the tapping status as S i+1 , and change the letters and states of the two taps to S Δ =S i -S i+1 Correspondingly, after all letter combinations have completed training, the corresponding state changes S Δ The key values ​​of the two consecutive taps that caused this state change are saved as a dictionary library.

Images