A finger posture recognition processing system and method based on use of a keyboard

Abstract

The application discloses a finger posture recognition processing system and method based on keyboard use, relates to the technical field of finger posture recognition processing, and comprises a data acquisition module, a data processing module, a data transmission modeling module, a model processing module and a character recognition output module. By combining a wearable device with artificial intelligence, a new type of virtual keyboard which is convenient to carry and flexible to use is designed. With the aid of an algorithm, the virtual keyboard can adapt to the fingering habits of users, is simple and comfortable to use, effectively reduces user input errors, and simultaneously proposes a finger posture recognition algorithm based on a convolutional neural network. The algorithm can recognize user input with less dimensions, has low power consumption, fast recognition speed and high accuracy, effectively reduces the model parameter quantity and volume by combining a polynomial attenuation function with dynamic pruning, and improves the accuracy and calculation speed of the model.

Description

Technical Field

[0001] This application relates to the field of finger posture recognition and processing technology, and in particular to a system and method for finger posture recognition and processing when using a keyboard. Background Technology

[0002] Traditional physical keyboards are bulky and inconvenient to carry, hindering the comfort and portability of human-computer interaction. Virtual keyboards are software components that lack physical keys, primarily relying on photoelectric sensors, gesture recognition, and touchpads to capture the user's input intent and achieve keyboard functionality. However, current technologies still have the following shortcomings;

[0003] Existing technologies reduce the usable display area of ​​mobile devices because touchscreens occupy a portion of the device screen. When the touchscreen is small, the keyboard keys are also small in size and spacing, making them prone to accidental presses and reducing input efficiency, resulting in a poor user experience. Projection-based virtual keyboards have high requirements for the usage environment. First, they need a flat surface with good reflectivity to project the keyboard; second, they need relatively suitable lighting to ensure the recognition efficiency of the projected keyboard. These keyboards generally use a camera to detect the user's finger movements. When the user presses a virtual key, the camera captures the finger movement and transmits the signal to a processing unit, which uses algorithms to convert the signal into keyboard characters. Because image recognition is required, this requires high computing power and consumes a lot of power. Existing virtual keyboards based on wearable devices are generally large in size and have low recognition rates due to technological limitations, resulting in a poor user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a system and method for recognizing and processing finger postures when using a keyboard, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a finger posture recognition and processing system based on keyboard usage, comprising:

[0006] Data acquisition module: used to detect the user's finger posture when typing, and obtain the finger posture information of each finger during the character input process;

[0007] Data processing module: used to process the finger posture information of each finger during the character input process to obtain a modelable data packet corresponding to the character input process;

[0008] Data transmission modeling module: used to transmit and model the modelable data packets corresponding to the character input process, and obtain the data model corresponding to the character input process;

[0009] Model processing module: used to perform model pruning and model transformation on the data model corresponding to the character input process, to obtain the finger posture recognition model during the character input process;

[0010] Character recognition output module: used to perform character recognition on the finger posture recognition model during the character input process, obtain the character recognition result corresponding to the character input, and output the character recognition result corresponding to the character input through a mobile terminal device.

[0011] In the preferred embodiment of this application, the data acquisition module is executed as follows:

[0012] Users type using standard typing techniques while wearing a wearable device. The wearable device uses a pre-installed MEMS sensor to detect the user's finger posture information, obtaining the finger posture information of each finger during character input. The finger posture information of each finger includes the linear acceleration of each fingertip, the angular velocity of each fingertip, the angle of each fingertip, the spatial orientation of each fingertip, and the spatial orientation of each fingertip.

[0013] It's important to note that the accelerometer in a MEMS sensor is used to measure the acceleration of the finger, specifically the linear acceleration change of the finger. By monitoring the finger's acceleration, the movement state of the finger can be determined, including whether it is stationary, accelerating, decelerating, or suddenly changing direction.

[0014] (2) A gyroscope is used to measure the angular velocity and angle changes of the fingers. It can detect the rotation, flipping, and rotational motion of the fingers in any direction. By monitoring the angular velocity and angle changes of the fingers, the posture and direction of the fingers can be determined.

[0015] (3) When collecting information on finger movement, a magnetometer may be used to help determine the spatial orientation and posture of the finger.

[0016] In the preferred embodiment of this application, the data processing module is executed as follows:

[0017] The wearable device processes the finger posture information of each finger during the character input process. When the acceleration of any fingertip exceeds a preset acceleration threshold, a data collection mechanism is triggered to collect the finger posture information of each finger. When the amount of data collected reaches a preset data volume threshold, the collected data is packaged to obtain a modelable data package corresponding to the character input process.

[0018] In the preferred embodiment of this application, the data transmission module is executed as follows:

[0019] The modelable data packets corresponding to the character input process are transmitted via the Bluetooth module of the wearable device.

[0020] The computer device receives the modelable data packet corresponding to the character input process via Bluetooth, and performs data parsing, data training, and data modeling on the modelable data packet to obtain the data model corresponding to the character input process.

[0021] The computer device transmits the data model corresponding to the character input process via Bluetooth, and the mobile device receives the data model corresponding to the character input process via Bluetooth.

[0022] In the preferred embodiment of this application, the model processing module is executed as follows:

[0023] The data model corresponding to the character input process is pruned by removing weights that have little impact on performance to reduce the number of parameters, reduce the model size and computational complexity, and obtain the data model to be transformed for the character input process.

[0024] The TensorFlow Lite converter transforms the data model to be transformed corresponding to the character input process, resulting in a TensorFlow Lite-compatible model format, which is denoted as the finger posture recognition model corresponding to the character input process.

[0025] In the preferred embodiment of this application, the character recognition output module is executed as follows:

[0026] The finger posture recognition model corresponding to the character input process is classified to obtain the part information corresponding to the finger posture recognition model corresponding to the character input process. The part information refers to whether the finger posture recognition model corresponding to the character input process corresponds to the left hand or the right hand.

[0027] A GRU neural network is constructed. Based on the part information corresponding to the finger posture recognition model for the character input process, the model is trained to obtain a classification array corresponding to the finger posture recognition model for the character input process. The classification array includes the similarity between the finger posture recognition model for the character input process and the standard finger posture recognition model for each character in the input process. The standard finger posture recognition model with the highest similarity is selected, and the character corresponding to the standard finger posture recognition model with the highest similarity is selected and denoted as the character corresponding to the finger posture recognition model for the character input process. The character corresponding to the finger posture recognition model for the character input process is denoted as the character recognition result for the character input, and the character recognition result for the character input is output through a mobile device.

[0028] It's important to note that building a GRU neural network for finger pose recognition is a multi-class classification problem. Users wear devices to input the 26 letters of the English alphabet (A, B, C, etc.). In standard finger placement, the left hand inputs 15 letters, and the right hand inputs 11. Therefore, separate models for the left and right hands are needed. The left-hand model has 15 categories, and the right-hand model has 11 categories. The model outputs an array containing the similarity score for each category; the category with the highest similarity score is the output value.

[0029] To achieve the above objectives, the present invention also provides the following technical solution: a method for recognizing and processing finger postures when using a keyboard, comprising the following steps:

[0030] The finger posture of the user is detected when typing, and the finger posture information of each finger during the character input process is obtained;

[0031] The finger posture information corresponding to the character input process is processed to obtain a modelable data package corresponding to the character input process.

[0032] Data transmission and data modeling are performed on the modelable data packets corresponding to the character input process to obtain the data model corresponding to the character input process.

[0033] The data model corresponding to the character input process is pruned and transformed to obtain the finger posture recognition model during the character input process.

[0034] The finger posture recognition model is used to perform character recognition during the character input process to obtain the character recognition result corresponding to the character input, and the character recognition result corresponding to the character input is output through a mobile terminal device.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This invention combines wearable devices with artificial intelligence to design a novel virtual keyboard that is portable and flexible in use. With the assistance of algorithms, this keyboard can adapt to the user's typing habits, making it simple and comfortable to use and effectively reducing user input errors. It also proposes a finger posture recognition algorithm based on convolutional neural networks. This algorithm can recognize user input using fewer dimensions, has low power consumption, fast recognition speed, and high accuracy. Furthermore, through a dynamic pruning method based on polynomial decay, it effectively reduces the number of model parameters and size, while improving the model's accuracy and computation speed. Attached Figure Description

[0037] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of module connections in an embodiment of the present invention. Detailed Implementation

[0039] Please see Figure 1 The present invention provides a finger posture recognition and processing system based on keyboard use. The system includes a data acquisition module, a data processing module, a data transmission modeling module, a model processing module, and a character recognition output module.

[0040] The data acquisition module is connected to the data processing module, the data processing module is connected to the data transmission modeling module, the data transmission modeling module is connected to the model processing module, and the model processing module is connected to the character recognition output module.

[0041] The data acquisition module is used to detect the user's finger posture when typing, and obtain the finger posture information of each finger during the character input process;

[0042] Furthermore, the specific execution method of the data acquisition module is as follows:

[0043] Users type using standard typing techniques while wearing a wearable device. The wearable device uses a pre-installed MEMS sensor to detect the user's finger posture information, obtaining the finger posture information of each finger during character input. The finger posture information of each finger includes the linear acceleration of each fingertip, the angular velocity of each fingertip, the angle of each fingertip, the spatial orientation of each fingertip, and the spatial orientation of each fingertip.

[0044] The data processing module is used to process the finger posture information of each finger during the character input process to obtain a modelable data packet corresponding to the character input process;

[0045] Furthermore, the specific execution method of the data processing module is as follows:

[0046] The wearable device processes the finger posture information of each finger during the character input process. When the acceleration of any fingertip exceeds a preset acceleration threshold, a data collection mechanism is triggered to collect the finger posture information of each finger. When the amount of data collected reaches a preset data volume threshold, the collected data is packaged to obtain a modelable data package corresponding to the character input process.

[0047] It's important to note that the user wears wearable devices on both hands, with each finger corresponding to a MEMS sensor. When the user types with standard finger placement, the STM32 microprocessor samples sensor data at a frequency of 100Hz. When the value of any accelerometer exceeds a threshold, it indicates that the user is inputting data, triggering the data collection mechanism.

[0048] The data transmission modeling module is used to transmit and model the modelable data packets corresponding to the character input process to obtain the data model corresponding to the character input process.

[0049] Furthermore, the specific execution method of the data transmission module is as follows:

[0050] The modelable data packets corresponding to the character input process are transmitted via the Bluetooth module of the wearable device.

[0051] The computer device receives the modelable data packet corresponding to the character input process via Bluetooth, and performs data parsing, data training, and data modeling on the modelable data packet to obtain the data model corresponding to the character input process.

[0052] The computer device transmits the data model corresponding to the character input process via Bluetooth, and the mobile device receives the data model corresponding to the character input process via Bluetooth.

[0053] The model processing module is used to perform model pruning and model transformation on the data model corresponding to the character input process to obtain the finger posture recognition model during the character input process;

[0054] Furthermore, the specific execution method of the model processing module is as follows:

[0055] The data model corresponding to the character input process is pruned by removing weights that have little impact on performance to reduce the number of parameters, reduce the model size and computational complexity, and obtain the data model to be transformed for the character input process.

[0056] The TensorFlow Lite converter transforms the data model to be transformed corresponding to the character input process, resulting in a TensorFlow Lite-compatible model format, which is denoted as the finger posture recognition model corresponding to the character input process.

[0057] The character recognition output module is used to perform character recognition on the finger posture recognition model during the character input process, obtain the character recognition result corresponding to the character input, and output the character recognition result corresponding to the character input through a mobile terminal device.

[0058] Furthermore, the specific execution method of the character recognition output module is as follows:

[0059] The finger posture recognition model corresponding to the character input process is classified to obtain the part information corresponding to the finger posture recognition model corresponding to the character input process. The part information refers to whether the finger posture recognition model corresponding to the character input process corresponds to the left hand or the right hand.

[0060] A GRU neural network is constructed. Based on the part information corresponding to the finger posture recognition model for the character input process, the model is trained to obtain a classification array corresponding to the finger posture recognition model for the character input process. The classification array includes the similarity between the finger posture recognition model for the character input process and the standard finger posture recognition model for each character in the input process. The standard finger posture recognition model with the highest similarity is selected, and the character corresponding to the standard finger posture recognition model with the highest similarity is selected and denoted as the character corresponding to the finger posture recognition model for the character input process. The character corresponding to the finger posture recognition model for the character input process is denoted as the character recognition result for the character input, and the character recognition result for the character input is output through a mobile device.

[0061] To achieve the above objectives, the present invention also provides the following technical solution: a method for recognizing and processing finger postures when using a keyboard, comprising the following steps:

[0062] The finger posture of the user is detected when typing, and the finger posture information of each finger during the character input process is obtained;

[0063] The finger posture information corresponding to the character input process is processed to obtain a modelable data package corresponding to the character input process.

[0064] Data transmission and data modeling are performed on the modelable data packets corresponding to the character input process to obtain the data model corresponding to the character input process.

[0065] The data model corresponding to the character input process is pruned and transformed to obtain the finger posture recognition model during the character input process.

[0066] The finger posture recognition model is used to perform character recognition during the character input process to obtain the character recognition result corresponding to the character input, and the character recognition result corresponding to the character input is output through a mobile terminal device.

[0067] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A finger posture recognition and processing system based on keyboard usage, characterized in that: include: Data acquisition module: used to detect the user's finger posture when typing, and obtain the finger posture information of each finger during the character input process; Data processing module: used to process the finger posture information of each finger during the character input process to obtain a modelable data packet corresponding to the character input process; Data transmission modeling module: used to perform data transmission and data modeling on the modelable data packets corresponding to the character input process, and obtain the data model corresponding to the character input process; Model processing module: used to perform model pruning and model transformation on the data model corresponding to the character input process, to obtain the finger posture recognition model during the character input process; Character recognition output module: used to perform character recognition on the finger posture recognition model during the character input process, obtain the character recognition result corresponding to the character input, and output the character recognition result corresponding to the character input through a mobile terminal device; The finger posture recognition model corresponding to the character input process is classified to obtain the part information corresponding to the finger posture recognition model corresponding to the character input process. The part information refers to whether the finger posture recognition model corresponding to the character input process corresponds to the left hand or the right hand. A neural network is constructed, and the finger posture recognition model corresponding to the character input process is trained based on the part information of the finger posture recognition model corresponding to the character input process. This yields a classification array corresponding to the finger posture recognition model, which includes the similarity between the finger posture recognition model corresponding to the character input process and the standard finger posture recognition model corresponding to each character in the input process. The standard finger posture recognition model with the highest similarity is selected, and the character corresponding to the standard finger posture recognition model with the highest similarity is selected and denoted as the character corresponding to the finger posture recognition model corresponding to the character input process. The character corresponding to the finger posture recognition model corresponding to the character input process is denoted as the character recognition result corresponding to the character input, and the character recognition result corresponding to the character input is output through a mobile device.

2. The finger posture recognition and processing system based on keyboard usage according to claim 1, characterized in that: The specific execution method of the data acquisition module is as follows: Users type using standard typing techniques while wearing a wearable device. The wearable device uses a pre-installed MEMS sensor to detect the user's finger posture information, obtaining the finger posture information of each finger during character input. The finger posture information of each finger includes the linear acceleration of each fingertip, the angular velocity of each fingertip, the angle of each fingertip, the spatial orientation of each fingertip, and the spatial orientation of each fingertip.

3. The finger posture recognition and processing system based on keyboard usage according to claim 1, characterized in that: The specific execution method of the data processing module is as follows: The wearable device processes the finger posture information of each finger during the character input process. When the acceleration of any fingertip exceeds a preset acceleration threshold, a data collection mechanism is triggered to collect the finger posture information of each finger. When the amount of data collected reaches a preset data volume threshold, the collected data is packaged to obtain a modelable data package corresponding to the character input process.

4. The finger posture recognition and processing system based on keyboard usage according to claim 1, characterized in that: The specific execution method of the data transmission modeling module is as follows: The modelable data packets corresponding to the character input process are transmitted via the Bluetooth module of the wearable device. The computer device receives the modelable data packet corresponding to the character input process via Bluetooth, and performs data parsing, data training, and data modeling on the modelable data packet to obtain the data model corresponding to the character input process. The computer device transmits the data model corresponding to the character input process via Bluetooth, and the mobile device receives the data model corresponding to the character input process via Bluetooth.

5. The finger posture recognition and processing system based on keyboard usage according to claim 1, characterized in that: The specific execution method of the model processing module is as follows: The data model corresponding to the character input process is pruned by removing weights that have little impact on performance to reduce the number of parameters, reduce the model size and computational complexity, and obtain the data model to be transformed for the character input process. The TensorFlow Lite converter transforms the data model to be transformed corresponding to the character input process, resulting in a TensorFlow Lite-compatible model format, which is denoted as the finger posture recognition model corresponding to the character input process.

6. A method for processing finger posture recognition when using a keyboard, applied to the finger posture recognition processing system for using a keyboard as described in any one of claims 1-5, characterized in that: include: The finger posture of the user is detected when typing, and the finger posture information of each finger during the character input process is obtained; Data processing is performed on the finger posture information corresponding to the character input process to obtain a modelable data package corresponding to the character input process; Data transmission and data modeling are performed on the modelable data packets corresponding to the character input process to obtain the data model corresponding to the character input process. The data model corresponding to the character input process is pruned and transformed to obtain the finger posture recognition model during the character input process. The finger posture recognition model is used to perform character recognition during the character input process to obtain the character recognition result corresponding to the character input, and the character recognition result corresponding to the character input is output through a mobile terminal device.

Images