Symbol input method and device based on tap gesture recognition

Abstract

The application provides a symbol input method based on a knocking gesture recognition, and relates to the technical field of human-computer interaction, which comprises the following steps: collecting an original knocking gesture signal through a sensor, and preprocessing the original knocking gesture signal to obtain a knocking gesture signal; performing gesture recognition based on the knocking gesture signal to determine a knocking gesture, wherein the knocking gesture comprises a knocking finger combination on a knocking plane and a knocking posture of each knocking finger; mapping the knocking gesture into a symbol by using a preset symbol mapping table, and displaying visual information corresponding to the symbol on a display device. The application adopting the above scheme does not depend on visual feedback, improves the efficiency and accuracy of symbol input, and provides rich gesture experience for users.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a symbol input method and apparatus based on tapping gesture recognition. Background Technology

[0002] With the popularity of head-mounted devices such as MR and AR, spatial computing is becoming a reality, leading to deeper levels of human-computer interaction and modern computing methods. Symbol input technology, as a primary input method for interacting with computing devices, is a fundamental requirement when interacting with the spatial environment. In the PC era, physical keyboards were always the preferred method for symbol input. Their logical layout allowed users to quickly and deftly press their hands and fingers on the operating area. However, in the mobile computing era, while physical keyboards offer a wealth of keys, they are inconvenient to carry and difficult to use anytime, anywhere. The popularity of smart mobile devices (such as smartphones, tablets, and watches) has made new touchscreen keyboards a more convenient option for everyday text and symbol input. As we move into the era of spatial computing, moving further away from PCs and mobile devices, the need for symbol input will emerge in various environments. However, in the complex real-world scenarios that spatial computing may face, there is currently no unified and efficient input solution. Existing solutions either have slow input speeds, hindering the efficiency of users' social learning, or require non-tactile feedback air input technology, leading to user fatigue and inefficiency. Either visual feedback is required, utilizing complex multi-sensor systems and computing resources to track the hand's tapping position, making it impossible to input without eye contact. While voice input is also a viable method, it may pose privacy risks and attract the attention of others in quiet settings. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first objective of this application is to propose a symbol input method based on tapping gesture recognition, which does not rely on visual feedback, improves the efficiency and accuracy of symbol input, and provides users with a rich gesture experience.

[0005] The second objective of this application is to propose a symbol input device based on tapping gesture recognition.

[0006] To achieve the above objectives, the first aspect of this application proposes a symbol input method based on tapping gesture recognition, comprising: acquiring original tapping gesture signals through a sensor and preprocessing the original tapping gesture signals to obtain tapping gesture signals; performing gesture recognition based on the tapping gesture signals to determine the tapping gestures, wherein the tapping gestures include a combination of tapping fingers on a tapping plane and the tapping posture of each tapping finger; mapping the tapping gestures to symbols using a preset symbol mapping table, and displaying the corresponding visual information of the symbols on a display device.

[0007] Optionally, in one embodiment of this application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0008] Optionally, in one embodiment of this application, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:

[0009] Filtering techniques are used to remove random noise from the original tapping gesture signal;

[0010] Data standardization is used to scale down the original tapping gesture signals to the same size.

[0011] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.

[0012] Optionally, in one embodiment of this application, gesture recognition based on the tapping gesture signal to determine the tapping gesture includes:

[0013] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:

[0014] The neural network architecture uses an encoder-decoder structure;

[0015] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.

[0016] Training neural network architectures, including:

[0017] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:

[0018] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.

[0019] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.

[0020] Optionally, in one embodiment of this application, the symbols in a preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.

[0021] To achieve the above objectives, a second aspect of the present invention provides a symbol input device based on tapping gesture recognition, comprising:

[0022] The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals.

[0023] The gesture recognition module is used to recognize gestures based on tapping gesture signals and determine the tapping gestures, wherein the tapping gestures include the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger.

[0024] The display module is used to map tapping gestures to symbols using a preset symbol mapping table and display the corresponding visual information on the display device.

[0025] Optionally, in one embodiment of this application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0026] Optionally, in one embodiment of this application, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:

[0027] Filtering techniques are used to remove random noise from the original tapping gesture signal;

[0028] Data standardization is used to scale down the original tapping gesture signals to the same size.

[0029] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.

[0030] Optionally, in one embodiment of this application, the gesture recognition module is specifically used for:

[0031] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:

[0032] The neural network architecture uses an encoder-decoder structure;

[0033] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.

[0034] Training neural network architectures, including:

[0035] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:

[0036] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.

[0037] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.

[0038] Optionally, in one embodiment of this application, the symbols in a preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.

[0039] The symbol input method and apparatus based on tapping gesture recognition in this application realizes a technical solution for typing input and command control on any plane without visual feedback by recognizing tapping gestures, including finger combinations and finger postures, through a neural network architecture. This embodiment does not rely on visual feedback, making operation natural and intuitive, allowing tapping input on any plane, and introduces a more granular set of gestures, achieving a richer symbol mapping scheme to enhance the user's symbol input application experience. Compared to ordinary methods for recognizing finger combinations, this application innovatively introduces finger postures to assist in improving the representation of finger combination space, thereby expanding the definition space of tapping gestures exponentially.

[0040] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 This is a flowchart illustrating a symbol input method based on tapping gesture recognition provided in Embodiment 1 of this application.

[0043] Figure 2This is an example diagram of a predefined set of tapping gestures in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the training process of a neural network architecture based on multimodal sensor data according to an embodiment of this application;

[0045] Figure 4 This is an example diagram showing the mapping of tapping gesture symbols according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of a symbol input device based on tapping gesture recognition, provided as an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0048] The symbol input method and apparatus based on tapping gesture recognition according to embodiments of this application are described below with reference to the accompanying drawings.

[0049] Figure 1 This is a flowchart illustrating a symbol input method based on tapping gesture recognition provided in Embodiment 1 of this application.

[0050] like Figure 1 As shown, the symbol input method based on tapping gesture recognition includes the following steps:

[0051] Step 101: Collect the original tapping gesture signal through the sensor, and preprocess the original tapping gesture signal to obtain the tapping gesture signal;

[0052] In this embodiment, multiple types of sensors, such as image sensors, depth sensors, and motion sensors, are integrated at different locations on the wristband to collect signals from different tapping gestures of the user (which can be a single complete hand image, multiple continuous motion images, or time-series data from the motion sensor throughout the movement). The collected image signals can include images from above or below the hand, depending on the wearing position of the image acquisition sensor (including but not limited to the wrist, arm, and head), requiring the image sensor to completely capture all tapping fingers when the user performs the tapping gesture.

[0053] In this embodiment, the preprocessing process includes:

[0054] Filtering techniques are used to remove random noise from signals such as images and motion sensors.

[0055] Because the signals from motion sensors, image sensors, and other sensors are unevenly distributed across multiple dimensions, data standardization is used to scale the signals to the same scale.

[0056] A time-series analysis algorithm is used to perform temporal segmentation on the gesture signal, and the image signal at the actual tapping moment is used for subsequent fine-grained tapping gesture recognition steps. Specifically, this process includes:

[0057] When a user taps a flat surface with their finger, a time-series analysis algorithm acquires a processed tapping gesture image. When multiple consecutive tapping images exist, the sequence is further subdivided to identify image frames in the gesture sequence where the finger image is clear and the movement is complete and obvious.

[0058] Step 102: Perform gesture recognition based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger;

[0059] In this embodiment, the gesture recognition process includes:

[0060] Key features of finger contact with a flat surface in the spatial or frequency domain of the processed tapping gesture signal can be extracted (e.g., fingertip position, flat surface location, degree of finger contour deformation, image intensity changes, image frequency response, etc.). Furthermore, deep feature extraction operators (including but not limited to convolutional networks, residual networks, sampling modules, etc.) can be used to extract deep features from the original or feature signals, thereby revealing subtle, deep gesture features that are difficult to detect directly.

[0061] The application of pattern recognition algorithms classifies the input signal to identify and distinguish various gestures within a predefined set of fine-grained gestures, which mainly consist of combinations of fingers and different finger postures. The predefined set of tapping gestures is shown below. Figure 2 It is defined by three finger postures and all finger combinations.

[0062] Accurate calibration of contact positions is crucial when studying discrete finger tapping combinations and postures. Contact positions contain rich regional information and are core features for gesture classification and discrimination. Therefore, to enhance the model's focus on contact areas, a dedicated supervision branch is designed in the pattern recognition algorithm to extract and strengthen contact area information, ensuring that the neural network focuses on key contact area features rather than being interfered with by image textures or other irrelevant environmental factors. Specifically, a neural network architecture based on multimodal sensor data is proposed. This architecture learns feature representations of the input data through an encoder-decoder module to predict finger combinations and postures. Simultaneously, to explicitly supervise contact area feature extraction, a dedicated decoding branch is added to the network. This branch generates heatmap representations of the contact areas for explicit supervision of finger contact points. This significantly increases the weight of contact areas in the feature extraction process, thereby optimizing the neural network's representational ability. This design, by combining multimodal data fusion with explicit contact area supervision, effectively improves the network's feature capture ability for tapping gestures, reduces interference from environmental factors, and enhances the model's robustness and generalization ability. The specific training process is as follows: Figure 3 As shown.

[0063] Furthermore, this embodiment also requires the collection of negative class samples in the dataset to cover various complex but non-tapping gestures in the real world. This design aims to improve the model's robustness and discrimination ability in diverse environments, thereby effectively distinguishing non-tapping gestures that may occur in real-world scenarios. For data encoding, a multi-label approach is used to represent finger states. Specifically, each finger is marked as 1 if it is in contact with a plane, and 0 otherwise. This encoding method can accurately describe the contact state of the fingers and provide rich feature information for the model. On the other hand, since negative class samples do not involve tapping actions, their finger pose information cannot be clearly defined. During model training, to avoid invalid supervision interfering with the optimization process, the finger pose loss of negative class samples is masked in the optimization function, and only the neural network branch for finger combination prediction is trained normally. This strategy ensures that the model can focus on finger combination prediction while reducing the impact of pose classification uncertainty caused by negative class samples on the overall training.

[0064] Based on pattern recognition results, the system determines the combination of finger tapping in the current pressing gesture (e.g., pressing down with the thumb, middle finger, and ring finger simultaneously) and the finger tapping posture (e.g., tapping the table with the fingertip). This is quantified into three intervals: the angle between the tapping finger and the table is less than 90 degrees, approximately 90 degrees, and greater than 90 degrees. Of course, the fingertip tapping posture is just an example; the specific tapping location needs to be adjusted according to user needs and habits. Specifically, users can define and divide the entire posture space according to their own tapping needs, selecting different areas of the fingers for tapping. This includes not only the front of the hand but also various possible tapping contact areas such as the nails and the sides of the fingers. Furthermore, multiple sensors (including but not limited to both wrists, arms, and head) can be worn in multiple different locations, or external image sensing devices (including but not limited to mobile phones and computers) can be used to support the recognition of fine-grained tapping gestures from multiple fingers of both hands.

[0065] Step 103: Use a preset symbol mapping table to map the tapping gestures to symbols, and display the corresponding visual information on the display device.

[0066] In this embodiment, the symbol mapping process includes:

[0067] Based on the identified fine-grained tapping gestures, a preset symbol mapping table is used to map them to symbols. During symbol mapping, fine-grained tapping gestures, composed of finger tapping combinations and gestures, can be used as input to map various possible symbols. It should be further noted that the symbol mapping method is related to the size of the contact area. If the contact plane is small, supporting only a few fingers instead of all fingers (e.g., the contact plane can only accommodate three fingers), then only consecutive finger combinations of fewer than three fingers are considered for key mapping. If the contact plane is narrow, supporting only specific finger gestures, then other finger gestures can be disregarded, degenerating into key mapping based solely on finger combinations. If the finger contact area is larger, allowing more fingers and gestures to tap simultaneously, a finer mapping division can be performed, resulting in more selectable fine-grained tapping gestures. From these, a symbol mapping method that better suits the user's specific needs and is ergonomic can be chosen.

[0068] There are infinitely many possibilities for symbol mapping. Below is an example of a possible text input mapping method for a relatively large contact area. See the mapping table below. Figure 4The recognizable fine-grained typing gestures can arbitrarily define letters. Note that the mapping layout is not fixed; users can map letters according to their own comfortable and habitual gestures. First, the comfort of typing individual letters was considered, so letter mapping was performed for all five single fingers and three finger gestures. Then, two-finger and three-finger gestures were used to map the remaining letters in turn. In this embodiment, letters are sorted from high to low frequency (i.e., the average number of times a letter appears in written language), and single-finger, two-finger, and three-finger gestures are used for mapping in that order.

[0069] In addition, to achieve input error correction and association functions, statistical models of natural language can be used to correct and associate input symbol combinations.

[0070] also, Figure 4 Only a portion of the symbols are listed; other symbols and command controls can be defined using fine-grained tapping gestures composed of all other finger combinations and three tapping postures.

[0071] In this embodiment, the display process includes:

[0072] The system outputs real-time visual feedback information of symbol input through various means such as visualization (displaying symbol mapping results), function activation prompts, vibration, etc., on the graphical interface of various device screens (including but not limited to VR / AR headsets, computers, mobile phones, tablets, and watches). The system can present the user's input in the form of text, symbols, images, etc., and dynamically update the displayed content according to the real-time changes in user interaction.

[0073] In this embodiment, the display process further includes:

[0074] The system locates the specific position of the input symbol during the symbol input process, thereby comprehensively improving the accuracy and convenience of input in various symbol input application scenarios.

[0075] The symbol input method based on tapping gesture recognition in this application collects gesture signals through accelerometers, gyroscopes, magnetometers, light sensors, touch sensors, image sensors, and depth sensors. The combination of these sensors can effectively capture multi-dimensional gesture data, providing more accurate gesture recognition capabilities. Furthermore, by integrating data from different sensors, the robustness and adaptability of the system can be improved to cope with diverse usage scenarios. Preprocessing of the signals is performed using algorithms such as inter-frame differencing, data denoising, normalization, illumination contrast adjustment, edge detection, and data fusion. For fine tapping gesture recognition, feature extraction is performed using algorithms such as deep learning descriptors, scale-invariant feature transform (SIFT), time series analysis, optical flow, and short-time Fourier transform (STFT) based on frequency domain analysis. This allows for the extraction of key features from complex gesture data, capturing subtle changes in gestures, and enhancing the accuracy and robustness of recognition. Finally, a custom symbol mapping relationship is used to implement tapping gesture symbol mapping, effectively enhancing the user experience.

[0076] The hardware carrier of this embodiment is applicable to various terminal electronic devices, including but not limited to mobile phones, smartwatches, tablets, personal computers, smart glasses, VR headsets, etc. Furthermore, the hardware carrier can also be applied to IoT devices, smart home systems, in-vehicle electronic devices, and medical monitoring instruments. By combining with different types of terminal devices, this invention provides broad compatibility and flexibility, meeting users' symbol input needs in various scenarios. Whether in daily life, sports and fitness, entertainment and interaction, or professional applications, the hardware carrier can fully leverage its performance advantages, bringing users a more convenient and efficient experience.

[0077] To implement the above embodiments, this application also proposes a symbol input device based on tapping gesture recognition.

[0078] Figure 5 This is a schematic diagram of a symbol input device based on tapping gesture recognition, provided as an embodiment of this application.

[0079] like Figure 5 As shown, the symbol input device based on tapping gesture recognition includes:

[0080] The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals.

[0081] The gesture recognition module is used to recognize gestures based on tapping gesture signals and determine the tapping gestures, wherein the tapping gestures include the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger.

[0082] The display module is used to map tapping gestures to symbols using a preset symbol mapping table and display the corresponding visual information on the display device.

[0083] Optionally, in one embodiment of this application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0084] Optionally, in one embodiment of this application, the original tapping gesture signal is preprocessed to obtain a tapping gesture signal, including:

[0085] Filtering techniques are used to remove random noise from the original tapping gesture signal;

[0086] Data standardization is used to scale down the original tapping gesture signals to the same size.

[0087] A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.

[0088] Optionally, in one embodiment of this application, the gesture recognition module is specifically used for:

[0089] The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the output is the tapping gesture. The construction of the neural network architecture includes:

[0090] The neural network architecture uses an encoder-decoder structure;

[0091] In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0.

[0092] Training neural network architectures, including:

[0093] A dataset is collected, and the neural network architecture is trained using this dataset. The dataset includes negative class samples. The training process includes:

[0094] During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas.

[0095] During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.

[0096] Optionally, in one embodiment of this application, the symbols in a preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.

[0097] It should be noted that the foregoing explanation of the symbol input method embodiment based on tapping gesture recognition also applies to the symbol input device based on tapping gesture recognition in this embodiment, and will not be repeated here.

[0098] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0099] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0100] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0101] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0102] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0105] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A symbol input method based on tapping gesture recognition, characterized in that, include: The original tapping gesture signal is acquired by a sensor and preprocessed to obtain the tapping gesture signal. Gesture recognition is performed based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes the combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger; The tapping gestures are mapped to symbols using a preset symbol mapping table, and the corresponding visual information is displayed on a display device. The process of recognizing and determining the tapping gesture based on the tapping gesture signal includes: The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the tapping gesture is output. The construction of the neural network architecture includes: The neural network architecture uses an encoder-decoder structure; In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0. Training the neural network architecture includes: A dataset is collected, and the neural network architecture is trained using the dataset. The dataset includes negative class samples. The training process includes: During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas. During training, the finger pose loss for negative class samples is masked in the optimization function, while the finger tapping combination loss is trained normally.

2. The method as described in claim 1, characterized in that, The sensors include image sensors, depth sensors, and motion sensors.

3. The method as described in claim 1, characterized in that, The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering techniques are used to remove random noise from the original tapping gesture signal; Data standardization is used to scale down the original tapping gesture signals to the same size. A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.

4. The method as described in claim 1, characterized in that, The symbols in the preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.

5. A symbol input device based on tapping gesture recognition, characterized in that, include: The gesture signal acquisition module is used to acquire raw tapping gesture signals through sensors and preprocess the raw tapping gesture signals to obtain tapping gesture signals. A gesture recognition module is used to perform gesture recognition based on the tapping gesture signal and determine the tapping gesture, wherein the tapping gesture includes a combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger; The gesture recognition module is specifically used for: The tapping gesture signal is input into a neural network architecture based on multimodal sensor data, and the tapping gesture is output. The construction of the neural network architecture includes: The neural network architecture uses an encoder-decoder structure; In terms of data encoding, a multi-label method is used to represent the tapping fingers. If any finger is in contact with the plane, the corresponding label is set to 1; otherwise, it is set to 0. Training the neural network architecture includes: A dataset is collected, and the neural network architecture is trained using the dataset. The dataset includes negative class samples. The training process includes: During training, a decoding branch is added to the neural network architecture so that the neural network can generate finger poses through the finger pose prediction branch, generate tapping finger combinations through the tapping finger prediction branch, and generate heat map representations of the contact areas through the added decoding branch. The finger contact points are then trained with explicit supervision using the heat map representations of the contact areas. During training, the finger pose loss for negative class samples is masked in the optimization function, while the tapping finger combination loss is trained normally. The display module is used to map the tapping gestures to symbols using a preset symbol mapping table, and to display the corresponding visual information on the display device.

6. The apparatus as claimed in claim 5, characterized in that, The sensors include image sensors, depth sensors, and motion sensors.

7. The apparatus as claimed in claim 5, characterized in that, The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering techniques are used to remove random noise from the original tapping gesture signal; Data standardization is used to scale down the original tapping gesture signals to the same size. A time series analysis algorithm is used to perform time-series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the moment of the tapping action.

8. The apparatus as claimed in claim 5, characterized in that, The symbols in the preset symbol mapping table correspond to the inputs of the input device, and the inputs of the input device include letters, numbers, punctuation marks, and function commands.

Images