Symbol input method and device based on tapping gesture recognition

Abstract

The invention provides a symbol input method based on knocking gesture recognition, and relates to the technical field of man-machine interaction, and the method comprises the steps: collecting an original knocking gesture signal through a sensor, and carrying out the preprocessing of the original knocking gesture signal, and obtaining a knocking gesture signal; gesture recognition is carried out based on the knocking gesture signal, a knocking gesture is determined, and the knocking gesture comprises a knocking finger combination of a knocking plane and a knocking posture of each knocking finger; and mapping the tapping gesture into a symbol by using a preset symbol mapping table, and displaying visual information corresponding to the symbol on a display device. According to the scheme, the method does not depend on visual feedback, symbol input efficiency and accuracy are improved, and meanwhile rich gesture experience is provided for a user.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a symbol input method and device based on tapping gesture recognition. Background Art

[0002] With the popularity of head-mounted devices such as MR and AR, the realization of spatial computing is becoming possible, which will lead to modern computing methods and deeper human-computer interaction. Symbol input technology is the main input method for interacting with computing devices, and it is a basic requirement when interacting with the spatial environment. In the PC era, physical keyboards have always been the first choice for symbol input. Due to its reasonable keyboard layout, users can quickly and flexibly press their hands and fingers on the operating area. However, in the mobile computing era, common physical keyboards are not easy to carry and difficult to use in daily life anytime and anywhere despite their rich keys. With the popularity of smart mobile devices (such as smartphones, tablets, watches, etc.), new touch screen keyboards have made daily text symbol input more convenient. As we enter the era of spatial computing and move further away from PCs and mobile devices, people's symbol input needs will appear in various environments. In the complex real-life scenarios that spatial computing may face, there is currently no unified and efficient input solution. Existing solutions either have a slow input speed, which hinders the efficiency of users' social learning. Or they require air input technology without tactile feedback, which makes users tired and inefficient. Either visual feedback is required, and complex multi-sensors and computing resources are used to track the hand tapping position, which makes it impossible to input without looking. Although voice input is also a feasible input method, it may bring privacy risks and the problem of attracting the attention of others in quiet occasions. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a symbol input method based on tapping gesture recognition, which does not rely on visual feedback, improves the efficiency and accuracy of symbol input, and provides users with a rich gesture experience.

[0005] The second objective of the present application is to provide a symbol input device based on tapping gesture recognition.

[0006] To achieve the above-mentioned objectives, the first aspect of the present application proposes a symbol input method based on tapping gesture recognition, including: collecting original tapping gesture signals through sensors, and preprocessing the original tapping gesture signals to obtain tapping gesture signals; performing gesture recognition based on the tapping gesture signals to determine the tapping gestures, wherein the tapping gestures include a combination of tapping fingers on the tapping plane and the tapping posture of each tapping finger; mapping the tapping gestures into symbols using a preset symbol mapping table, and displaying visual information corresponding to the symbols on a display device.

[0007] Optionally, in one embodiment of the present application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0008] Optionally, in one embodiment of the present application, preprocessing the original tapping gesture signal to obtain the tapping gesture signal includes:

[0009] Filtering technology is used to remove random noise from the original tapping gesture signal;

[0010] Data normalization is used to scale the original tapping gesture signals to the same scale;

[0011] The time series analysis algorithm is used to perform time series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the tapping action moment.

[0012] Optionally, in one embodiment of the present application, performing gesture recognition based on the tap gesture signal to determine the tap gesture includes:

[0013] The tap gesture signal is input into a neural network architecture based on multimodal sensor data, and the tap gesture is output, wherein the neural network architecture is constructed, including:

[0014] The neural network architecture adopts an encoder-decoder structure;

[0015] In data encoding, a multi-label method is used to represent the tapping fingers. If any finger touches the plane, the corresponding label is set to 1, otherwise, it is set to 0;

[0016] Training neural network architectures, including:

[0017] Collect a data set and train the neural network architecture through the data set, where the data set includes negative class samples. The training process includes:

[0018] During training, a decoding branch is added to the neural network architecture, so that the neural network generates finger postures through the finger posture prediction branch, generates a tapping finger combination through the tapping finger prediction branch, and generates a heat map representation of the contact area through the added decoding branch, and performs display supervision training on the finger contact parts through the heat map representation of the contact area;

[0019] During training, the finger posture loss of negative samples is masked in the optimization function, and the tapping finger combination loss is trained normally.

[0020] Optionally, in one embodiment of the present application, the symbols in the preset symbol mapping table correspond to the input of the input device, and the input of the input device includes letters, numbers, punctuation marks, and functional instructions.

[0021] To achieve the above-mentioned purpose, a second embodiment of the present invention provides a symbol input device based on tapping gesture recognition, comprising:

[0022] A gesture signal acquisition module, used to acquire original tapping gesture signals through sensors, and pre-process the original tapping gesture signals to obtain tapping gesture signals;

[0023] A gesture recognition module, configured to perform gesture recognition based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes a combination of tapping fingers on the tapping plane and a tapping posture of each tapping finger;

[0024] The display module is used to map the tapping gestures into symbols using a preset symbol mapping table, and display visual information corresponding to the symbols on a display device.

[0025] Optionally, in one embodiment of the present application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0026] Optionally, in one embodiment of the present application, preprocessing the original tapping gesture signal to obtain the tapping gesture signal includes:

[0027] Filtering technology is used to remove random noise from the original tapping gesture signal;

[0028] Data normalization is used to scale the original tapping gesture signals to the same scale;

[0029] The time series analysis algorithm is used to perform time series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the tapping action moment.

[0030] Optionally, in one embodiment of the present application, the gesture recognition module is specifically used to:

[0031] The tap gesture signal is input into a neural network architecture based on multimodal sensor data, and the tap gesture is output, wherein the neural network architecture is constructed, including:

[0032] The neural network architecture adopts an encoder-decoder structure;

[0033] In data encoding, a multi-label method is used to represent the tapping fingers. If any finger touches the plane, the corresponding label is set to 1, otherwise, it is set to 0;

[0034] Training neural network architectures, including:

[0035] Collect a data set and train the neural network architecture through the data set, where the data set includes negative class samples. The training process includes:

[0036] During training, a decoding branch is added to the neural network architecture, so that the neural network generates finger postures through the finger posture prediction branch, generates a tapping finger combination through the tapping finger prediction branch, and generates a heat map representation of the contact area through the added decoding branch, and performs display supervision training on the finger contact parts through the heat map representation of the contact area;

[0037] During training, the finger posture loss of negative samples is masked in the optimization function, and the tapping finger combination loss is trained normally.

[0038] Optionally, in one embodiment of the present application, the symbols in the preset symbol mapping table correspond to the input of the input device, and the input of the input device includes letters, numbers, punctuation marks, and functional instructions.

[0039] The symbol input method and device based on tapping gesture recognition in the embodiment of the present application recognizes tapping gestures in any plane including finger combinations and finger postures through a neural network architecture, realizing a technical solution for typing input and command control in any plane without visual feedback. The present embodiment does not rely on visual feedback, and the operation is natural and intuitive. It can perform tapping input on any plane, and introduces a more fine-grained gesture set, realizing a richer symbol mapping scheme to enhance the user's symbol input application experience. Compared with the ordinary method of identifying finger combinations, the embodiment of the present application innovatively introduces finger postures to assist in improving the representation of the finger combination space, thereby doubling the definition space of the tapping gesture.

[0040] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A schematic diagram of a flow chart of a method for symbol input based on tapping gesture recognition provided in the first embodiment of the present application;

[0043] Figure 2This is an example diagram of a predefined tap gesture set in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the training process of a neural network architecture based on multimodal sensor data according to an embodiment of the present application;

[0045] Figure 4 This is an example diagram of a tapping gesture symbol mapping table according to an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of a symbol input device based on tapping gesture recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0048] The following describes a method and device for symbol input based on tapping gesture recognition according to an embodiment of the present application with reference to the accompanying drawings.

[0049] Figure 1 A flowchart of a symbol input method based on tapping gesture recognition provided in Example 1 of the present application.

[0050] like Figure 1 As shown, the symbol input method based on tapping gesture recognition includes the following steps:

[0051] Step 101, collecting an original tapping gesture signal through a sensor, and preprocessing the original tapping gesture signal to obtain a tapping gesture signal;

[0052] In this embodiment, by integrating various types of sensors such as image sensors, depth sensors, motion sensors, etc. at different positions of the wristband, signals of different tapping gestures of the user are collected (which may be a single complete hand image, multiple continuous action images, time series data of the motion sensor during the entire movement process, etc.). Among them, the collected image signal may include images of the upper or lower position of the hand, etc., which depends on the wearing position of the image acquisition sensor (including but not limited to the wrist, arm, head, etc.), requiring the image sensor to completely capture all tapping fingers when the user performs the tapping gesture action.

[0053] In this embodiment, the preprocessing process includes:

[0054] Filtering techniques are used to remove random noise from signals such as images and motion sensors.

[0055] Since the signals of motion sensors, image sensors, etc. are unevenly distributed in multiple dimensions, data normalization is used to scale the signals to the same scale.

[0056] The time series analysis algorithm is used to perform time series segmentation on the gesture signal, and the image signal at the actual tapping action moment is determined for use in the subsequent fine-grained tapping gesture recognition step. Specifically, the process includes:

[0057] When the user taps the flat surface with his finger, the time series analysis algorithm obtains the processed tapping gesture image. When there are multiple frames of continuous tapping images, the sequence is divided more finely to determine the image frames in the gesture action sequence where the finger image is clear and the action is complete and obvious.

[0058] Step 102, performing gesture recognition based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes a tapping finger combination of the tapping plane and a tapping posture of each tapping finger;

[0059] In this embodiment, the gesture recognition process includes:

[0060] The key features of the finger contacting the flat surface in the spatial domain or frequency domain of the gesture (such as fingertip position, flat surface positioning, finger contour deformation degree, image intensity change, image frequency domain response, etc.) are extracted from the processed tapping gesture signal. In addition, deep feature extraction operators (including but not limited to convolutional networks, residual networks, sampling modules, etc.) can also be used to perform deep feature extraction on the original signal or feature signal, thereby extracting deep and subtle features of the gesture that are difficult to directly discover.

[0061] A pattern recognition algorithm is applied to classify the input signal to recognize and distinguish various gestures in a predefined fine-grained gesture set, which mainly consists of multiple finger combinations and different finger postures. The predefined tap gesture set is shown in Figure 2 , which is defined by three finger postures and all finger combinations.

[0062] When studying discrete finger tapping combinations and gestures, accurate calibration of contact positions is crucial. The contact position contains rich regional information and is the core feature for gesture classification and discrimination. Therefore, in order to enhance the model's attention to the contact area, a dedicated supervision branch is designed in the pattern recognition algorithm to extract and enhance the information of the contact area, ensuring that the neural network focuses on key contact area features rather than being disturbed by image texture or other irrelevant environmental factors. Specifically, a neural network architecture based on multimodal sensor data is proposed. The architecture implements feature representation learning of input data through an encoding-decoding module to predict finger combinations and gestures. At the same time, in order to explicitly supervise the feature extraction of the contact area, a dedicated decoding branch is added to the network, which generates a heat map representation of the contact area for explicit supervision of the finger contact position. This can significantly improve the weight of the contact area in the feature extraction process, thereby optimizing the representation ability of the neural network. This design effectively improves the network's ability to capture the features of tapping gestures, reduces the interference of environmental factors, and improves the robustness and generalization ability of the model by combining explicit contact area supervision on the basis of multimodal data fusion. The specific training process is as follows Figure 3 shown.

[0063] In addition, this embodiment also needs to collect negative samples in the data set to cover various complex but non-tapping gestures in the real world. This design aims to improve the robustness and discrimination ability of the model in a variety of environments, so as to effectively distinguish non-tapping gestures that may appear in real scenes. In data encoding, a multi-label method is used to represent the finger state. Specifically, each finger is marked as 1 if it is in contact with the plane, otherwise it is 0. This encoding method can accurately describe the contact state of the finger and provide rich feature information for the model. On the other hand, since the negative samples do not involve tapping actions, their finger posture information cannot be clearly defined. During the model training process, in order to avoid invalid supervision interfering with the optimization process, the finger posture loss of the negative samples is shielded in the optimization function, and only the neural network branch for finger combination prediction is trained normally. This strategy ensures that the model can focus on finger combination prediction while reducing the impact of posture classification uncertainty caused by negative samples on the overall training.

[0064] According to the pattern recognition result, the combination of finger tapping in the current pressing gesture (for example, the thumb, middle finger, and ring finger are pressed at the same time) and the finger tapping posture (for example, the pressing finger tip taps the desktop) are judged (for example, quantified into three intervals: the angle between the tapping finger and the desktop is less than 90 degrees, approximately equal to 90 degrees, and greater than 90 degrees). Of course, the finger tapping posture in which the finger tip taps the desktop is only an example, and the specific tapping position needs to be adjusted accordingly according to the user's needs and habits. Specifically, the user can define and divide the entire gesture space according to his own tapping needs, and can choose different areas of the finger to tap, which does not only include the front posture of the hand, but also includes various possible tapping contact areas such as nails and finger sides. In addition, it is also possible to support the recognition of fine-grained tapping gestures of multiple fingers of both hands by wearing multiple sensors in multiple different positions (including but not limited to wrists, arms, heads, etc.) or combining external image sensing devices (including but not limited to mobile phones, computers, etc.).

[0065] Step 103: Map the tapping gesture to a symbol using a preset symbol mapping table, and display visual information corresponding to the symbol on a display device.

[0066] In this embodiment, the symbol mapping process includes:

[0067] According to the identified fine-grained tapping gesture, a preset symbol mapping table is used to map it to a symbol. When performing matching mapping, a fine-grained tapping gesture composed of a finger tapping combination and a finger tapping posture can be used as input and mapped to various possible symbols. It should be further explained that the symbol mapping method is related to the size of the contact area. If the contact plane is small and only supports a few fingers instead of all finger inputs, for example, the contact plane can only accommodate three fingers, then only less than three fingers and continuous finger combinations are considered for key mapping. If the contact plane is narrow and only supports contact with specific finger postures, fingers with other finger postures can be ignored, and it is degraded to key mapping only based on finger combinations. If the finger contact area is larger, allowing more fingers and more postures to tap at the same time, a more refined mapping division can be performed, and there are more fine-grained tapping gestures to choose from, from which a symbol mapping method that is more in line with the user's special needs and ergonomic can be selected.

[0068] There are infinite possibilities for symbol mapping. The following is an example of a possible text input mapping method under a larger contact area. Figure 4. The recognizable fine-grained tapping gestures can define letters arbitrarily. Note that the mapping layout is not fixed, and users can map letters according to their own accustomed and comfortable gestures. First, the comfort of single letter tapping is considered, so letters are mapped for 5 single fingers and 3 finger gestures, and then the remaining letters are mapped using two-finger and three-finger gestures in turn. This embodiment sorts the letters by frequency (i.e., the average number of times a letter appears in written language) from high to low, and uses single-finger, two-finger, and three-finger gestures for mapping in turn.

[0069] In addition, in order to realize input error correction and association functions, the statistical model of natural language can also be used to correct and associate the input symbol combination.

[0070] also, Figure 4 Only some symbols are listed. You can also use the fine tapping gestures composed of all other finger combinations and three tapping postures to define other symbols and command control mappings.

[0071] In this embodiment, the display process includes:

[0072] The input is output in real time through various device screens (including but not limited to VR / AR head-mounted display devices, computers, mobile phones, tablets, watches) through visualization of graphical interfaces (displaying symbol mapping results), function activation prompt sounds, vibrations, and other methods. The user input content can be presented in the form of text, symbols, images, etc., and the displayed content can be dynamically updated according to real-time changes in user interaction.

[0073] In this embodiment, the display process further includes:

[0074] During the symbol input process, the specific position of the input symbol is located, and the accuracy and convenience of input in various symbol input application scenarios are comprehensively improved.

[0075] The symbol input method based on tap gesture recognition in the embodiment of the present application collects gesture signals through accelerometers, gyroscopes, magnetometers, light sensors, touch sensors, image sensors, depth sensors, etc. The combination of these sensors can effectively capture multi-dimensional data of gestures and provide more accurate gesture recognition capabilities. At the same time, by integrating data from different sensors, the robustness and adaptability of the system can be improved to cope with diverse usage scenarios; the signal is pre-processed through algorithms such as frame difference, data denoising, normalization processing, illumination contrast adjustment, edge detection, and data fusion; when recognizing fine tap gestures, feature extraction is performed through algorithms such as deep learning descriptors, scale-invariant feature transform (SIFT), time series analysis, optical flow method, and short-time Fourier transform (STFT) based on frequency domain analysis, which can extract key features from complex gesture data, capture subtle changes in gestures, and enhance recognition accuracy and robustness; tap gesture symbol mapping is achieved through a customized symbol mapping relationship, effectively enhancing the user experience.

[0076] The hardware carrier of this embodiment is suitable for various types of terminal electronic devices, including but not limited to mobile phones, electronic watches, tablet computers, personal computers, smart glasses, VR headsets and other devices. In addition, the hardware carrier can also be applied to Internet of Things devices, smart home systems, vehicle-mounted electronic devices, and medical monitoring instruments. By combining with different types of terminal devices, the present invention can provide wide compatibility and flexibility to meet the user's symbol input and usage needs in various scenarios. Whether in daily life, sports and fitness, entertainment interaction, or in professional application fields, the hardware carrier can give full play to its performance advantages and bring users a more convenient and efficient experience.

[0077] In order to implement the above embodiment, the present application also proposes a symbol input device based on tapping gesture recognition.

[0078] Figure 5 A schematic diagram of the structure of a symbol input device based on tapping gesture recognition provided in an embodiment of the present application.

[0079] like Figure 5 As shown, the symbol input device based on tapping gesture recognition includes:

[0080] A gesture signal acquisition module, used to acquire original tapping gesture signals through sensors, and pre-process the original tapping gesture signals to obtain tapping gesture signals;

[0081] A gesture recognition module, configured to perform gesture recognition based on the tapping gesture signal to determine the tapping gesture, wherein the tapping gesture includes a combination of tapping fingers on the tapping plane and a tapping posture of each tapping finger;

[0082] The display module is used to map the tapping gestures into symbols using a preset symbol mapping table, and display visual information corresponding to the symbols on a display device.

[0083] Optionally, in one embodiment of the present application, the sensor includes an image sensor, a depth sensor, and a motion sensor.

[0084] Optionally, in one embodiment of the present application, preprocessing the original tapping gesture signal to obtain the tapping gesture signal includes:

[0085] Filtering technology is used to remove random noise from the original tapping gesture signal;

[0086] Data normalization is used to scale the original tapping gesture signals to the same scale;

[0087] The time series analysis algorithm is used to perform time series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the tapping action moment.

[0088] Optionally, in one embodiment of the present application, the gesture recognition module is specifically used to:

[0089] The tap gesture signal is input into a neural network architecture based on multimodal sensor data, and the tap gesture is output, wherein the neural network architecture is constructed, including:

[0090] The neural network architecture adopts an encoder-decoder structure;

[0091] In data encoding, a multi-label method is used to represent the tapping fingers. If any finger touches the plane, the corresponding label is set to 1, otherwise, it is set to 0;

[0092] Training neural network architectures, including:

[0093] Collect a data set and train the neural network architecture through the data set, where the data set includes negative class samples. The training process includes:

[0094] During training, a decoding branch is added to the neural network architecture, so that the neural network generates finger postures through the finger posture prediction branch, generates a tapping finger combination through the tapping finger prediction branch, and generates a heat map representation of the contact area through the added decoding branch, and performs display supervision training on the finger contact parts through the heat map representation of the contact area;

[0095] During training, the finger posture loss of negative samples is masked in the optimization function, and the tapping finger combination loss is trained normally.

[0096] Optionally, in one embodiment of the present application, the symbols in the preset symbol mapping table correspond to the input of the input device, and the input of the input device includes letters, numbers, punctuation marks, and functional instructions.

[0097] It should be noted that the aforementioned explanation of the embodiment of the symbol input method based on tapping gesture recognition is also applicable to the symbol input device based on tapping gesture recognition in this embodiment, and will not be repeated here.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0099] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0100] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0101] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0102] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0103] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0105] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A symbol input method based on tapping gesture recognition, characterized in that: include: Collecting an original tapping gesture signal through a sensor, and preprocessing the original tapping gesture signal to obtain a tapping gesture signal; Performing gesture recognition based on the tapping gesture signal to determine a tapping gesture, wherein the tapping gesture includes a tapping finger combination of the tapping plane and a tapping posture of each tapping finger; The tapping gesture is mapped into a symbol using a preset symbol mapping table, and visual information corresponding to the symbol is displayed on a display device.

2. The method according to claim 1, characterized in that The sensors include image sensors, depth sensors, and motion sensors.

3. The method according to claim 1, characterized in that The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering technology is used to remove random noise from the original tapping gesture signal; Data normalization is used to scale the original tapping gesture signals to the same scale; The time series analysis algorithm is used to perform time series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the tapping action moment.

4. The method according to claim 1, characterized in that Performing gesture recognition based on the tap gesture signal to determine the tap gesture includes: The tap gesture signal is input into a neural network architecture based on multimodal sensor data, and a tap gesture is output, wherein the neural network architecture is constructed, including: The neural network architecture adopts an encoder-decoder structure; In data encoding, a multi-label method is used to represent the tapping fingers. If any finger touches the plane, the corresponding label is set to 1, otherwise, it is set to 0; Training the neural network architecture, comprising: Collecting a data set, and training the neural network architecture through the data set, wherein the data set includes negative class samples, and the training process includes: During training, a decoding branch is added to the neural network architecture, so that the neural network generates finger postures through the finger posture prediction branch, generates a tapping finger combination through the tapping finger prediction branch, and generates a heat map representation of the contact area through the added decoding branch, and performs display supervision training on the finger contact parts through the heat map representation of the contact area; During training, the finger posture loss of negative samples is masked in the optimization function, and the tapping finger combination loss is trained normally.

5. The method according to claim 1, characterized in that The symbols in the preset symbol mapping table correspond to the input of the input device, and the input of the input device includes letters, numbers, punctuation marks, and function instructions.

6. A symbol input device based on tapping gesture recognition, characterized in that: include: A gesture signal acquisition module, used to acquire original tapping gesture signals through sensors, and pre-process the original tapping gesture signals to obtain tapping gesture signals; a gesture recognition module, configured to perform gesture recognition based on the tapping gesture signal to determine a tapping gesture, wherein the tapping gesture includes a combination of tapping fingers on the tapping plane and a tapping posture of each tapping finger; The display module is used to map the tapping gesture into symbols using a preset symbol mapping table, and display visual information corresponding to the symbols on a display device.

7. The device according to claim 6, characterized in that The sensors include image sensors, depth sensors, and motion sensors.

8. The device according to claim 6, characterized in that The preprocessing of the original tapping gesture signal to obtain the tapping gesture signal includes: Filtering technology is used to remove random noise from the original tapping gesture signal; Data normalization is used to scale the original tapping gesture signals to the same scale; The time series analysis algorithm is used to perform time series segmentation on the original tapping gesture signal to determine the tapping gesture signal at the tapping action moment.

9. The device according to claim 6, characterized in that The gesture recognition module is specifically used for: The tap gesture signal is input into a neural network architecture based on multimodal sensor data, and a tap gesture is output, wherein the neural network architecture is constructed, including: The neural network architecture adopts an encoder-decoder structure; In data encoding, a multi-label method is used to represent the tapping fingers. If any finger touches the plane, the corresponding label is set to 1, otherwise, it is set to 0; Training the neural network architecture, comprising: Collecting a data set, and training the neural network architecture through the data set, wherein the data set includes negative class samples, and the training process includes: During training, a decoding branch is added to the neural network architecture, so that the neural network generates finger postures through the finger posture prediction branch, generates a tapping finger combination through the tapping finger prediction branch, and generates a heat map representation of the contact area through the added decoding branch, and performs display supervision training on the finger contact parts through the heat map representation of the contact area; During training, the finger posture loss of negative samples is masked in the optimization function, and the tapping finger combination loss is trained normally.

10. The device according to claim 6, characterized in that The symbols in the preset symbol mapping table correspond to the input of the input device, and the input of the input device includes letters, numbers, punctuation marks, and function instructions.

Images