Character input method, device, electronic device and storage medium
By collecting and identifying gesture vibration signals, it is possible to accurately identify the character information written in gestures without large movement, solving the problem of high complexity of volley handwriting in the prior art, improving input convenience and human-computer interaction Experience.
Patent Information
- Application Number
- CN202011270920.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-13
AI Technical Summary
The existing camera-based volley handwriting technology cannot achieve fine finger writing recognition, and the user requires large movements to write based on gesture movement trajectory tracking, resulting in high input complexity.
By collecting the initial gesture vibration signal generated during gesture execution, the target gesture vibration signal is extracted, and the gesture type is determined through vibration feature recognition, and character input operation based on gesture type is realized.
It can accurately identify the character information written by gestures without large movement, simplify remote input, improve convenience, reduce device development and implementation costs, and enhance human-computer interactive experience.
Smart Images

Figure CN112363622B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of human-computer interaction technology, and in particular to a character input method, device, electronic device and storage medium. Background Art
[0002] With the continuous development of technology, handwriting in the air can be applied to the remote input operation of various smart devices, such as smart glasses, smart TVs, etc. However, the camera-based handwriting in the air cannot achieve fine finger writing recognition, and the method based on gesture movement trajectory tracking also requires the device to move in order to track, resulting in users having to make large movements to write characters. Therefore, it is very necessary to realize the fine realization of handwriting in the air. Summary of the invention
[0003] The embodiments of the present application provide a character input method, device, electronic device and storage medium to achieve refined mid-air writing and character input without the need for customized equipment.
[0004] In a first aspect, an embodiment of the present application provides a character input method, which is applied to an electronic device, and the method includes:
[0005] Determine an initial gesture vibration signal generated during the process of performing air writing of characters through a gesture;
[0006] Extracting a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal;
[0007] The vibration feature recognition is performed on the target gesture vibration signal to determine the gesture type when writing characters in mid-air, so as to perform a character input operation based on the gesture type.
[0008] In a second aspect, an embodiment of the present application further provides a character input device, which is configured in an electronic device, and the device includes:
[0009] An initial signal determination module, used to determine an initial gesture vibration signal generated during the process of performing mid-air writing of characters by gesture;
[0010] A target signal extraction module, used to extract a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal;
[0011] The gesture character input module is used to determine the gesture type when writing characters in the air by performing vibration feature recognition on the target gesture vibration signal, so as to perform a character input operation based on the gesture type.
[0012] In a third aspect, an electronic device is also provided in an embodiment of the present application, including:
[0013] one or more processors;
[0014] A storage device for storing one or more programs;
[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the character input method provided in any embodiment of the present application.
[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a character input method as provided in any embodiment of the present application is implemented.
[0017] In the embodiment of the present application, a character input method is provided. When the user needs to perform remote input for various intelligent electronic devices, the electronic device can determine the initial gesture vibration signal generated by the user in the process of writing characters in the air through gestures, and extract the target gesture vibration signal used to effectively characterize the information of the characters written in the air from the initial gesture vibration signal, and then determine the gesture type when writing characters in the air by identifying the vibration characteristics of the target gesture vibration signal, so as to implement the operation of performing character input based on the gesture type. With the scheme of the present application, the user does not need to move significantly to write characters with gestures, and the character information written by the user's gestures can be accurately identified, which simplifies the complexity of remote input and improves its input convenience. At the same time, it does not need to customize special equipment to realize the recognition of fine finger writing in the air, which reduces the equipment development cost and implementation cost, realizes fine finger writing in the air, and enhances the human-computer interaction experience.
[0018] The above application content is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the drawings to represent the same components. In the drawings:
[0020] Figure 1 is a flow chart of a character input method provided in an embodiment of the present invention;
[0021] Figure 2 is a schematic diagram of a gesture vibration signal during a process of writing characters in mid-air provided in an embodiment of the present invention;
[0022] Figure 3 is a flow chart of another character input method provided in an embodiment of the present invention;
[0023] Figure 4 is a process diagram of detecting a gesture signal provided in an embodiment of the present invention;
[0024] Figure 5 is a partial schematic diagram of an endpoint detection based on frame energy provided in an embodiment of the present invention;
[0025] Figure 6 is a flowchart of another character input method provided in an embodiment of the present invention;
[0026] Figure 7 is a schematic diagram of a video feature after digital gravity removal provided in an embodiment of the present invention;
[0027] Figure 8 is a structural block diagram of a character input device provided in an embodiment of the present invention;
[0028] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present application. It should also be noted that, for ease of description, only parts related to the present application, rather than all structures, are shown in the accompanying drawings.
[0030] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0031] Figure 1 : is a flowchart of a character input method provided in an embodiment of the present invention. The embodiment of the present application can be applied to the case of remote input to an intelligent electronic device. The method can be executed by a character input device, which can be implemented in software and / or hardware, and the device can be configured in an electronic device with network communication function; for example, it can be a terminal device such as an electronic watch, smart glasses, smart TV, and mobile phone, and can also be a server device. Figure 1 As shown, the character input method provided in the embodiment of the present application may include the following steps:
[0032] S110: Determine an initial gesture vibration signal generated during the process of writing characters in the air through gestures.
[0033] In this embodiment, the user's hand gesture writing can be applied to various smart electronic devices, but the current hand gesture-based writing in the air is all achieved by tracking the hand gesture to achieve character recognition of the written characters, which causes various limitations. For example, the user is required to move significantly to write, otherwise the user's gesture path trace cannot be accurately tracked. Figure 2 is a schematic diagram of a gesture vibration signal during a mid-air writing process provided in an embodiment of the present invention. Figure 2 , the user can write characters in the air through gestures, and a certain vibration signal is usually generated in the process of writing characters, so the vibration signal generated by the vibration of writing characters can be collected, which is recorded here as the initial gesture vibration signal.
[0034] In an optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. Wherein, determining the initial gesture vibration signal generated in the process of writing characters in the air by gesture may include: using a preset vibration signal sensor to collect the initial gesture vibration signal generated in the process of writing characters in the air by gesture.
[0035] The vibration signal sensor includes an accelerometer and a gyroscope, and the vibration signal sensor is integrated with the electronic device or separately arranged.
[0036] In this embodiment, the vibration signal sensor can capture the vibration signal generated by the user writing characters in the air through hand gestures, but the vibration signal is usually not screened and captured to obtain an initial gesture vibration signal. At the same time, the vibration signal sensor can be directly integrated into terminal devices such as electronic watches and electronic bracelets. Since such devices are usually worn on the wrist and are relatively close to the fingers, the corresponding initial gesture vibration signal can be easily captured when writing characters with fingers; of course, the vibration signal sensor can also be set separately from electronic devices such as smart glasses, smart TVs and mobile phones. Because the above-mentioned electronic devices are usually not worn on the wrist all the time and are far away from the fingers, they are not directly integrated into the above-mentioned electronic devices, but are connected to the electronic devices wirelessly. The vibration signal sensor is separately configured near the finger, and after the gesture vibration signal is collected, it is sent wirelessly to the electronic device that is separately set.
[0037] S120: extracting a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal.
[0038] In this embodiment, in the process of writing characters in the air through gestures, not only the writing of characters will cause vibrations, but other operations will also cause vibrations. Therefore, the initial gesture vibration signal obtained includes not only the effective vibration signal caused by the writing of characters, but also the interference vibration signal caused by other factors. For this reason, after obtaining the initial gesture vibration signal, it is necessary to extract the vibration signal that can reflect the user's gesture writing of characters, and use the extracted vibration signal as the target gesture vibration signal to avoid directly using the initial gesture vibration signal to identify the gesture type, which may cause gesture type recognition errors due to the interference signal included in the initial gesture vibration signal.
[0039] S130. Determine the gesture type when writing characters in mid-air by performing vibration feature recognition on the target gesture vibration signal, so as to perform a character input operation based on the gesture type.
[0040] In this embodiment, different characters have different writing trajectories when writing characters, so the vibration characteristics of the gesture vibration signals generated by writing different characters are different. Based on this principle, the specific gesture type when writing characters can be identified by identifying the vibration characteristics of each target gesture vibration signal. Optionally, multiple gestures for writing characters in the air can be obtained in advance to form a gesture set, for example, gestures for writing ten Arabic numerals and twenty-six English letters are obtained to form a digital gesture set and a letter gesture set. For each gesture in the gesture set, the vibration characteristics corresponding to the gesture can be associated, so that when the vibration characteristics of the target gesture vibration signal are identified, the gesture type associated with the corresponding vibration characteristics can be known, and then the type of character being written can be known.
[0041] According to the character input method provided in the embodiment of the present application, the user does not need to move drastically to write characters with gestures, and the character information written by the user's gestures can be accurately recognized, which simplifies the complexity of remote input and improves its input convenience. At the same time, it is also unnecessary to customize special equipment to realize the recognition of fine finger writing in the air, thereby reducing the equipment development cost and implementation cost, realizing fine finger writing in the air, and enhancing the human-computer interaction experience.
[0042] Figure 3 1 is a flowchart of another character input method provided in an embodiment of the present invention. The technical solution of the embodiment of the present application is optimized on the basis of the above embodiment, and the embodiment of the present application can be combined with each optional solution in one or more of the above embodiments. Figure 3 As shown, the character input method provided in the embodiment of the present application may include the following steps:
[0043] S310: Determine an initial gesture vibration signal generated during the process of writing characters in the air through gesture.
[0044] S320, performing endpoint detection according to the signal strength of the initial gesture vibration signal, and determining a target start endpoint and a target end endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal.
[0045] In this embodiment, Figure 4 FIG. 1 is a diagram of a process for detecting a gesture signal provided in an embodiment of the present invention. Figure 4 The initial gesture vibration signal includes a valid gesture vibration signal and an invalid interference vibration signal, wherein the invalid vibration signal may include a vibration signal that appears when a character is being written and a vibration signal that appears before and after writing a character. Therefore, the starting endpoint and the ending endpoint of the gesture vibration signal that can effectively represent the information of the character written in the air in the initial gesture vibration signal can be identified by performing endpoint detection on the signal, so as to extract the vibration signal between the two endpoints from the initial gesture vibration signal as a valid signal as the subsequent recognition object, thereby avoiding the vibration signals before and after writing characters from being mixed in and affecting the subsequent vibration feature recognition process.
[0046] In an optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. Among them, endpoint detection is performed based on the signal strength of the initial gesture vibration signal to determine the target starting endpoint and target ending endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information, which may include the following steps A1-A3:
[0047] Step A1: Detect the signal strength of the initial gesture vibration signal at a preset time interval.
[0048] Step A2: if it is detected that the signal strength of the initial gesture vibration signal begins to be greater than a first strength threshold, the detection point where the signal strength begins to be greater than the first strength threshold is used as the target starting endpoint; and after a target starting endpoint is detected, the signal strength of the initial gesture vibration signal continues to be detected.
[0049] Step A3: If it is detected during the continued detection process that the signal strength of the initial gesture vibration signal begins to be less than the second strength threshold, the detection point when the signal strength begins to be less than the second strength threshold is used as the target termination endpoint.
[0050] In this embodiment, the first intensity threshold and the second intensity threshold are the same or the threshold difference is within a preset range. The endpoint detection of the initial gesture vibration signal can be performed based on the set signal strength threshold. For example, a first intensity threshold for detecting the starting endpoint of the gesture vibration signal and a second intensity threshold for detecting the ending endpoint of the gesture vibration signal are set. In this way, the signal strength value of the initial gesture vibration signal can be detected at a preset time interval. During the detection process, when the signal strength begins to be greater than the first intensity threshold, the detection point is considered to be the starting endpoint of a valid gesture vibration signal; after the starting endpoint of the gesture vibration signal is detected, the initial gesture vibration signal continues to be detected until the signal strength is detected to be lower than the second intensity threshold, and the detection point is considered to be the ending endpoint of a valid gesture vibration signal. The gesture vibration signal from the starting endpoint to the ending endpoint is the required vibration signal.
[0051] In this embodiment, based on the single threshold endpoint detection of the starting endpoint, two first intensity thresholds, one large and one small, can be set for the detection of the starting endpoint. When the initial gesture vibration signal is detected for the starting endpoint, the initial gesture vibration signal first passes the detection of the smaller first intensity threshold. After passing the detection of the smaller first intensity threshold, the initial gesture vibration signal needs to be detected using the larger first intensity threshold. When the signal strength of the initial gesture vibration signal is detected to be successively greater than the smaller first intensity threshold and the larger first intensity threshold, it is determined that the starting endpoint of the signal is detected. In the above scheme, the use of dual thresholds can remove the influence of outliers, and because two thresholds, one large and one small, are used, the signal can be detected to the starting endpoint more accurately.
[0052] In another optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. Among them, endpoint detection is performed based on the signal strength of the initial gesture vibration signal to determine the target starting endpoint and target ending endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information, which may include the following steps B1-B3:
[0053] Step B1: extracting a gesture vibration signal of a preset frame length from an initial gesture vibration signal according to a preset frame shift step length, and calculating the signal energy of the gesture vibration signal of the preset frame length.
[0054] Step B2: if it is detected that the signal energy of the extracted gesture vibration signal of the preset frame length begins to be greater than the first energy threshold, the extraction point when it begins to be greater than the first energy threshold is used as the target starting endpoint; and after a target starting endpoint is detected, continue to extract the gesture vibration signal of the preset frame length and calculate the signal energy value.
[0055] Step B3: if it is detected during the continued extraction process that the signal energy of the gesture vibration signal of the preset frame length extracted begins to be less than the second energy threshold, the extraction point when the signal energy begins to be less than the second energy threshold is used as the target termination endpoint.
[0056] In this embodiment, the first energy threshold and the second energy threshold are the same or the threshold difference is within a preset range. In the threshold-based endpoint detection scheme, in order to avoid the signal strength at a certain moment being higher than a given intensity threshold due to the error of the vibration sensor or the influence of the environment and being mistakenly identified as the starting endpoint of the gesture vibration signal. Therefore, in order to detect the effective gesture vibration signal when the user writes characters as accurately as possible, it is necessary to reduce the amount of calculation at the same time. Instead of detecting the signal strength at a single time point, the intra-frame energy is calculated for the data of a time period (for example, a frame of data), and endpoint detection is performed based on the signal energy for a period of time. By considering the signal of a time segment, the influence of outliers on endpoint detection can also be avoided.
[0057] In this embodiment, a first energy threshold for detecting the starting endpoint of the gesture vibration signal and a second energy threshold for detecting the ending endpoint of the gesture vibration signal are set. In this way, the gesture vibration signal of the preset frame length can be extracted from the initial gesture vibration signal in sequence according to the preset frame shift step length, and the signal energy of the gesture vibration signal of the preset frame length can be calculated. During the detection process, when the frame energy begins to be greater than the first energy threshold, the detection point is considered to be the starting endpoint of the valid gesture vibration signal; after the starting endpoint of the gesture vibration signal is detected, the initial gesture vibration signal continues to be detected until the frame energy is detected to be lower than the second energy threshold, and the detection point is considered to be the ending endpoint of the valid gesture vibration signal, and the gesture vibration signal from the starting endpoint to the ending endpoint is the required vibration signal.
[0058] In this embodiment, Figure 5 FIG. 1 is a partial schematic diagram of an endpoint detection based on frame energy provided in an embodiment of the present invention. Figure 5 By setting the frame length and frame shift reasonably, when performing endpoint detection on the initial gesture vibration signal based on frame energy, it is possible to effectively avoid false detection of noise signals. For example, the frame length is set to 0.2s and the frame shift is set to 0.01s. Using the data calculated by frame energy, when it is detected that the frame energy is greater than the threshold (according to the actual signal situation, the threshold is set to 0.03 here), the current position is recorded as b (signal start position) and the frame energy is continued to be detected. When the frame energy is less than the threshold, the current position is recorded as e (signal end position). Figure 5 The actual collected ten digital gesture signal segments from 0 to 9 are shown, and it can be seen that the system can accurately segment the gesture signals.
[0059] By adopting the above optional method, after calculating the frame energy of the signal, not only can the noise signal be better suppressed, but also the effective signal can be amplified significantly. Moreover, the frame energy calculation can be realized within a constant number of operations on a real-time system, thus avoiding the performance burden caused by excessive calculations.
[0060] S330: intercepting a valid target gesture vibration signal from the initial gesture vibration signal according to the target start endpoint and the target end endpoint.
[0061] In an optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. According to the target starting endpoint and the target ending endpoint, intercepting a valid target gesture vibration signal from the initial gesture vibration signal may include the following operations:
[0062] If the endpoint length between the two endpoints is determined to be within a preset endpoint length threshold range based on the target start endpoint and the target end endpoint, a valid target gesture vibration signal is intercepted from the initial gesture vibration signal according to the target start endpoint and the target end endpoint.
[0063] In this embodiment, in the endpoint detection stage, slight vibrations are also considered as gesture vibration signals generated by writing characters, which can effectively avoid the omission of gesture vibration signal detection, but at the same time it will increase the false detection of noise signals. Therefore, although the signal passes the endpoint detection, it is necessary to further judge the validity of the obtained signal to reduce the probability of false detection of noise. By judging whether the inter-endpoint length between the target starting endpoint and the target ending endpoint is within the preset endpoint length threshold range, it is determined whether the target starting endpoint and the target ending endpoint are valid. For example, according to the length of the signal l=eb (e is the target ending endpoint, b is the target starting endpoint), it is judged whether the intercepted signal is a legal gesture vibration signal. Because the noise signal is usually short, such as when the finger moves occasionally, or in most cases it will be longer, and the length of the gesture signal is usually within a time range. A length requirement is added to the valid gesture signal, and the signal length l needs to meet 0.6s<l<2.0s to be considered as a valid gesture vibration signal, which can reduce the probability of the noise signal being mistakenly identified as a gesture vibration signal.
[0064] In an optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. According to the target starting endpoint and the target ending endpoint, intercepting a valid target gesture vibration signal from the initial gesture vibration signal may include the following operations:
[0065] The gesture vibration signal from the preset frame length before the target start endpoint to the preset length after the target end endpoint in the initial gesture vibration signal is determined as the target gesture vibration signal.
[0066] In this embodiment, after judging the validity of the gesture vibration signal, the target gesture vibration signal can be extracted according to the target starting endpoint and the target ending endpoint obtained by endpoint detection. However, since the preprocessing of the signal in the subsequent feature extraction step requires retaining a buffer data before and after the signal, in addition to the signal detected at the interception endpoint in the signal interception step, it is necessary to intercept an additional section before and after the signal. Here, the buffer length is set to 0.2s. When intercepting gesture signals, the length of the gesture signal is fixed to 2.0s. The fixed-length interception method is to move 0.2s forward from the starting point position as the feature extraction signal preprocessing buffer data according to the starting point position of the signal position, and then intercept the signal with a frame length of 2.0s from this position backward.
[0067] In an optional solution of this embodiment, it can be combined with each optional solution in one or more of the above embodiments. Before determining the target starting endpoint and the target ending endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal, the following operations can also be included:
[0068] The six-axis initial gesture vibration signals of the three-axis accelerometer and the three-axis gyroscope in the vibration sensor are fused to obtain a fused initial gesture vibration signal.
[0069] In this embodiment, see Figure 5 In order to detect subtle changes in gesture vibration signals more sensitively, the six-axis data of the three-axis accelerometer and the three-axis gyroscope need to be fused before frame energy calculation. By adding the energy of the six-axis data, the omission of gesture vibration signal detection can be effectively avoided.
[0070] S340: Determine the type of gesture when writing characters in mid-air by performing vibration feature recognition on the target gesture vibration signal, so as to perform a character input operation based on the gesture type.
[0071] According to the character input method provided in the embodiment of the present application, an effective target gesture vibration signal is intercepted from the initial gesture vibration signal generated by the user in the process of writing characters through gesture, so as to avoid the invalid vibration signal image and the subsequent vibration feature recognition result, thereby causing erroneous character input. Moreover, the user does not need to move drastically to write characters with gestures, and the character information written by the user's gestures can be accurately recognized, thereby simplifying the complexity of remote input and improving its input convenience. At the same time, there is no need to customize special equipment to realize the recognition of fine finger writing in the air, thereby reducing the equipment development cost and implementation cost, realizing fine finger writing in the air, and enhancing the human-computer interaction experience.
[0072] On the basis of the above embodiment, it can be combined with various optional solutions in one or more of the above embodiments. Before determining the target starting endpoint and the target ending endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal, the following operations can also be included:
[0073] The initial gesture vibration signal is filtered using a preset high-pass filter to obtain a filtered initial gesture vibration signal.
[0074] In this embodiment, before performing endpoint detection on the initial gesture vibration signal, the initial gesture vibration signal needs to be high-pass filtered. Among them, high-pass filtering can remove the gravity component contained in the collected accelerometer signal, reduce the influence of gravity on signal detection; and reduce the influence of noise (such as slight shaking of the user's arm during signal collection). Optionally, by analyzing the frequency domain distribution characteristics of the gesture vibration signal, the initial gesture vibration signal can be processed by high-pass 5HZ, and the filter selects Butter-Worth as the prototype.
[0075] Figure 6 1 is a flowchart of another character input method provided in an embodiment of the present invention. The technical solution of the embodiment of the present application is optimized on the basis of the above embodiment, and the embodiment of the present application can be combined with each optional solution in one or more of the above embodiments. Figure 6 As shown, the character input method provided in the embodiment of the present application may include the following steps:
[0076] S610: Determine an initial gesture vibration signal generated during the process of writing characters in the air through gesture.
[0077] S620: Extracting a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal.
[0078] S630: extracting time domain features and frequency domain features of the target gesture vibration signal, and obtaining vibration signal fusion features that fuse the time domain features and the frequency domain features.
[0079] In this embodiment, Figure 7 FIG. 1 is a schematic diagram of a video feature after digital gravity removal provided in an embodiment of the present invention. Figure 7 In the final time domain feature extraction, the target gesture vibration signal can be first intercepted from the original initial gesture vibration signal, and then the time domain features and frequency domain features can be extracted from it; or the original initial gesture vibration signal can be first filtered to obtain a filtered initial gesture vibration signal, and then the time domain features and frequency domain features can be extracted from it.
[0080] In this embodiment, see Figure 7, shows that through the time-frequency analysis of the gesture signals of the ten digits 0-9 collected continuously, it can be seen that the gesture signals are mainly distributed below 25Hz, and there are also distributions below 5Hz, so it is necessary to retain the low-frequency information. Without filtering, the change of gravity is also retained. This part of information is also helpful in distinguishing gestures: the change of gravity of the watch under different gestures is also inconsistent. Considering that different gestures also have differences in the frequency domain, frequency domain information is added to the features. The frequency domain information of the target gesture vibration signal can be expressed by the following Fourier transform formula (1).
[0081]
[0082] In this embodiment, the frequency domain information can be extracted from the time domain information by using an FFT (Fast Fourier Transformation) algorithm, and the vibration signal fusion feature finally selected includes both the time domain feature information and the frequency domain feature information.
[0083] S640: Input the vibration signal fusion feature of the target gesture vibration signal into a pre-trained gesture classification model, output the gesture type when writing characters in the air, and perform a character input operation based on the gesture type.
[0084] In this embodiment, in the signal detection stage, there will inevitably be a certain offset when performing endpoint detection on different gesture vibration signals, and the classifier uses time domain information as a feature. Even a slight offset between different signals will inevitably increase the classification difficulty of the classifier and reduce the classification accuracy of the classifier. In the calculation of distance, it is not difficult to know that even if there are two identical signals, after offsetting one of the signals and then calculating the Euclidean distance or Manhattan distance of the two signals, a relatively large value will be calculated, even if the two signals are the same signal under different time delays. Therefore, the offset between signals will undoubtedly have a certain impact on the classification algorithm based on the distance information between signals. In order to solve this problem, GCC (Generalized Cross-Correlation) can be used to calculate the time delay between the two signals to achieve alignment of the gesture signals.
[0085] Before inputting the vibration signal fusion features of the target gesture vibration signal into the pre-trained gesture classification model, the method may further include: performing feature normalization processing on the vibration signal fusion features of the target gesture vibration signal.
[0086] In this embodiment, feature normalization is to normalize the dimensions of features of different dimensions. By normalizing the features, the different optimization degrees of different features in the classification algorithm using distance as a measure of similarity due to inconsistent dimensions between different features can be eliminated, which is beneficial to improving the classification accuracy of the model. At the same time, for neural networks, normalized features can make the local optimal coefficients of different features be at the same order of magnitude, which speeds up the convergence speed of target optimization using the gradient descent algorithm.
[0087] In this embodiment, there are many data normalization algorithms to choose from, and the commonly used ones are 1) min-max normalization and z-score zero mean normalization. Min-max normalization will normalize the values of a set of data to the range of [0,1]. The specific calculation is shown in formula (2), where y(t) is the data after normalization at time t, and x(t) is the data at time t. The z-score processing method is to subtract the mean from a set of data and divide it by the standard deviation to convert this set of data into data with a mean of 0 and a variance of 1. The specific calculation is shown in formula (3), where y(t) is the data after normalization at time t, x(t) is the data at time t, μ is the mean of data x, and σ is the standard deviation of data x.
[0088]
[0089]
[0090] In this embodiment, min-max normalization and z-score standardization are essentially scaling and translating the data, and both formulas can be expressed using Formula 4. Min-max only considers the minimum and maximum values of the data during scaling and translating, and scales the data to a fixed range [0,1]. Compared to min-max, z-score needs to calculate the mean and variance of the data during scaling and translating, uses all the data, and is not fixed in the range after data scaling.
[0091] In this embodiment, optionally, the data collected by the gesture vibration signal is acceleration sensor data and gyroscope sensor data, and the fluctuation of the data value is relatively unstable, which is not suitable for scaling the data to a fixed range. Therefore, it is more appropriate to use the z-score method to standardize the data.
[0092] According to the character input method provided in the embodiment of the present application, the user does not need to move drastically to write characters with gestures, and the character information written by the user's gestures can be accurately recognized, which simplifies the complexity of remote input and improves its input convenience. At the same time, it is also unnecessary to customize special equipment to realize the recognition of fine finger writing in the air, thereby reducing the equipment development cost and implementation cost, realizing fine finger writing in the air, and enhancing the human-computer interaction experience.
[0093] Figure 8 : is a structural block diagram of a character input device provided in an embodiment of the present invention. The embodiment of the present application can be applied to the case of remote input of intelligent electronic devices. The device can be implemented in software and / or hardware, and the device can be configured in an electronic device with network communication function; for example, it can be a terminal device such as an electronic watch, smart glasses, smart TV, and mobile phone. Figure 8 As shown, the character input device provided in the embodiment of the present application may include the following: an initial signal determination module 810, a target signal extraction module 820 and a gesture character input module 830. Among them:
[0094] An initial signal determination module 810 is used to determine an initial gesture vibration signal generated during the process of performing air writing of characters through gestures;
[0095] A target signal extraction module 820 is used to extract a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal;
[0096] The gesture character input module 830 is used to determine the gesture type when writing characters in the air by performing vibration feature recognition on the target gesture vibration signal, so as to perform a character input operation based on the gesture type.
[0097] Based on the above embodiment, optionally, the initial signal determination module 810 includes:
[0098] Using a preset vibration signal sensor to collect initial gesture vibration signals generated during the process of writing characters in the air through gestures;
[0099] The vibration signal sensor includes an accelerometer and a gyroscope, and the vibration signal sensor is integrated with or separately from the electronic device.
[0100] Based on the above embodiment, optionally, the target signal extraction module 820 includes:
[0101] Performing endpoint detection according to the signal strength of the initial gesture vibration signal to determine a target start endpoint and a target end endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal;
[0102] According to the target start endpoint and the target end endpoint, a valid target gesture vibration signal is intercepted from the initial gesture vibration signal.
[0103] On the basis of the above embodiment, optionally, endpoint detection is performed according to the signal strength of the initial gesture vibration signal to determine the target start endpoint and target end endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information, including:
[0104] Detecting the signal strength of the initial gesture vibration signal at a preset time interval;
[0105] If it is detected that the signal strength of the initial gesture vibration signal begins to be greater than a first strength threshold, the detection point at which the signal strength begins to be greater than the first strength threshold is used as a target starting endpoint; and after a target starting endpoint is detected, the signal strength of the initial gesture vibration signal continues to be detected;
[0106] If it is detected during the continued detection process that the signal strength of the initial gesture vibration signal begins to be less than the second strength threshold, the detection point when the signal strength begins to be less than the second strength threshold is used as the target termination endpoint;
[0107] The first intensity threshold and the second intensity threshold have the same value or the threshold difference is within a preset range.
[0108] On the basis of the above embodiment, optionally, endpoint detection is performed according to the signal strength of the initial gesture vibration signal to determine the target start endpoint and target end endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information, including:
[0109] Extracting a gesture vibration signal of a preset frame length from the initial gesture vibration signal according to a preset frame shift step length, and calculating the signal energy of the gesture vibration signal of the preset frame length;
[0110] If it is detected that the signal energy of the extracted gesture vibration signal of the preset frame length begins to be greater than the first energy threshold, the extraction point when the signal energy begins to be greater than the first energy threshold is used as the target starting endpoint; and after a target starting endpoint is detected, continue to extract the gesture vibration signal of the preset frame length and calculate the signal energy value;
[0111] If it is detected during the continued extraction process that the signal energy of the gesture vibration signal of the preset frame length extracted begins to be less than the second energy threshold, the extraction point when it begins to be less than the second energy threshold is used as the target termination endpoint;
[0112] The first energy threshold and the second energy threshold have the same value or the threshold difference is within a preset range.
[0113] Based on the above embodiment, optionally, according to the target start endpoint and the target end endpoint, intercepting a valid target gesture vibration signal from the initial gesture vibration signal includes:
[0114] If it is determined that the endpoint length between the two endpoints is within a preset endpoint length threshold range based on the target starting endpoint and the target ending endpoint, a valid target gesture vibration signal is intercepted from the initial gesture vibration signal according to the target starting endpoint and the target ending endpoint.
[0115] Based on the above embodiment, optionally, according to the target start endpoint and the target end endpoint, intercepting a valid target gesture vibration signal from the initial gesture vibration signal includes:
[0116] The gesture vibration signal from the preset frame length before the target start endpoint to the preset length after the target end endpoint in the initial gesture vibration signal is determined as the target gesture vibration signal.
[0117] Based on the above embodiment, optionally, before determining the target starting endpoint and the target ending endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal, the method further includes:
[0118] The initial gesture vibration signal is filtered using a preset high-pass filter to obtain a filtered initial gesture vibration signal.
[0119] Based on the above embodiment, optionally, before determining the target starting endpoint and the target ending endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal, the method further includes:
[0120] The six-axis initial gesture vibration signals of the three-axis accelerometer and the three-axis gyroscope in the vibration sensor are fused to obtain a fused initial gesture vibration signal.
[0121] Based on the above embodiment, optionally, the gesture character input module 830 includes:
[0122] Extracting the time domain features and the frequency domain features of the target gesture vibration signal to obtain a vibration signal fusion feature that fuses the time domain features and the frequency domain features;
[0123] The vibration signal fusion feature of the target gesture vibration signal is input into a pre-trained gesture classification model to output the gesture type when writing characters in the air.
[0124] Based on the above embodiment, optionally, before the vibration signal fusion feature of the target gesture vibration signal is input into a pre-trained gesture classification model, the method further includes:
[0125] A feature normalization process is performed on the vibration signal fusion feature of the target gesture vibration signal.
[0126] The character input device provided in the embodiments of the present application can execute the character input method provided in any of the embodiments of the present application, and has the corresponding functions and beneficial effects of executing the character input method. For technical details not described in detail in the above embodiments, please refer to the character input method provided in any of the embodiments of the present application.
[0127] Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 The electronic device provided in the embodiment of the present application includes: one or more processors 910 and a storage device 920; the processor 910 in the electronic device may be one or more, Fig. 9 A processor 910 is taken as an example; the storage device 920 is used to store one or more programs; the one or more programs are executed by the one or more processors 910, so that the one or more processors 910 implement the character input method as described in any one of the embodiments of the present application.
[0128] The electronic device may further include: an input device 930 and an output device 940 .
[0129] The processor 910, storage device 920, input device 930 and output device 940 in the electronic device can be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.
[0130] The storage device 920 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the character input method provided in the embodiment of the present application. The processor 910 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the storage device 920, that is, implementing the application control method in the above method embodiment.
[0131] The storage device 920 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 920 may further include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0132] The input device 930 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 940 may include a display device such as a display screen.
[0133] Furthermore, when one or more programs included in the electronic device are executed by the one or more processors 910, the program performs the following operations:
[0134] Determine an initial gesture vibration signal generated during the process of performing air writing of characters through a gesture;
[0135] Extracting a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal;
[0136] The vibration feature recognition is performed on the target gesture vibration signal to determine the gesture type when writing characters in mid-air, so as to perform a character input operation based on the gesture type.
[0137] Of course, those skilled in the art can understand that when one or more programs included in the above-mentioned electronic device are executed by the one or more processors 910, the program can also perform related operations in the character input method provided in any embodiment of the present application.
[0138] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it is used to perform a character input method, the method comprising:
[0139] Determine an initial gesture vibration signal generated during the process of performing air writing of characters through a gesture;
[0140] Extracting a target gesture vibration signal for effectively representing mid-air writing character information from the initial gesture vibration signal;
[0141] The vibration feature recognition is performed on the target gesture vibration signal to determine the gesture type when writing characters in mid-air, so as to perform a character input operation based on the gesture type.
[0142] Optionally, when the program is executed by a processor, it can also be used to execute the character input method provided in any embodiment of the present application.
[0143] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, - but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive lists) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (Random Access Memory, RAM), read-only memories (Read Only Memory, ROM), erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), flash memory, optical fibers, portable CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination of the above. Computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0144] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0145] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.
[0146] Computer program code for performing the operation of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).
[0147] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0148] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims
1. A character input method, characterized in that: Applied to remote input of electronic equipment, the method comprises: An initial gesture vibration signal generated during the process of writing characters in the air with a finger is determined by a preset vibration signal sensor; wherein the vibration signal sensor is arranged near the finger; the initial gesture vibration signal is obtained by fusing the six-axis initial gesture vibration signals of the three-axis accelerometer and the three-axis gyroscope in the vibration signal sensor; the initial gesture vibration signal includes a valid gesture vibration signal and an invalid interference vibration signal; the invalid interference vibration signal includes a vibration signal that occurs when a character is being written and a vibration signal that occurs before and after writing a character; Using a preset high-pass filter to filter the initial gesture vibration signal to obtain a filtered initial gesture vibration signal; Performing endpoint detection according to the signal strength of the initial gesture vibration signal after the filtering process, and determining a target starting endpoint and a target ending endpoint of the gesture vibration signal that can effectively represent the mid-air writing character information in the initial gesture vibration signal; If it is determined that the endpoint length between the two endpoints is within a preset endpoint length threshold range according to the target start endpoint and the target end endpoint, the gesture vibration signal from the preset frame length before the target start endpoint to the preset length after the target end endpoint in the initial gesture vibration signal is determined as the target gesture vibration signal; Pre-acquire a plurality of gestures of writing characters in mid-air to form a gesture set; wherein each gesture in the gesture set is associated with a corresponding vibration feature; Based on the gesture set, extracting the time domain features and the frequency domain features of the target gesture vibration signal, and obtaining a vibration signal fusion feature that fuses the time domain features and the frequency domain features; Performing feature normalization processing on the vibration signal fusion feature; Inputting the processed vibration signal fusion features into a pre-trained gesture classification model, outputting the gesture type when writing characters in the air, and performing a character input operation based on the gesture type; The step of performing endpoint detection based on the signal strength of the initial gesture vibration signal after filtering to determine a target start endpoint and a target end endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information includes: Extracting a gesture vibration signal of a preset frame length from the initial gesture vibration signal according to a preset frame shift step length, and calculating the signal energy of the gesture vibration signal of the preset frame length; If it is detected that the signal energy begins to be greater than a first energy threshold, the extraction point when the energy begins to be greater than the first energy threshold is used as the target starting endpoint; and after detecting a target starting endpoint, continue to extract the gesture vibration signal of the preset frame length and calculate the signal energy; If it is detected during the continued extraction process that the signal energy begins to be less than the second energy threshold, the extraction point when it begins to be less than the second energy threshold will be used as the target termination endpoint; wherein, the first energy threshold is the same as the second energy threshold or the threshold difference is within a preset range.
2. A character input device, characterized in that: The device is configured to perform remote input on an electronic device, and the device comprises: An initial signal determination module is used to determine an initial gesture vibration signal generated during the process of writing characters in the air by a finger through a preset vibration signal sensor; wherein the vibration signal sensor is arranged near the finger; the initial gesture vibration signal is obtained by fusing the six-axis initial gesture vibration signals of the three-axis accelerometer and the three-axis gyroscope in the vibration signal sensor; the initial gesture vibration signal includes a valid gesture vibration signal and an invalid interference vibration signal; the invalid interference vibration signal includes a vibration signal that occurs when a character is being written and a vibration signal that occurs before and after writing a character; a preset high-pass filter is used to filter the initial gesture vibration signal to obtain the filtered initial gesture vibration signal; a target signal extraction module, configured to perform endpoint detection according to the signal strength of the initial gesture vibration signal after filtering, and determine a target start endpoint and a target end endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively characterize the mid-air writing character information; if, according to the target start endpoint and the target end endpoint, it is determined that the end-to-end length between the two end points is within a preset endpoint length threshold range, then the gesture vibration signal in the initial gesture vibration signal from a preset frame length before the target start endpoint to a preset length after the target end endpoint is determined as the target gesture vibration signal; A gesture character input module is used to pre-acquire a plurality of gestures for writing characters in the air to form a gesture set; wherein each gesture in the gesture set is associated with a corresponding vibration feature; based on the gesture set, extract the time domain features and frequency domain features of the target gesture vibration signal to obtain a vibration signal fusion feature that fuses the time domain features and the frequency domain features; perform feature normalization processing on the vibration signal fusion feature; input the processed vibration signal fusion feature into a pre-trained gesture classification model, output the gesture type when writing characters in the air, and perform a character input operation based on the gesture type; The step of performing endpoint detection based on the signal strength of the initial gesture vibration signal after filtering to determine a target start endpoint and a target end endpoint of the gesture vibration signal in the initial gesture vibration signal that can effectively represent the mid-air writing character information includes: Extracting a gesture vibration signal of a preset frame length from the initial gesture vibration signal according to a preset frame shift step length, and calculating the signal energy of the gesture vibration signal of the preset frame length; If it is detected that the signal energy begins to be greater than a first energy threshold, the extraction point when the energy begins to be greater than the first energy threshold is used as the target starting endpoint; and after detecting a target starting endpoint, continue to extract the gesture vibration signal of the preset frame length and calculate the signal energy; If it is detected during the continued extraction process that the signal energy begins to be less than the second energy threshold, the extraction point when it begins to be less than the second energy threshold will be used as the target termination endpoint; wherein, the first energy threshold is the same as the second energy threshold or the threshold difference is within a preset range.
3. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the character input method described in claim 1.
4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the character input method according to claim 1 is implemented.
Citation Information
Patent Citations
Method and device for identifying type of finger point
CN102411427A
Virtual gesture input method and gesture collecting device
CN103760983A
Cloud processing based gesture identification method
CN105224066A
Aerial handwriting character recognition method based on inertia sensor
CN107316067A
Skin type finger gesture recognition method based on smart watch
CN110069199A
