A gesture recognition method, apparatus and terminal device

Abstract

The embodiment of the present application discloses a gesture recognition method, device and terminal equipment, the gesture recognition method comprises: acquiring the elastic wave signal detected by at least one elastic wave sensor arranged on a touch device; judging the signal characteristics met by the elastic wave signal, the signal characteristics include: synchronous characteristics, asynchronous characteristics or preset click gesture waveform characteristics; and performing gesture recognition according to the signal characteristics met by the elastic wave signal. The gesture recognition method, device and terminal equipment disclosed by the embodiment of the present application can realize user gesture recognition and improve the user's human-computer interaction experience.

Description

Technical Field

[0001] This invention relates to, but is not limited to, electronic technology, and particularly to a gesture recognition method, apparatus, and terminal device. Background Technology

[0002] With the continuous development and widespread adoption of terminal devices, human-computer interaction technology has had a significant impact on their applications. Currently, gestures are widely considered a more intuitive and convenient interaction technology for communicating with terminal devices. Therefore, the recognition of user gestures by terminal devices is extremely important, as the accuracy of gesture recognition directly affects the user experience. Improving the accuracy of gesture recognition has become a pressing issue. Summary of the Invention

[0003] This application provides a gesture recognition method, including:

[0004] Acquire elastic wave signals detected by at least one elastic wave sensor disposed on the touch device;

[0005] The signal characteristics that the elastic wave signal conforms to are determined, including synchronous characteristics, asynchronous characteristics, or preset click gesture waveform characteristics;

[0006] Gesture recognition is performed based on the signal characteristics of the elastic wave signal.

[0007] This application also provides a gesture recognition device, including a signal processing device and at least one elastic wave sensor, wherein the signal processing device is electrically connected to each elastic wave sensor and is used to perform the gesture recognition method described in any embodiment.

[0008] This application also provides a terminal device, including a touch device and a gesture recognition device as described in any embodiment, wherein the touch device includes a display screen.

[0009] The gesture recognition method, apparatus, and terminal device provided in at least one embodiment of this application have the following advantages compared with the prior art: the user's hand touches the touch device as an excitation source to drive the touch device to vibrate or deform, so as to generate an elastic wave signal. The elastic wave signal is detected by at least one elastic wave sensor. The gesture recognition on the touch device can be performed based on the elastic wave signal, thereby realizing the user's gesture recognition and improving the user's human-computer interaction experience.

[0010] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0011] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0012] Figure 1 This is a flowchart of a gesture recognition method provided in an example embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram illustrating the synchronization characteristics of multiple effective elastic wave signals provided in an exemplary embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram illustrating the asynchronous characteristics of multiple effective elastic wave signals provided in an example embodiment of the present invention.

[0015] Figure 4 This is a schematic diagram of signal endpoint detection provided in an example embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of signal endpoint correction provided in an example embodiment of the present invention;

[0017] Figure 6 A flowchart of a gesture recognition method provided in another exemplary embodiment of the present invention;

[0018] Figure 7 This is a structural block diagram of a gesture recognition device provided in an example embodiment of the present invention;

[0019] Figure 8 This is a structural block diagram of a terminal device provided in an example embodiment of the present invention. Detailed Implementation

[0020] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0021] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0022] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0023] Figure 1 This is a flowchart of a gesture recognition method provided in an example embodiment of the present invention, as follows: Figure 1 As shown, the gesture recognition method may include: S101, S102 and S103.

[0024] S101: Acquire the elastic wave signal detected by at least one elastic wave sensor disposed on the touch device.

[0025] In this embodiment, the user's hand touching the touch device can be used as an excitation source to drive the touch device to vibrate or deform, thereby generating an elastic wave signal. Gesture recognition on the touch device can be performed based on the generated elastic wave signal, improving the accuracy of gesture recognition. The touch device may include a display screen.

[0026] Elastic wave sensors can be installed on the touch device to acquire the elastic wave signals generated when a hand touches the touch device. When there are multiple elastic wave sensors, they are located in different positions on the touch device.

[0027] In an alternative embodiment, the elastic wave sensor can be disposed on the device body or device housing where the touch device is located. For example, the elastic wave sensor can be disposed on the terminal device body or housing.

[0028] Elastic wave sensors can be sensors with both positive and inverse piezoelectric effects, such as piezoelectric sensors. Types of piezoelectric sensors can include piezoelectric ceramic sensors, piezoelectric thin film sensors, and piezoelectric crystal sensors.

[0029] S102: Determine the signal characteristics that the elastic wave signal conforms to. The signal characteristics may include: synchronous characteristics, asynchronous characteristics, or preset click gesture waveform characteristics.

[0030] In this embodiment, gesture recognition can be performed by determining whether the elastic wave signal conforms to synchronous characteristics, asynchronous characteristics, or preset click gesture waveform characteristics.

[0031] Synchronization characteristics refer to the phase difference of multiple elastic wave signals being less than or equal to a preset time difference value, and the phase difference of multiple elastic wave signals appearing at peaks or troughs being less than or equal to a preset time difference value.

[0032] The asynchronous characteristic refers to the fact that the phase difference between the peaks or troughs of multiple elastic wave signals is greater than a preset time difference value, and they will not intersect at the same time point.

[0033] The preset click gesture waveform features refer to the preset click gesture waveforms corresponding to different predefined click gesture categories.

[0034] S103: Gesture recognition is performed based on the signal characteristics of the elastic wave signal.

[0035] In this embodiment, the gesture can be identified as a click gesture or a swipe gesture based on whether multiple elastic wave signals have synchronous or asynchronous characteristics. Alternatively, the gesture can be identified as a click gesture based on whether the waveform of an elastic wave signal is a preset click gesture waveform.

[0036] In one example, the number of elastic wave sensors can be multiple, and gesture recognition based on the signal characteristics of the elastic wave signals can include:

[0037] When there is synchronous characteristic among the multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is recognized as a click gesture; when there is asynchronous characteristic among the multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is recognized as a swipe gesture.

[0038] In this embodiment, gesture recognition can be achieved based on multiple elastic wave sensors on the touch device. Depending on whether the multiple elastic wave signals corresponding to the multiple elastic wave sensors have synchronous or asynchronous characteristics, gestures of both position-invariant and position-variant types can be realized. Position-invariant gestures can include single clicks, multiple clicks, or prolonged presses, while position-variant gestures can include directional swipes, such as swipes up, down, left, or right.

[0039] In one example, the number of elastic wave sensors can be one, and gesture recognition based on the signal characteristics conforming to the elastic wave signal can include:

[0040] When the waveform of the elastic wave signal is the preset click gesture waveform, the gesture is determined to be a click gesture.

[0041] In this embodiment, gesture recognition can be achieved based on an elastic wave sensor on the touch device. Depending on whether the waveform of the elastic wave sensor matches the preset click gesture waveform, gestures of the position-invariant type can be realized.

[0042] Different click gesture categories can be predefined with preset click gesture waveforms. The elastic wave signal detected by the elastic wave sensor is compared with the preset click gesture waveform. When the waveform of the elastic wave signal matches the preset click gesture waveform, the gesture is determined to be a click gesture. When the waveform of the elastic wave signal does not match the preset click gesture waveform, the gesture is determined to be a non-click gesture.

[0043] The gesture recognition method provided in this invention uses the user's hand touching the touch device as an excitation source to drive the touch device to vibrate or deform, thereby generating an elastic wave signal. The elastic wave signal is detected by at least one elastic wave sensor, and gesture recognition on the touch device can be performed based on the elastic wave signal, thereby realizing the user's gesture recognition and improving the user's human-computer interaction experience.

[0044] Figure 2 This is a schematic diagram illustrating the synchronization characteristics of multiple elastic wave signals provided in an example embodiment of the present invention, as shown below. Figure 2 As shown, the synchronization characteristic refers to the phase difference between multiple elastic wave signals corresponding to multiple elastic wave sensors being less than or equal to a preset phase difference value. The synchronization characteristic can be that the amplitudes of multiple elastic wave signals corresponding to multiple elastic wave sensors begin to change synchronously in time (including positive and negative fluctuations), simultaneously exhibiting peaks or troughs, and intersecting at the same point. Alternatively, the synchronization characteristic can be that multiple elastic wave signals corresponding to multiple elastic wave sensors change synchronously (including positive and negative fluctuations) within a certain time difference threshold (preset time difference value), and simultaneously exhibit peaks or troughs within the same time difference threshold.

[0045] In one example, for wave-type elastic wave sensors, the synchronization characteristic means that the multiple elastic wave signals corresponding to multiple elastic wave sensors intersect at the same point in the middle. Alternatively, the synchronization characteristic can be that the multiple elastic wave signals corresponding to multiple elastic wave sensors intersect within a certain time difference threshold (a preset time difference value). For non-wave-type elastic wave sensors (such as piezoresistive sensors), the multiple elastic wave signals corresponding to multiple elastic wave sensors may not necessarily intersect at the middle point, but their amplitudes change synchronously between the beginning and the end.

[0046] Figure 3 This is a schematic diagram illustrating the asynchronous characteristics of multiple elastic wave signals provided in an example embodiment of the present invention, as shown below. Figure 3 As shown, the asynchronous characteristic refers to the phase difference between multiple elastic wave signals corresponding to multiple elastic wave sensors being greater than a preset phase difference value. The asynchronous characteristic also means that the peak changes of the multiple elastic wave signals corresponding to multiple elastic wave sensors have a sequential order, and they do not intersect at the same point. The peak changes can be sequential changes of positive peaks (peaks) and / or sequential changes of negative peaks (troughs). For example... Figure 3 In the example, if four elastic wave sensors are set, the elastic wave signals corresponding to the four elastic wave sensors are CH1, CH2, CH3 and CH4 respectively. The order of peak change is as follows: signal CH1 has a peak or trough -> signal CH2 has a peak or trough -> signal CH3 has a peak or trough -> signal CH4 has a peak or trough. -> indicates earlier than.

[0047] User gestures are identified based on extracted synchronous or asynchronous gesture features. Gesture categories can be predefined, such as clicks and swipes. When multiple elastic wave signals from multiple elastic wave sensors exhibit synchronous characteristics, the gesture is determined to be a click. When multiple elastic wave signals from multiple elastic wave sensors exhibit asynchronous characteristics, the gesture is determined to be a swipe.

[0048] In one exemplary embodiment of the present invention, after recognizing the gesture as a click gesture, the method may further include:

[0049] Obtain multiple gesture recognition results within a preset time period; if only one click gesture appears in the gesture recognition results of multiple consecutive detection cycles within the preset time period, the click gesture is determined to be a single click; if all click gestures appear in the gesture recognition results of N consecutive detection cycles within the preset time period, the click gesture is determined to be N clicks, where N is greater than or equal to 2.

[0050] In this embodiment, the predefined gesture categories may include: single click and multiple clicks. When multiple elastic wave signals have synchronous characteristics and it is a single action within a detection period, it is identified as a single click gesture. When multiple elastic wave signals have synchronous characteristics and correspond to multiple actions within multiple detection periods, and one action corresponds to one detection period, it is identified as a multiple click gesture.

[0051] In one example embodiment of the present invention, after recognizing the gesture as a click gesture, the method may further include: determining whether the click gesture is a press gesture with a preset press duration based on the duration of the elastic wave signal.

[0052] In this embodiment, the predefined gesture categories may include: long press, which refers to a press with a duration greater than or equal to a press time threshold. A long press can be referred to as a press with a preset press duration. When multiple elastic wave signals exhibit synchronous characteristics and the press time exceeds a certain time threshold, it is identified as a long press gesture.

[0053] In one example, determining whether a click gesture is a press gesture with a preset press duration based on the duration of the elastic wave signal may include:

[0054] When the duration of any elastic wave signal is greater than or equal to the first press time threshold, the click gesture is determined to be a press gesture with a preset press duration.

[0055] The gesture can be determined as a press gesture with a preset press duration based on whether multiple elastic wave signals exhibit synchronization characteristics and the duration of any one elastic wave signal. If multiple elastic wave signals exhibit synchronization characteristics and the duration of any one elastic wave signal is greater than or equal to a first press time threshold, the click gesture is determined to be a press gesture with a preset press duration. If multiple elastic wave signals exhibit synchronization characteristics and the durations of all multiple elastic wave signals are less than the first press time threshold, the gesture is determined to be a click gesture. The first press time threshold can be determined based on empirical values, and this embodiment does not limit or elaborate on it here.

[0056] In one example, determining whether a click gesture is a press gesture with a preset press duration based on the duration of the elastic wave signal may include:

[0057] The duration of each elastic wave signal among multiple elastic wave signals is determined. When the maximum duration of the multiple elastic wave signals is greater than or equal to the second pressing time threshold, the click gesture is determined to be a pressing gesture with a preset pressing duration.

[0058] The gesture can be determined as a press gesture with a preset press duration based on whether multiple elastic wave signals exhibit synchronization characteristics and the duration of each elastic wave signal. If multiple elastic wave signals exhibit synchronization characteristics, and the longest duration among the multiple elastic wave signals is greater than or equal to a second press duration threshold, the click gesture is determined to be a press gesture with a preset press duration. If multiple elastic wave signals exhibit synchronization characteristics, and the durations of all multiple elastic wave signals are less than the second press duration threshold, the gesture is determined to be a click gesture. The second press duration threshold can be determined based on empirical values, and this embodiment does not limit or elaborate on it here.

[0059] In one exemplary embodiment of the present invention, after recognizing the gesture as a swipe gesture, the method may further include:

[0060] The direction of the swipe gesture is determined by the location of multiple elastic wave sensors corresponding to multiple elastic wave signals that appear successively at peaks or troughs.

[0061] When the location of the elastic wave sensor corresponding to the earliest elastic wave signal with a peak or trough is the first position, and the location of the elastic wave sensor corresponding to the latest elastic wave signal with a peak or trough is the second position, the direction of the sliding gesture is determined to be from the first position to the second position.

[0062] In this embodiment, the predefined gesture categories may include: sliding in a certain direction, such as sliding to the left, sliding to the right, sliding up, or sliding down.

[0063] Based on whether multiple elastic wave signals have asynchronous characteristics and the change in the position of the elastic wave sensor corresponding to the time sequence of the elastic wave signals reaching their peak values, it is possible to determine whether the gesture is a sliding gesture and to determine the sliding direction when making a sliding gesture.

[0064] like Figure 3 In the diagram, the four elastic wave sensors correspond to elastic wave signals CH1, CH2, CH3, and CH4, respectively. The order of peak changes is as follows: signal CH1 shows a peak or trough -> signal CH2 shows a peak or trough -> signal CH3 shows a peak or trough -> signal CH4 shows a peak or trough, where -> indicates earlier. When signals CH1, CH2, CH3, and CH4 exhibit asynchronous characteristics, and the elastic wave sensor corresponding to signal CH1 is located at position one, and the elastic wave sensor corresponding to signal CH4 is located at position two, the gesture is determined to be a sliding gesture, and the sliding direction of the gesture is from position one to position two.

[0065] In one example, the first position is the left (or right), the second position is the right (or left), and the swipe gesture is swiped from left (or right) to right (or left).

[0066] In one example, the first position is the top (or bottom), the second position is the bottom (or top), and the sliding gesture is slid from top (or bottom) to bottom (or top).

[0067] In one example embodiment of the present invention, the gesture recognition method may further include: after acquiring the elastic wave signal, determining the valid elastic wave signal.

[0068] Determining a valid elastic wave signal may include: determining the signal start and signal end points of the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal.

[0069] In this embodiment, endpoint detection can be performed based on the amplitude of the elastic wave signal to determine the signal start endpoint and signal end endpoint of the gesture touching the touch device. The elastic wave signal between the signal start endpoint and the signal end endpoint is taken as the valid elastic wave signal. Gesture recognition is then performed based on the valid elastic wave signal, improving the accuracy of gesture recognition. The signal start endpoint can be simply referred to as the start endpoint, and the signal end endpoint can be simply referred to as the end endpoint.

[0070] In one example, determining the start and end points of a gesture touching a touch device within the elastic wave signal based on the amplitude of the elastic wave signal may include:

[0071] The start and end points of an elastic wave signal are determined based on its amplitude. The minimum time point corresponding to the amplitude whose absolute value is greater than a first threshold is taken as the start point of the signal, and the minimum time point corresponding to the amplitude whose absolute value is less than a second threshold is taken as the end point of the signal.

[0072] Endpoint detection can be performed based on the elastic wave signal detected by an elastic wave sensor. The absolute value of the amplitude of the elastic wave signal is compared with a first threshold and a second threshold to determine the start and end endpoints of the gesture touch control device signal in the elastic wave signal. The first and second thresholds can be determined based on empirical values, and this embodiment will not limit or elaborate on them.

[0073] In one example, determining the start and end points of a gesture touching a touch device within the elastic wave signal based on the amplitude of the elastic wave signal may include:

[0074] The start and end points of the signals are determined based on the amplitudes of multiple elastic wave signals. The cumulative value of the absolute amplitude of multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative value greater than the third threshold is taken as the start point of the signal, and the minimum time point corresponding to the cumulative value less than the fourth threshold is taken as the end point of the signal.

[0075] Endpoint detection can be performed based on elastic wave signals detected by multiple elastic wave sensors. The cumulative value of the absolute amplitude of the multiple elastic wave signals at the same time point is compared with a third threshold and a fourth threshold to determine the start and end endpoints of the gesture touch control device signal in the elastic wave signal. The third and fourth thresholds can be determined based on empirical values, and are not limited or described in this embodiment.

[0076] In one example, determining the start and end points of a gesture touching a touch device within the elastic wave signal based on the amplitude of the elastic wave signal may include:

[0077] The start and end points of the signal are determined based on the amplitude of multiple elastic wave signals. The cumulative average of the absolute values ​​of the amplitudes of multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative average value greater than the fifth threshold is taken as the start point of the signal, and the minimum time point corresponding to the cumulative average value less than the sixth threshold is taken as the end point of the signal.

[0078] Endpoint detection can be performed based on elastic wave signals detected by multiple elastic wave sensors. The cumulative average of the absolute values ​​of the amplitudes of the multiple elastic wave signals at the same time point is compared with a fifth threshold and a sixth threshold to determine the start and end endpoints of the gesture touch control device signal in the elastic wave signal. The fifth and sixth thresholds can be determined based on empirical values, and are not limited or elaborated here in this embodiment.

[0079] In one example, determining the valid elastic wave signal may further include: intercepting the elastic wave signal between the signal start endpoint and the signal end endpoint, and using the intercepted signal as the valid elastic wave signal. In this embodiment, the elastic wave signal between the signal start endpoint and the signal end endpoint can be directly intercepted to obtain the valid elastic wave signal.

[0080] In one example, the elastic wave signal between the start and end points of the signal is captured, and the captured signal is time-domain reduced or amplified to obtain an effective elastic wave signal of a preset length.

[0081] Due to different gestures and user operating habits, the duration of gesture signals may vary. To facilitate gesture recognition, the captured signal can be time-domain reduced or magnified to the same length, resulting in an effective elastic wave signal of n*m, where n is the signal length of each effective elastic wave signal (preset length), and m is the number of elastic wave sensors, with m being greater than or equal to 1.

[0082] When the length of the intercepted signal is greater than the preset length, signal sampling is performed to obtain an effective elastic wave signal of the preset length; when the length of the intercepted signal is less than the preset length, signal interpolation is performed to obtain an effective elastic wave signal of the preset length.

[0083] In an exemplary embodiment of the present invention, after determining the signal start endpoint and signal end endpoint of the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal, the method may further include:

[0084] Correct the start and end points of the signal: Determine the position of the maximum amplitude between the start and end points of the signal. Extend outwards from the position of the maximum amplitude to both ends of the signal. Find the left and right positions where the difference between two adjacent amplitudes is less than the difference threshold. Use the left and right positions as the corrected start and end points of the signal, respectively.

[0085] The signal start endpoint may include: a calibrated signal start endpoint, and the signal end endpoint may include: a calibrated signal end endpoint.

[0086] Endpoint correction can be performed on the start and end points of the gesture-touched touch device in the determined elastic wave signal. The position of the maximum amplitude between the start and end points is taken as the midpoint of the signal, and the signal is extended to both ends. The corrected start and end points are determined based on the difference between two adjacent amplitudes. The elastic wave signal between the corrected start and end points is determined as the valid elastic wave signal.

[0087] In one example, four elastic wave sensors are used. Figure 4 This is a schematic diagram of signal endpoint detection provided in an example embodiment of the present invention. Figure 5 This is a schematic diagram of signal endpoint correction provided in an example embodiment of the present invention, as shown below. Figure 4 and Figure 5 As shown, the elastic wave signals corresponding to the four elastic wave sensors are CH1, CH2, CH3 and CH4 respectively. The start and end points of the signals can be determined based on the elastic wave signals CH1, CH2, CH3 and CH4 corresponding to the four elastic wave sensors, and the start and end points of the signals can be calibrated.

[0088] In an exemplary embodiment of the present invention, before performing gesture recognition based on the signal characteristics conforming to the elastic wave signal, the method may further include:

[0089] The validity of a gesture is determined based on the elastic wave signal. If the gesture is valid, gesture recognition is performed based on the signal characteristics of the elastic wave signal.

[0090] In this embodiment, before performing gesture recognition, it can be determined whether the user's gesture is a valid gesture. A valid gesture means that it meets all the requirements of a gesture, including gesture position, contact duration, whether the gesture is standardized, and whether the force reaches the threshold.

[0091] Determining whether a gesture is valid can include, but is not limited to, checking whether the gesture conforms to a predefined gesture category and whether the touch location of the gesture is within a predefined valid touch area. For example, if the predefined gesture category is a short press, a long press would render the gesture invalid. Or, if the predefined valid touch area is the center of the display screen, a touch at the edge of the display screen would render the gesture invalid.

[0092] In one example, determining whether a gesture is valid based on the elastic wave signal may include:

[0093] The location of the gesture touch device is determined based on the location of the elastic wave sensor that detects the elastic wave signal. If the gesture touch device is located within the set area, the gesture is determined to be a valid gesture; if the gesture touch device is located outside the set area, the gesture is determined to be an invalid gesture.

[0094] In this embodiment, the location of the elastic wave sensor corresponding to the elastic wave signal can be used to determine whether the touch position of the gesture is within the predefined effective touch area, thereby determining whether the gesture is a valid gesture.

[0095] When the location of the elastic wave sensor corresponding to the elastic wave signal is within the predefined effective touch area, the gesture touch device is determined to be within the set area, thus determining the gesture as a valid gesture. When the location of the elastic wave sensor corresponding to the elastic wave signal is not within the predefined effective touch area, the gesture touch device is determined to be outside the set area, thus determining the gesture as an invalid gesture.

[0096] In one example, determining whether a gesture is valid based on the elastic wave signal may include:

[0097] The duration of the gesture touch on the touch device is determined based on the length of the elastic wave signal. If the duration of the gesture touch on the touch device is greater than or equal to the set duration, the gesture is determined to be a valid gesture; if the duration of the gesture touch on the touch device is less than the set duration, the gesture is determined to be an invalid gesture.

[0098] In this embodiment, the duration of the elastic wave signal can be used to determine whether the gesture is effectively triggered, thereby determining whether the gesture is a valid gesture.

[0099] If the duration of the elastic wave signal is greater than or equal to a set duration, the gesture is determined to be a valid touch, thus preventing misrecognition of gestures when the user briefly and accidentally triggers the touch device. If the duration of the elastic wave signal is less than the set duration, the gesture is determined to be an invalid touch, thus invalidating the gesture.

[0100] In one example, the method for determining whether a gesture is valid can be an existing approach, such as using a convolutional neural network (CNN) for binary classification, which categorizes gestures as valid or invalid, to identify whether a gesture is valid or invalid.

[0101] Figure 6 A flowchart of a gesture recognition method provided in another exemplary embodiment of the present invention is shown below. Figure 6 As shown, gesture recognition methods may include:

[0102] S601: Multiple elastic wave sensors collect elastic wave signals respectively.

[0103] In this embodiment, multiple elastic wave sensors can be set at different locations to collect or detect the elastic wave signals generated when a gesture touches the display screen.

[0104] S602: Capture the valid elastic wave signal after endpoint detection.

[0105] According to the endpoint detection of any of the above embodiments, after determining the signal start endpoint and the signal end endpoint, the elastic wave signal between the signal start endpoint and the signal end endpoint is intercepted as the effective elastic wave signal.

[0106] The signal start endpoint may include: the corrected signal start endpoint, and the signal end endpoint may include: the corrected signal end endpoint.

[0107] S603: Time-domain reduction or amplification of the intercepted signal to the same length.

[0108] Due to different gestures and user operating habits, the duration of gesture signals may vary. To facilitate gesture recognition, the captured signal can be time-domain reduced or magnified to the same length, resulting in an effective elastic wave signal of n*m, where n is the signal length of each effective elastic wave signal (preset length), and m is the number of elastic wave sensors, with m being greater than or equal to 1.

[0109] When the length of the intercepted signal is greater than the preset length, signal sampling is performed to obtain an effective elastic wave signal of the preset length; when the length of the intercepted signal is less than the preset length, signal interpolation is performed to obtain an effective elastic wave signal of the preset length.

[0110] S604: Determine if the gesture is valid. If valid, proceed to S605; otherwise, end.

[0111] Before performing gesture recognition, it is necessary to determine whether the user's gesture is valid. A valid gesture is one that meets all the requirements of a gesture, including gesture position, contact duration, whether the gesture is standard, and whether the force reaches the threshold.

[0112] The determination of whether a gesture is valid can be found in the gesture validity determination method shown in any of the above embodiments, and will not be repeated here.

[0113] S605: Extract gesture features.

[0114] S606: Perform gesture recognition.

[0115] In S605 and S606, gesture features can be extracted based on the effective elastic wave signal corresponding to each elastic wave sensor. These gesture features can include synchronous or asynchronous features. Based on the synchronous or asynchronous features among the multiple effective elastic wave signals corresponding to multiple elastic wave sensors, the gesture can be identified as a click or a swipe gesture. Gesture recognition can be found in any of the above embodiments regarding gesture feature extraction and gesture recognition; this embodiment will not elaborate further.

[0116] Figure 7 This is a structural block diagram of a gesture recognition device provided in an example embodiment of the present invention, as shown below. Figure 7 As shown, the gesture recognition device may include a signal processing device 71 and at least one elastic wave sensor 72, and the signal processing device may be electrically connected to each elastic wave sensor.

[0117] The signal processing apparatus may include a memory and a processor. The memory is used to store execution instructions, and the processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that implement the embodiments of the present invention.

[0118] When the signal processing device is running, the processor communicates with the memory, and the processor calls and executes instructions to perform the following operations:

[0119] Acquire elastic wave signals detected by at least one elastic wave sensor disposed on the touch device;

[0120] The signal characteristics that the elastic wave signal conforms to can include: synchronous characteristics, asynchronous characteristics, or preset click gesture waveform characteristics;

[0121] Gesture recognition is performed based on the signal characteristics of the elastic wave signal.

[0122] The signal processing device provided in this embodiment of the invention is used to execute the technical solution of any gesture recognition method in the embodiment. Its implementation principle and effect are similar, and will not be described again here.

[0123] In one example embodiment of the present invention, the number of elastic wave sensors can be multiple, and the processor performs gesture recognition based on the signal characteristics conforming to the elastic wave signals, which may include:

[0124] When there is synchronization between multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is identified as a click gesture.

[0125] When there are asynchronous characteristics among the multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is identified as a sliding gesture;

[0126] The synchronous characteristic refers to the phase difference of the multiple elastic wave signals being less than or equal to a preset time difference value, and the phase difference of the multiple elastic wave signals at the peaks or troughs being less than or equal to the preset time difference value; the asynchronous characteristic refers to the phase difference of the multiple elastic wave signals at the peaks or troughs being greater than the preset time difference value, and the signals not intersecting at the same time point.

[0127] In one exemplary embodiment of the present invention, the processor is further configured to:

[0128] After recognizing the gesture as a click gesture, multiple gesture recognition results are obtained within a preset time period;

[0129] If, within the preset time period, there is only one click gesture among the gesture recognition results of multiple consecutive detection cycles, the click gesture is determined to be a single click.

[0130] If, within the preset time period, all N consecutive detection cycles show a click gesture, then the click gesture is determined to be N clicks, where N is greater than or equal to 2.

[0131] In one exemplary embodiment of the present invention, the processor is further configured to:

[0132] After recognizing the gesture as a click gesture, if the duration of any effective elastic wave signal is greater than or equal to the first pressing time threshold, the click gesture is determined to be a pressing gesture with a preset pressing duration.

[0133] or,

[0134] After recognizing the gesture as a click gesture, the duration of each elastic wave signal among multiple elastic wave signals is determined. When the maximum duration among multiple elastic wave signals is greater than or equal to the second pressing time threshold, the click gesture is determined to be a pressing gesture with a preset pressing duration.

[0135] In one example embodiment of the present invention, it is also used in a processor:

[0136] After recognizing the gesture as a sliding gesture, the direction of the sliding gesture is determined based on the locations of multiple elastic wave sensors corresponding to the multiple elastic wave signals that appear sequentially at peaks or troughs:

[0137] When the location of the elastic wave sensor corresponding to the earliest elastic wave signal with a peak or trough is the first position, and the location of the elastic wave sensor corresponding to the latest elastic wave signal with a peak or trough is the second position, the direction of the sliding gesture is determined to be from the first position to the second position.

[0138] In one example embodiment of the present invention, the number of elastic wave sensors may be one, and the processor performs gesture recognition based on the signal characteristics conforming to the elastic wave signal, which may include:

[0139] When the waveform of the elastic wave signal is a preset click gesture waveform, the gesture is determined to be a click gesture.

[0140] In one example embodiment of the present invention, the processor is further configured to determine a valid elastic wave signal after acquiring the elastic wave signal.

[0141] The processor can determine a valid elastic wave signal by: determining the signal start endpoint and signal end endpoint of the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal;

[0142] The processor determines the start and end points of the gesture touch control device signal in the elastic wave signal based on the amplitude of the elastic wave signal, which may include:

[0143] The start and end points of an elastic wave signal are determined based on the amplitude of the elastic wave signal. The minimum time point corresponding to the amplitude whose absolute value is greater than a first threshold is taken as the start point of the signal, and the minimum time point corresponding to the amplitude whose absolute value is less than a second threshold is taken as the end point of the signal.

[0144] or,

[0145] The start endpoint and the end endpoint of the signal are determined based on the amplitude of multiple elastic wave signals. The cumulative value of the absolute value of the amplitude of multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative value greater than the third threshold is taken as the start endpoint of the signal, and the minimum time point corresponding to the cumulative value less than the fourth threshold is taken as the end endpoint of the signal.

[0146] or,

[0147] The start and end points of the signal are determined based on the amplitudes of multiple elastic wave signals. The cumulative average of the absolute values ​​of the amplitudes of the multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative average value greater than the fifth threshold is taken as the start point of the signal, and the minimum time point corresponding to the cumulative average value less than the sixth threshold is taken as the end point of the signal.

[0148] In one exemplary embodiment of the present invention, the processor is further configured to:

[0149] After determining the start and end points of the signal from the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal, the start and end points of the signal are corrected: the position of the maximum amplitude between the start and end points of the signal is determined, and the signal is extended outward from the position as the center to find the left and right positions where the difference between two adjacent amplitudes is less than the difference threshold. The left and right positions are then used as the corrected start and end points of the signal, respectively.

[0150] The signal start endpoint includes: a corrected signal start endpoint, and the signal end endpoint includes: a corrected signal end endpoint.

[0151] In one exemplary embodiment of the present invention, the processor determining the valid elastic wave signal may further include:

[0152] The elastic wave signal between the start endpoint and the end endpoint of the signal is intercepted, and the intercepted signal is time-domain reduced or amplified to obtain an effective elastic wave signal of a preset length.

[0153] or,

[0154] The elastic wave signal between the start endpoint and the end endpoint of the signal is intercepted, and the intercepted signal is taken as the effective elastic wave signal.

[0155] In one exemplary embodiment of the present invention, the processor is further configured to:

[0156] Before performing gesture recognition based on the signal characteristics conforming to the elastic wave signal, it is determined whether the gesture is a valid gesture based on the elastic wave signal. If the gesture is a valid gesture, gesture recognition is performed based on the signal characteristics conforming to the elastic wave signal.

[0157] In an exemplary embodiment of the present invention, the processor determining whether the gesture is a valid gesture based on the elastic wave signal may include:

[0158] The location of the gesture touch device is determined based on the location of the elastic wave sensor that detects the elastic wave signal. When the location of the gesture touch device is within a set area, the gesture is determined to be a valid gesture; when the location of the gesture touch device is not within the set area, the gesture is determined to be an invalid gesture.

[0159] or,

[0160] The duration of the gesture touch on the touch device is determined based on the duration of the elastic wave signal. When the duration of the gesture touch on the touch device is greater than or equal to the set duration, the gesture is determined to be a valid gesture; when the duration of the gesture touch on the touch device is less than the set duration, the gesture is determined to be an invalid gesture.

[0161] Figure 8 This is a structural block diagram of a terminal device provided in an example embodiment of the present invention, such as... Figure 8 As shown, the terminal device may include: a touch device 81 and a gesture recognition device 82 as shown in any embodiment, and the touch device may include a display screen.

[0162] Gesture recognition devices can be applied to terminal devices to recognize user gestures on the terminal device display screen, thereby improving the user's human-computer interaction experience.

[0163] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A gesture recognition method, characterized in that, include: Acquire elastic wave signals detected by multiple elastic wave sensors installed on the touch device; The signal characteristics that the elastic wave signal conforms to are determined, and the signal characteristics include: synchronous characteristics and asynchronous characteristics; Gesture recognition is performed based on the signal characteristics of the elastic wave signal. The gesture recognition based on the signal characteristics conforming to the elastic wave signal includes: When there is synchronization between multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is identified as a click gesture. When there are asynchronous characteristics among the multiple elastic wave signals corresponding to multiple elastic wave sensors, the gesture is identified as a sliding gesture; The synchronous characteristic refers to the phase difference of the multiple elastic wave signals being less than or equal to a preset time difference value, and the phase difference of the multiple elastic wave signals at the peaks or troughs being less than or equal to the preset time difference value; the asynchronous characteristic refers to the phase difference of the multiple elastic wave signals at the peaks or troughs being greater than the preset time difference value, and the signals not intersecting at the same time point.

2. The method according to claim 1, characterized in that, After recognizing the gesture as a click gesture, the method further includes: Obtain multiple gesture recognition results within a preset time period; If, within the preset time period, there is only one click gesture among the gesture recognition results of multiple consecutive detection cycles, the click gesture is determined to be a single click. If, within the preset time period, all N consecutive detection cycles show a click gesture, then the click gesture is determined to be N clicks, where N is greater than or equal to 2.

3. The method according to claim 1, characterized in that, After recognizing the gesture as a click gesture, the method further includes: When the duration of any elastic wave signal is greater than or equal to the first pressing time threshold, the click gesture is determined to be a pressing gesture with a preset pressing duration. or, The duration of each elastic wave signal among multiple elastic wave signals is determined. When the maximum duration among multiple elastic wave signals is greater than or equal to a second pressing time threshold, the click gesture is determined to be a pressing gesture with a preset pressing duration.

4. The method according to claim 1, characterized in that, After recognizing the gesture as a swipe gesture, the method further includes: The direction of the sliding gesture is determined based on the location of multiple elastic wave sensors corresponding to multiple elastic wave signals that appear successively at peaks or troughs. When the location of the elastic wave sensor corresponding to the earliest elastic wave signal with a peak or trough is the first position, and the location of the elastic wave sensor corresponding to the latest elastic wave signal with a peak or trough is the second position, the direction of the sliding gesture is determined to be from the first position to the second position.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: after acquiring the elastic wave signal, determining the effective elastic wave signal; The determination of a valid elastic wave signal includes: determining the signal start endpoint and signal end endpoint of the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal; The step of determining the signal start and end points of the gesture touch control device in the elastic wave signal based on the amplitude of the elastic wave signal includes: The start and end points of an elastic wave signal are determined based on the amplitude of the elastic wave signal. The minimum time point corresponding to the amplitude whose absolute value is greater than a first threshold is taken as the start point of the signal, and the minimum time point corresponding to the amplitude whose absolute value is less than a second threshold is taken as the end point of the signal. or, The start endpoint and the end endpoint of the signal are determined based on the amplitude of multiple elastic wave signals. The cumulative value of the absolute value of the amplitude of multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative value greater than the third threshold is taken as the start endpoint of the signal, and the minimum time point corresponding to the cumulative value less than the fourth threshold is taken as the end endpoint of the signal. or, The start and end points of the signal are determined based on the amplitudes of multiple elastic wave signals. The cumulative average of the absolute values ​​of the amplitudes of the multiple elastic wave signals at the same time point is obtained. The minimum time point corresponding to the cumulative average value greater than the fifth threshold is taken as the start point of the signal, and the minimum time point corresponding to the cumulative average value less than the sixth threshold is taken as the end point of the signal.

6. The method according to claim 5, characterized in that, After determining the start and end points of the gesture-based touch control device signal in the elastic wave signal based on the amplitude of the elastic wave signal, the method further includes: The signal start endpoint and the signal end endpoint are corrected by: determining the position of the maximum amplitude between the signal start endpoint and the signal end endpoint, expanding outwards from the position to both ends of the signal, and finding the left and right positions where the difference between two adjacent amplitudes is less than the difference threshold, and taking the left and right positions as the corrected signal start endpoint and the corrected signal end endpoint, respectively. The signal start endpoint includes: a corrected signal start endpoint, and the signal end endpoint includes: a corrected signal end endpoint; The determination of the effective elastic wave signal also includes: The elastic wave signal between the start endpoint and the end endpoint of the signal is intercepted, and the intercepted signal is time-domain reduced or amplified to obtain an effective elastic wave signal of a preset length; or, the elastic wave signal between the start endpoint and the end endpoint of the signal is intercepted, and the intercepted signal is used as the effective elastic wave signal.

7. The method according to any one of claims 1-4, characterized in that, Before performing gesture recognition based on the signal characteristics conforming to the elastic wave signal, the method further includes: The gesture is determined to be valid based on the elastic wave signal. If the gesture is valid, the gesture is recognized based on the signal characteristics of the elastic wave signal. The step of determining whether the gesture is a valid gesture based on the elastic wave signal includes: The location of the gesture touch device is determined based on the location of the elastic wave sensor that detects the elastic wave signal. When the location of the gesture touch device is within a set area, the gesture is determined to be a valid gesture; when the location of the gesture touch device is not within the set area, the gesture is determined to be an invalid gesture. or, The duration of the gesture touch on the touch device is determined based on the duration of the elastic wave signal. When the duration of the gesture touch on the touch device is greater than or equal to the set duration, the gesture is determined to be a valid gesture; when the duration of the gesture touch on the touch device is less than the set duration, the gesture is determined to be an invalid gesture.

8. A gesture recognition device, characterized in that, It includes a signal processing device and at least one elastic wave sensor, the signal processing device being electrically connected to each elastic wave sensor, for performing the gesture recognition method as described in any one of claims 1-7.

9. A terminal device, characterized in that, It includes a touch device and a gesture recognition device as described in claim 8, wherein the touch device includes a display screen.

Images