Gesture recognition method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a gesture recognition method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining current user operation data collected in a current sliding window; determining a starting point and an ending point of a gesture according to the variance of the current user operation data, and performing data interception according to the starting point and the ending point to determine gesture data; the gesture recognition is performed according to the gesture data, the gesture of the user is determined, the problem of low gesture recognition accuracy is solved, the starting point and the ending point of the gesture are accurately determined by analyzing the variance of the current user operation data, the implementation process is simple, complex operation is not needed, the operation amount is low, and the user experience is improved. The gesture data are accurately intercepted according to the starting point and the ending point of the gesture, gesture recognition is performed on the gesture data to determine the gesture of the user, and the accuracy of gesture recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a gesture recognition method, device, electronic device, storage medium and program product. Background Art

[0002] The bare-hand click recognition method based on IMU (Inertial Measurement Unit) data is a technology that uses accelerometer sensor data to detect and identify click actions performed by the user's bare hands (that is, without using any external devices or tools). This method is often used in fields such as virtual reality (VR), augmented reality (AR), smart home, and wearable devices to enhance the interactive experience between users and devices. Among them, the accelerometer in the IMU sensor is used to measure acceleration and can detect the acceleration and deceleration of hand movements. A rapid acceleration peak is generated when clicking, and this change can be detected through accelerometer data. However, since the IMU data contains noise, it will affect the accuracy of gesture recognition during bare-hand clicking. Summary of the invention

[0003] The present invention provides a gesture recognition method, device, electronic device, storage medium and program product to solve the problem of low gesture recognition accuracy.

[0004] According to one aspect of the present invention, there is provided a gesture recognition method, comprising:

[0005] Obtain the current user operation data collected in the current sliding window;

[0006] Determine the starting point and the end point of the gesture according to the variance of each of the current user operation data, intercept data according to the starting point and the end point, and determine the gesture data;

[0007] Gesture recognition is performed according to the gesture data to determine the user's gesture.

[0008] According to another aspect of the present invention, there is provided a gesture recognition device, comprising:

[0009] An operation data acquisition module is used to acquire the current user operation data collected in the current sliding window;

[0010] A gesture data determination module, used to determine the starting point and the end point of the gesture according to the variance of each current user operation data, and to intercept data according to the starting point and the end point to determine the gesture data;

[0011] The gesture recognition module is used to perform gesture recognition according to the gesture data to determine the user's gesture.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] at least one processor, and a memory communicatively coupled to the at least one processor;

[0014] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the gesture recognition method described in any embodiment of the present invention.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gesture recognition method described in any embodiment of the present invention when executed.

[0016] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the gesture recognition method according to any embodiment of the present invention is implemented.

[0017] The technical solution of the embodiment of the present invention obtains the current user operation data collected in the current sliding window; determines the starting point and end point of the gesture according to the variance of each of the current user operation data, intercepts data according to the starting point and end point, and determines the gesture data; performs gesture recognition according to the gesture data to determine the user's gesture, thereby solving the problem of low gesture recognition accuracy. By analyzing the variance of the current user operation data, the starting point and end point of the gesture are accurately determined, and the implementation process is simple, does not require complex calculations, and has a low amount of calculations. Then, the gesture data is accurately intercepted according to the starting point and end point of the gesture, and gesture recognition is performed on the gesture data to determine the user's gesture, thereby improving the accuracy of gesture recognition.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a gesture recognition method provided according to Embodiment 1 of the present invention;

[0021] Figure 2is a flowchart of a gesture recognition method provided according to Embodiment 2 of the present invention;

[0022] Figure 3 is a schematic diagram of data processed by low-pass filtering according to Embodiment 2 of the present invention;

[0023] Figure 4 is a schematic diagram of data subjected to standardization according to Embodiment 2 of the present invention;

[0024] Figure 5 is a schematic diagram of normalized data provided according to Embodiment 2 of the present invention;

[0025] Figure 6 is a schematic diagram of a variance provided according to the second embodiment of the present invention;

[0026] Figure 7 is an example diagram of a starting point and an end point provided according to the second embodiment of the present invention;

[0027] Figure 8 is an example diagram of gesture data, a starting point and an end point provided according to the second embodiment of the present invention;

[0028] Fig. 9 is a schematic diagram of gesture data provided according to Embodiment 2 of the present invention;

[0029] Fig.10 is a schematic diagram of candidate gesture data provided according to Embodiment 2 of the present invention;

[0030] Fig.11 is a schematic diagram of a target starting point and a target end point provided according to Embodiment 2 of the present invention;

[0031] Fig.12 is a structural diagram of a gesture recognition device provided according to Embodiment 3 of the present invention;

[0032] Fig.13 It is a structural schematic diagram of an electronic device for implementing the gesture recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 This is a flowchart of a gesture recognition method provided in the first embodiment of the present invention. This embodiment is applicable to the case of accurately recognizing gestures. The method can be performed by a gesture recognition device. The gesture recognition device can be implemented in the form of hardware and / or software. The gesture recognition device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0037] S101. Acquire current user operation data collected in the current sliding window.

[0038] In this embodiment, the current sliding window can be understood as a sliding window currently used to obtain data within a period of time; the current user operation data can be understood as relevant data of user operations obtained within the current sliding window.

[0039] When using virtual reality, augmented reality, wearable devices and other products, users can wear measuring devices on their wrists or other hand positions, install IMU sensors on the measuring devices, and collect corresponding data of user gestures through the IMU sensors. For example, the user's thumb and index finger pinch together to form a single / double-click vibration signal. The vibration signal is collected through the IMU sensor, and the vibration signal is recognized to determine the user's gesture.

[0040] The size of the sliding window is preset, for example, 10ms, 20ms, etc. The IMU collects the operation data triggered by the user in real time, that is, the vibration signal, and transmits the data to the execution subject. The execution subject determines the current sliding window, receives data in the current sliding window, and uses the data as the current user operation data. The current sliding window can be generated after the gesture recognition is started and updated over time, for example. For example, the size of the sliding window is preset to 10ms, the time t0 is the starting point of the current sliding window, and t0+10ms is the end point of the current sliding window. The execution subject determines the current sliding window according to the current time, and receives the user operation data within 10ms corresponding to this window as the current user operation data. The embodiment of the present application can obtain multiple current user operation data, each current user operation data is user operation data collected once, and multiple current user operation data collected multiple times can be obtained in the current sliding window.

[0041] The current sliding window can slide with time, for example, 0-10ms is a window, when the time is between 0 and 10ms, the time corresponding to the current sliding window is [0ms, 10ms), after 10ms, the sliding window moves backward, and the time corresponding to the new current sliding window is (10ms, 20ms], and the sliding window slides every 10ms to generate a new sliding window; or, the current sliding window slides every 1ms, and the current sliding window is the time corresponding to 10ms forward from the current moment. Before the 10ms, since it is less than 10ms, only the time from the current moment to time 0 can be used as the current sliding window, for example, at the 8th ms, the current sliding window is 0-8ms, at the 12th ms, the current sliding window is 2ms-12ms, and at the 13th ms, the current sliding window is 3ms-13ms.

[0042] When performing gesture recognition at the current moment, the embodiment of the present application can obtain the current user operation data in the current sliding window corresponding to the current moment.

[0043] S102: Determine the starting point and the end point of the gesture according to the variance of each current user operation data, intercept data according to the starting point and the end point, and determine the gesture data.

[0044] In this embodiment, gesture data can be understood as data representing user gesture information. The variance of each current user operation data is analyzed, and the fluctuation of the variance analysis data is determined according to the fluctuation. Whether the currently acquired data is data formed by the user performing a corresponding gesture is determined according to the fluctuation, that is, whether each current user operation data is gesture data, starting point data or end point data is determined, and then the starting point and end point of the gesture are determined. For example, if the current user operation data in the current sliding window is determined as the starting point, then wait for the subsequent data collection and analysis to determine the end point, or if the current user operation data in the current sliding window is the end point, analyze the historical data, that is, the data before the current sliding window, and determine the starting point closest to the end point. After determining the starting point and the end point, intercept the data between the starting point and the end point. At this time, the intercepted data is the data acquired before the current moment, and the data between the starting point and the end point is used as gesture data.

[0045] S103: Perform gesture recognition according to the gesture data to determine the user's gesture.

[0046] The gesture data is identified by a preset algorithm to determine the user's gesture. For example, the gesture data is identified by using the dynamic time warping (DTW) algorithm, and templates of different gestures are pre-generated, such as a single-click template and a double-click template. The gesture data is matched with the pre-generated gesture template based on the dynamic time warping (DTW) algorithm to determine the gesture template matched by the gesture data, and the user's gesture is determined according to the gesture corresponding to the matched gesture template; for example, if the gesture data successfully matches the gesture template of a single-click, the user's gesture is determined to be a single-click, and if the gesture data successfully matches the gesture template of a double-click, the user's gesture is determined to be a double-click.

[0047] The embodiment of the present invention provides a gesture recognition method, which obtains current user operation data collected in a current sliding window; determines the starting point and end point of a gesture according to the variance of each of the current user operation data, intercepts data according to the starting point and end point, and determines the gesture data; performs gesture recognition according to the gesture data to determine the user's gesture, thereby solving the problem of low accuracy in gesture recognition, accurately determining the starting point and end point of a gesture by analyzing the variance of the current user operation data, and the implementation process is simple, does not require complex calculations, and has a low amount of calculations, and then accurately intercepts the gesture data according to the starting point and end point of the gesture, performs gesture recognition on the gesture data to determine the user's gesture, thereby improving the accuracy of gesture recognition.

[0048] Embodiment 2

[0049] Figure 2 This is a flowchart of a gesture recognition method provided in Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 2 As shown, the method includes:

[0050] S201. Acquire current user operation data collected in the current sliding window.

[0051] As an optional embodiment, this optional embodiment is further optimized to include: after acquiring the current user operation data collected in the current sliding window, filtering, normalizing and standardizing the current user operation data in sequence.

[0052] The filtering algorithm is pre-set, for example, filtering is performed by a low-pass filtering algorithm. The current user operation data is low-pass filtered by a low-pass algorithm to smooth the data and remove high-frequency components, thereby extracting effective motion information; then standardization is performed, and finally normalization is performed, wherein the standardization is performed by subtracting the mean and dividing by the standard deviation so that the distribution of the data conforms to the standard normal distribution (mean is 0, variance is 1), even if the data is not strictly normally distributed, it can be approximated, and the purpose of normalization is to scale the feature data so that it falls into a specific range (-1, 1).

[0053] The embodiment of the present application can also filter, normalize and standardize the vibration data of the accelerometer collected by the imu in advance, and use the obtained data as user operation data, and then obtain the collected current user operation data in the current sliding window. At this time, the current user operation data obtained has been filtered, normalized and standardized. The processed data filters out invalid data, which can effectively improve the accuracy of the data.

[0054] For example, Figure 3 A schematic diagram of data processed by low-pass filtering is provided, wherein the yellow curve is a curve formed by the data after filtering, and the blue curve is a curve formed by the data before filtering; in the curve in the figure, the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz. Figure 4 A schematic diagram of standardized data is provided, wherein the yellow curve is a curve formed by the standardized data, and the blue curve is a curve formed by the data before standardization; in the curves in the figure, the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz. Figure 5 A schematic diagram of normalized data is provided, wherein the yellow curve is the curve formed by the normalized data, and the blue curve is the curve formed by the data before normalization; in the curves in the figure, the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz.

[0055] S202: Calculate variance according to each current user operation data.

[0056] According to the variance calculation formula, the current user operation data is substituted to calculate the variance. The current user operation data in this step may be data obtained after filtering, normalization and standardization, and the variance is calculated for this part of the data and subsequent processing is performed.

[0057] For example, Figure 6 A schematic diagram of variance is provided. The yellow curve in the figure is the data formed by the variance of each data point, and the blue curve is the processed current user operation data. In the curve in the figure, the horizontal axis is time, the unit can be s, and the vertical axis is the variance size, the unit can be hz squared.

[0058] S203: Determine the data type corresponding to the current sliding window according to the variance.

[0059] In this embodiment, the data type may be node data, gesture data, etc., which is used to distinguish whether the user has a gesture action at this time. For example, when it is determined that the user has a gesture action, the data type is determined to be gesture data; when it is determined that the user has no gesture action, the data type is determined to be node data, and the node data may indicate that the data at this time may be the starting point or end point of the user's gesture.

[0060] According to the variance, it is determined whether the current user operation data in the current sliding window has a large fluctuation. If it has a large fluctuation, it is determined that the user performs a gesture operation in the current sliding window, and the data type corresponding to the current sliding window is gesture data; if the fluctuation is small, it is determined that the user does not perform a gesture operation in the current sliding window, and the data type corresponding to the current sliding window is node data, which may be the starting point of the user gesture or the end point of the user gesture. When judging whether the current user operation data has a large fluctuation according to the variance, the corresponding relationship between the variance and the fluctuation can be pre-set, and whether it has a large fluctuation can be determined according to the corresponding relationship, or the calculation formula of the fluctuation coefficient can be pre-determined, and the variance can be substituted as a parameter in the formula for calculation to obtain the fluctuation coefficient, and whether it has a large fluctuation can be judged according to the fluctuation coefficient, or a neural network model can be pre-trained, and the variance can be used as the input of the model, and the model determines whether it has a large fluctuation and outputs it according to the pre-learned knowledge.

[0061] As an optional embodiment, this optional embodiment further optimizes determining the data type corresponding to the current sliding window based on the variance as follows: judging whether the variance is within a preset variance range, and if so, determining that the data type corresponding to the current sliding window is node data; otherwise, determining that the data type corresponding to the current sliding window is gesture data.

[0062] In this embodiment, the preset variance range can be understood as a preset variance size range, for example, 0-3, which can be determined by analyzing historical user operation data. The preset variance range is predetermined and saved. After the variance is determined, the variance is compared with the preset variance range to determine whether the variance is within the preset variance range. If so, it can be considered that the data fluctuation is small at this time, and the data type corresponding to the current sliding window is determined to be node data; otherwise, it can be considered that the data fluctuation is large at this time, and the data type corresponding to the current sliding window is determined to be gesture data. The preset variance range can be pre-set with an initial value, and then dynamically updated during the recognition process. For example, every 10 days, the preset variance range is updated according to the data collected within 10 days, or the preset variance range is updated after a certain amount of data is detected, and so on.

[0063] S204: When the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is node data, determine that the current sliding window is a candidate node.

[0064] In this embodiment, the previous sliding window can be understood as the previous sliding window. After the current sliding window slides backward, it will serve as the previous sliding window of the new current sliding window; the candidate node can be understood as the node that can be used as the starting point of the gesture, that is, the data collected at this time can be used to represent the starting point of the gesture.

[0065] If the data type corresponding to the current sliding window is node data, it means that this may be the starting point or end point of the user's gesture, and the data type of the previous sliding window corresponding to the current sliding window is determined; in the gesture recognition process, the data type of the current sliding window is determined and saved accordingly, and the saved data can be directly obtained in the subsequent data processing process. If the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is node data, it can be determined that the data corresponding to the current sliding window is not the end point of the gesture, but it is not certain whether it is the starting point of the gesture, and the current sliding window is determined to be a candidate node.

[0066] After determining that the current sliding window is a candidate node, the starting point and the end point of the gesture are not determined at this time, so the gesture cannot be recognized, and new data continues to be collected, that is, waiting to obtain the current user operation data collected in the new current sliding window.

[0067] S205. When the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is gesture data, the current sliding window is determined to be the end point of the gesture, and the previous candidate node corresponding to the current sliding window is determined to be the starting point of the gesture.

[0068] In this embodiment, the previous candidate node can be understood as the previous candidate node, that is, a candidate node that is closest to the current sliding window in time.

[0069] If the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is gesture data, then the current sliding window can be determined as the end point of the gesture, that is, the data has stabilized after a period of fluctuation, and it can be considered that the gesture operation is ended; analyze the sliding window before the current sliding window, determine the previous candidate node corresponding to the current sliding window, and determine the previous candidate node as the starting point of the gesture.

[0070] When the data type corresponding to the current sliding window is gesture data, the current sliding window indicates that the user is performing a gesture operation, and the current sliding window is neither the starting point nor the end point of the gesture.

[0071] Exemplarily, the data type corresponding to sliding window 1 is node data, the data type corresponding to sliding window 2 is node data, the data type corresponding to sliding window 3 is gesture data, the data type corresponding to sliding window 4 is gesture data, and the data type corresponding to sliding window 5 is node data. When the current sliding window is sliding window 2, the previous sliding window is sliding window 1. At this time, the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is node data, and sliding window 2 is determined to be a candidate node. When the current sliding window is sliding window 3, the data type corresponding to the current sliding window is node data. At this time, the user is performing a gesture operation, which is neither the starting point nor the end point of the gesture. When the current sliding window is sliding window 5, the previous sliding window is sliding window 4. At this time, the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is gesture data. It is determined that sliding window 5 is the end point of the gesture, the previous candidate node is sliding window 2, and sliding window 2 is the starting point of the gesture.

[0072] S206: Intercept data according to the starting point and the end point to determine gesture data.

[0073] For example, Figure 7 An example diagram of the starting point and the end point is provided, wherein point 31 on the curve is the starting point and point 32 is the end point. The diagram only shows one set of starting points and end points by way of example, and other similar points may also be considered as starting points and end points. The horizontal axis is time, the unit may be s, and the vertical axis is the variance, the unit may be hz.

[0074] For example, Figure 8An example diagram of gesture data, starting point and end point is provided, where the red rectangle represents the sliding window. At the current moment, the current sliding window can be determined and gesture recognition can be performed. Taking the sliding window within a period of time as an example, multiple sliding windows can be collected within a period of time, and each sliding window can be used as the current sliding window for gesture analysis. The final starting point and end point are shown in the figure. The data with large fluctuations between the starting point and the end point is the gesture data.

[0075] S207 . Resample the gesture data according to the slope corresponding to the gesture data to obtain evenly distributed candidate gesture data.

[0076] In this embodiment, the candidate gesture data may be understood as gesture data obtained after processing.

[0077] The intercepted gesture data may have the problem of uneven distribution. The gesture data is analyzed and processed to determine the slope corresponding to each point, or each point is fitted to determine the slopes of different points after fitting. The gesture data is resampled according to the slope to determine the positions with small data changes and the positions with large data changes. More points are sampled at the positions with large changes and fewer points are sampled at the positions with small changes to obtain uniformly distributed candidate gesture data. For example, more points are sampled at the positions with larger absolute values ​​of the slope and fewer points are sampled at the positions with smaller absolute values ​​of the slope to obtain uniformly distributed candidate gesture data.

[0078] As an optional embodiment, this optional embodiment further optimizes the gesture data by resampling according to the slope corresponding to the gesture data:

[0079] A1. The gesture data is segmented according to the slope corresponding to each data point in the gesture data and a preset slope threshold to obtain segmented areas.

[0080] In this embodiment, the slope threshold may be preset; the segmented region may be understood as a data region obtained by dividing the data points into segments according to a certain rule.

[0081] Preset the slope threshold, process each data point in turn according to the order in which each data point is collected, and determine the slope corresponding to each data point. The slope corresponding to the data point can be determined based on this data point and the previous data point or the next data point. For example, the slope is calculated based on two points, or each data point is fitted to determine the expression of the fitting curve. After the expression is determined, the slope of each data point is determined based on the expression. Compare the slope of the data point with the slope threshold. When the slope is greater than the slope threshold, it can be determined that the change speed of this data point is faster. If the slopes of the data points within a period of time are all greater than the slope threshold, or the proportion greater than the slope threshold is greater than a certain proportion, the data points within this time period are divided into a segmented area. When performing segmentation, the embodiment of the present application can perform a rough division, that is, each data point is divided into segmented areas according to the slope, and the difference in the slope size of the data points in each segmented area is within a certain range.

[0082] A2. For each segmented area, the slope of the data points in the segmented area is matched with a preset slope interval, and the segmented area is resampled according to the sampling number corresponding to the matched slope interval.

[0083] The greater the absolute value of the slope endpoint corresponding to the slope interval, the greater the number of samples corresponding to the slope interval.

[0084] In this embodiment, the slope interval can be understood as an interval consisting of two slopes of different sizes, the endpoint value of the interval is the slope endpoint, and each slope interval has two slope endpoints. The embodiment of the present application can pre-set different slope intervals, and each slope area corresponds to a sampling number. Since the slope has positive and negative, the greater the absolute value of the slope, the greater the change in data; the greater the absolute value of the slope endpoint corresponding to the slope interval, the greater the number of samples corresponding to the slope interval, that is, for different slope intervals, the absolute value of the slope endpoint of the slope interval increases, and the number of samples corresponding to the slope area increases.

[0085] Determine in advance the different slope intervals and the number of samples corresponding to each slope area. After the segmentation is completed, for each segment area, compare the slope of the data points in the segment area with the size relationship of each slope interval, determine the slope interval where the slope of the data points in the segment area is located, determine this slope interval as the matching slope interval, determine the number of samples corresponding to this slope interval, and resample the data in this segment area according to the number of samples.

[0086] For example, Fig. 9 A schematic diagram of gesture data is provided, where the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz. Fig.10A schematic diagram of candidate gesture data is provided. As shown in the figure, the data is relatively uniform after resampling, wherein the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz.

[0087] S208: Identify the candidate gesture data and determine the target starting point and the target end point.

[0088] In this embodiment, the target starting point can be understood as the re-determined, more accurate starting point of the gesture; the target end point can be understood as the re-determined, more accurate end point of the gesture. The candidate gesture data is re-identified, and the identification method can be to divide the data according to a smaller time difference t, and determine the target starting point and the target end point according to the variance of the divided data. The time difference t used for this division can be smaller than the size of the current sliding window; the target starting point and the target end point are determined through identification.

[0089] As an optional embodiment, this optional embodiment further identifies the candidate gesture data and optimizes the determination of the target starting point and the target end point as follows:

[0090] B1. Determine the peaks and troughs based on the candidate gesture data.

[0091] Perform curve fitting on the candidate gesture data, analyze the fitted curve, and determine the peaks and troughs. The peaks are usually the user's gestures. Generally, one peak will appear when the user clicks, and two peaks will appear when the user double-clicks.

[0092] B2. Analyze the positions of the peaks and troughs, and determine the two troughs on both sides of the peak.

[0093] Analyze the positions of each peak and trough, that is, determine the time when the peak and the time when the trough appear, and determine the two troughs on both sides of the peak. The number of peaks between the two troughs can be one or more, usually one or two.

[0094] B3. Use the two troughs as the target starting point and target end point respectively.

[0095] According to the time, the trough that appears first is regarded as the starting point of the target, and the trough that appears later is regarded as the end point of the target.

[0096] For example, Fig.11 A schematic diagram of a target start point and a target end point is provided, where point 41 is the target start point and point 42 is the target end point; wherein the horizontal axis is time, the unit may be s, and the vertical axis is frequency, the unit may be hz.

[0097] S209: intercepting candidate gesture data according to the target starting point and the target end point to determine data to be recognized.

[0098] In this embodiment, the data to be identified can be understood as data with identification requirements. The data between the target starting point and the target end point in the candidate gesture data is intercepted and this part of the data is used as the data to be identified.

[0099] S210: Perform gesture recognition on the data to be recognized to determine the user's gesture.

[0100] Perform gesture recognition based on the data to be recognized by the DTW algorithm to determine the user's gesture.

[0101] It has been proved by experiments that when gesture recognition is performed directly on gesture data without resampling, the single-click and double-click recognition accuracy is 85.5856%, of which the single-click accuracy is 96.4912% and the double-click accuracy is 74.0741%; after resampling using the method provided in the embodiment of the present application, the data is relatively uniform, and a more accurate target starting point and target end point are re-determined, so the recognition accuracy is higher, and the single-click and double-click recognition accuracy is 98.6395%, of which the single-click accuracy is 100% and the double-click accuracy is 97.4359%. In addition, the implementation process of the embodiment of the present application is simple and can be deployed on edge devices to achieve high-precision recognition of bare-hand clicks; the existing technology usually uses AI algorithms to realize gesture recognition. Due to the large amount of computational complexity of AI algorithms, they are usually deployed in the cloud. The cloud communicates with the edge device to obtain data for recognition, and then feeds back the recognition results to the edge device. The delay is large and cannot meet the real-time requirements.

[0102] The embodiment of the present invention provides a gesture recognition method, which solves the problem of low gesture recognition accuracy. By analyzing the variance of current user operation data, the starting point and end point of the gesture are accurately determined. The implementation process is simple, no complex calculations are required, and the amount of calculation is low. The gesture data is accurately intercepted according to the starting point and end point of the gesture, and the gesture data is resampled according to the slope of the gesture data to obtain evenly distributed candidate gesture data. The target starting point and target end point are re-identified to obtain more accurate data to be recognized. Gesture recognition is then performed based on the data to be recognized to determine the user's gesture, thereby improving the accuracy of gesture recognition and achieving high-precision recognition. In addition, the implementation process of the embodiment of the present application is simple and can be deployed on edge devices to meet real-time requirements. On the basis of high click accuracy, it has high real-time response capabilities to ensure the immediacy of user operations and the smoothness of interaction.

[0103] Embodiment 3

[0104] Fig.12 This is a schematic diagram of the structure of a gesture recognition device provided by Embodiment 3 of the present invention. Fig.12 As shown, the device includes: an operation data acquisition module 51, a gesture data determination module 52 and a gesture recognition module 53.

[0105] The operation data acquisition module 51 is used to acquire the current user operation data collected in the current sliding window;

[0106] A gesture data determination module 52, configured to determine a start point and an end point of a gesture according to the variance of each of the current user operation data, and to intercept data according to the start point and the end point to determine gesture data;

[0107] The gesture recognition module 53 is used to perform gesture recognition according to the gesture data to determine the user's gesture.

[0108] The embodiment of the present invention provides a gesture recognition device, which obtains current user operation data collected in a current sliding window; determines the starting point and end point of a gesture according to the variance of each of the current user operation data, intercepts data according to the starting point and end point, and determines the gesture data; performs gesture recognition according to the gesture data to determine the user's gesture, thereby solving the problem of low accuracy in gesture recognition, accurately determining the starting point and end point of a gesture by analyzing the variance of the current user operation data, and the implementation process is simple, does not require complex calculations, and has a low amount of calculations, and then accurately intercepts the gesture data according to the starting point and end point of the gesture, performs gesture recognition on the gesture data to determine the user's gesture, thereby improving the accuracy of gesture recognition.

[0109] Optionally, the gesture data determination module 52 includes:

[0110] A variance calculation unit, used for calculating the variance according to each of the current user operation data;

[0111] A data type determining unit, configured to determine the data type corresponding to the current sliding window according to the variance;

[0112] A candidate node determination unit, configured to determine that the current sliding window is a candidate node when the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is node data;

[0113] A starting point and end point determination unit is used to determine that the current sliding window is the end point of the gesture and the previous candidate node corresponding to the current sliding window is the starting point of the gesture when the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is gesture data.

[0114] Optionally, the data type determination unit is specifically used to: determine whether the variance is within a preset variance range, and if so, determine that the data type corresponding to the current sliding window is node data; otherwise, determine that the data type corresponding to the current sliding window is gesture data.

[0115] Optionally, the gesture recognition module 53 includes:

[0116] a resampling unit, configured to resample the gesture data according to a slope corresponding to the gesture data to obtain evenly distributed candidate gesture data;

[0117] A candidate gesture recognition unit, used to recognize the candidate gesture data and determine a target starting point and a target end point;

[0118] a unit for determining data to be identified, configured to intercept the candidate gesture data according to the target starting point and the target end point, and determine the data to be identified;

[0119] The gesture determination unit is used to perform gesture recognition on the data to be recognized to determine the user's gesture.

[0120] Optionally, the resampling unit is specifically used to: segment the gesture data according to the slope corresponding to each data point in the gesture data and a preset slope threshold to obtain segmented areas; for each segmented area, match the slope of the data point in the segmented area with a preset slope interval, and resample the segmented area according to the number of samples corresponding to the matched slope interval; wherein, the larger the absolute value of the slope endpoint corresponding to the slope interval, the larger the number of samples corresponding to the slope interval.

[0121] Optionally, the candidate gesture recognition unit is specifically used to: determine the peak and the trough according to the candidate gesture data; analyze the positions of the peak and the trough to determine two troughs on both sides of the peak; and use the two troughs as the target starting point and the target end point, respectively.

[0122] Optionally, the device further comprises:

[0123] The data processing module is used to filter, normalize and standardize the current user operation data in sequence.

[0124] The gesture recognition device provided in the embodiment of the present invention can execute the gesture recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] Embodiment 4

[0126] Fig.13A schematic diagram of the structure of an electronic device 60 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0127] like Fig.13 As shown, the electronic device 60 includes at least one processor 61, and a memory connected to the at least one processor 61, such as a read-only memory (ROM) 62, a random access memory (RAM) 63, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 61 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 62 or the computer program loaded from the storage unit 68 to the random access memory (RAM) 63. In the RAM 63, various programs and data required for the operation of the electronic device 60 can also be stored. The processor 61, the ROM 62, and the RAM 63 are connected to each other via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0128] A number of components in the electronic device 60 are connected to the I / O interface 65, including: an input unit 66, such as a keyboard, a mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 68, such as a disk, an optical disk, etc.; and a communication unit 69, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 69 allows the electronic device 60 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The processor 61 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 61 executes the various methods and processes described above, such as a gesture recognition method.

[0130] In some embodiments, the gesture recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 60 via the ROM 62 and / or the communication unit 69. When the computer program is loaded into the RAM 63 and executed by the processor 61, one or more steps of the gesture recognition method described above may be performed. Alternatively, in other embodiments, the processor 61 may be configured to perform the gesture recognition method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] An embodiment of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the gesture recognition method described in any embodiment of the present invention is implemented.

[0134] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A gesture recognition method, characterized in that: include: Obtain the current user operation data collected in the current sliding window; Determine the starting point and the end point of the gesture according to the variance of each of the current user operation data, intercept data according to the starting point and the end point, and determine the gesture data; Gesture recognition is performed according to the gesture data to determine the user's gesture.

2. The method according to claim 1, characterized in that The step of determining the starting point and the end point of the gesture according to the variance of each current user operation data comprises: Calculating variance according to each of the current user operation data; Determine the data type corresponding to the current sliding window according to the variance; When the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is node data, determining that the current sliding window is a candidate node; When the data type corresponding to the current sliding window is node data and the data type of the previous sliding window corresponding to the current sliding window is gesture data, the current sliding window is determined to be the end point of the gesture, and the previous candidate node corresponding to the current sliding window is determined to be the starting point of the gesture.

3. The method according to claim 2, characterized in that The determining, according to the variance, the data type corresponding to the current sliding window includes: It is determined whether the variance is within a preset variance range. If so, it is determined that the data type corresponding to the current sliding window is node data; otherwise, it is determined that the data type corresponding to the current sliding window is gesture data.

4. The method according to claim 1, characterized in that The performing gesture recognition according to the gesture data to determine the user's gesture includes: Resampling the gesture data according to the slope corresponding to the gesture data to obtain evenly distributed candidate gesture data; Identify the candidate gesture data to determine a target starting point and a target end point; Intercepting the candidate gesture data according to the target starting point and the target end point to determine data to be recognized; Perform gesture recognition on the data to be recognized to determine the user's gesture.

5. The method according to claim 4, characterized in that The resampling the gesture data according to the slope corresponding to the gesture data includes: The gesture data is segmented according to the slope corresponding to each data point in the gesture data and a preset slope threshold to obtain segmented areas; For each segmented area, the slope of the data points in the segmented area is matched with a preset slope interval, and the segmented area is resampled according to the number of samples corresponding to the matched slope interval; The greater the absolute value of the slope endpoint corresponding to the slope interval is, the greater the number of samples corresponding to the slope interval is.

6. The method according to claim 4, characterized in that The step of identifying the candidate gesture data and determining a target starting point and a target end point includes: determining peaks and troughs according to the candidate gesture data; Analyze the positions of the wave crest and the wave trough to determine two wave troughs on both sides of the wave crest; The two troughs are taken as the target starting point and target end point respectively.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the current user operation data collected in the current sliding window, the method further includes: The current user operation data is filtered, normalized and standardized in sequence.

8. A gesture recognition device, characterized in that: include: An operation data acquisition module is used to acquire the current user operation data collected in the current sliding window; A gesture data determination module, used to determine the starting point and the end point of the gesture according to the variance of each current user operation data, and to intercept data according to the starting point and the end point to determine the gesture data; The gesture recognition module is used to perform gesture recognition according to the gesture data to determine the user's gesture.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the gesture recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gesture recognition method according to any one of claims 1 to 7 when executed.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the gesture recognition method according to any one of claims 1 to 7.

Images