Gesture interaction method, system and apparatus

Abstract

This application provides a gesture interaction method. Implementing this method, smartwatches and other smart wearable electronic devices can acquire multiple types of data when a user performs a specific gesture. By recognizing the user's gestures using this multi-type data, the accuracy and robustness of gesture recognition can be improved, reducing the probability of misrecognition. In particular, when the user's environment changes significantly, smartwatches and other electronic devices can accurately recognize the user's gestures. Therefore, users can obtain a better user experience when using gesture recognition to control the operation of electronic devices.

Description

Technical Field

[0001] This application relates to the field of smart wearables, and more particularly to gesture interaction methods, systems and devices. Background Technology

[0002] Currently, gesture recognition technology on smart wearable devices has poor environmental adaptability. When the environment changes significantly, the recognition ability of these technologies weakens, resulting in poor recognition performance. Consequently, the user experience based on gesture recognition is subpar. Summary of the Invention

[0003] This application provides a gesture interaction method. By implementing this method, smart wearable electronic devices such as smartwatches can combine multiple types of data on the gestures performed by the user to recognize the gestures, thereby improving the accuracy of gesture recognition and providing users with a better gesture interaction experience.

[0004] In a first aspect, embodiments of this application provide a gesture interaction method applied to an electronic device. The method includes: acquiring first posture data via an inertial motion unit; acquiring pulse wave data via a pulse wave sensor while acquiring the first posture data; determining that a user has made a first gesture based on the first posture data and the pulse wave data; and performing a first operation in response to determining that the user has made a first gesture.

[0005] By implementing the method provided in the first aspect, electronic devices can recognize user gestures using user posture data and pulse wave data, thereby controlling the operation of the electronic device. In particular, gesture recognition algorithms that combine posture data and pulse wave data can improve recognition accuracy and robustness. This reduces the probability of misrecognition by the electronic device and also enhances the user experience of controlling the electronic device with gestures.

[0006] In some embodiments, in conjunction with the first aspect, before acquiring the first attitude data via the inertial motion unit, the method further includes: acquiring second attitude data; determining, based on the second attitude data, that the user has performed a preparatory action; and activating the pulse wave sensor in response to determining that the user has performed the preparatory action.

[0007] By implementing the method provided in the above embodiments, the electronic device can turn off the pulse wave sensor and only turn on the inertial motion unit when gesture recognition is not performed, thereby reducing the power consumption of the electronic device and improving its battery life.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: turning on the screen in response to determining that the user has performed the pre-action.

[0009] In this way, users can determine whether an electronic device has entered gesture recognition mode by whether the screen of the electronic device is lit up.

[0010] In some embodiments of the first aspect, before determining that the user has made the first gesture using the first posture data and the pulse wave data, the method further includes: acquiring the user's location data; confirming that the user is in a first location based on the location data; and performing the first operation if the user is detected making the first gesture at the first location.

[0011] By implementing the method provided in the above embodiments, an electronic device can associate a gesture with multiple operations. Then, based on the location of the electronic device, it can determine which of the multiple operations should be executed at the current location. In this way, users can control more services with as few gestures as possible, thereby further improving the user experience.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, determining that a user has made a first gesture using the first posture data and the pulse wave data specifically includes: using a first window to obtain a first posture data block and a first pulse wave data block from the first posture data and the pulse wave data, the window having a first length; filtering the first posture data block and the first pulse wave data block to obtain a second posture data block and a second pulse wave data block; calculating a first feature of the second posture data block and the second pulse wave data block; and using the feature to determine that the user has made a first gesture.

[0013] By implementing the method provided in the above embodiments, the electronic device can extract features contained in posture data and pulse wave data. By comparing these features with the features of learned gestures, the electronic device can recognize the gestures made by the user.

[0014] In some embodiments, in conjunction with the first aspect, the method further includes: acquiring location data; if the location data is determined to be a first location and the user's gesture is identified as not being the first gesture, displaying a first interface, the first interface being used to prompt the user to repeat the previous gesture.

[0015] By implementing the method provided in the above embodiments, an electronic device can determine the gesture that a user is most likely to perform at a given location using location data. When recognizing a gesture made by the user, if the recognized result is inconsistent with the gesture described above, the electronic device can recognize it again, thereby improving the accuracy of gesture recognition results in specific scenarios.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, performing the first operation specifically includes: displaying a user interface containing a payment QR code.

[0017] By implementing the method provided in the above embodiments, after the electronic device recognizes a gesture associated with a payment transaction, it can display a user interface containing a payment QR code. The user then completes the payment operation using the payment QR code. This saves the user the need to manually open the payment QR code, improving the user experience.

[0018] In some embodiments, in conjunction with the first aspect, the method further includes: in response to determining that the user has performed the preparatory action, turning on the microphone; acquiring sound signals through the microphone; and determining that the user has performed the first gesture through the first posture data and the pulse wave data further includes: determining whether the user has performed the first gesture through the first posture data, the pulse wave data, and the sound signal.

[0019] By implementing the method provided in the above embodiments, electronic devices can combine posture data, pulse wave data, and sound signals of gestures to recognize user gestures, further improving the accuracy of gesture recognition and enhancing the user experience.

[0020] In conjunction with some embodiments of the first aspect, in some embodiments, determining whether the user has made a first gesture is done using the first posture data, the pulse wave data, and the sound signal. Specifically, this includes determining whether the user has made a first gesture using the first posture data, the first feature of the pulse wave data, and the frequency feature of the sound signal.

[0021] By implementing the method provided in the above embodiments, electronic devices can recognize user gestures by using the characteristics of posture data and pulse wave data, as well as the frequency characteristics of sound signals.

[0022] In conjunction with some embodiments of the first aspect, in some embodiments, the first feature includes: trough feature, vibration feature, peak factor, waveform factor, root mean square frequency, and two or more features characterizing the dispersion and concentration of the spectrum.

[0023] By implementing the method provided in the above embodiments, the electronic device can determine whether the posture data and pulse wave data match the posture data and pulse wave data of a learned gesture by calculating the performance of the posture data and pulse wave data in terms of the above features, and thus determine whether the gesture was made by the user.

[0024] In conjunction with some embodiments of the first aspect, in some embodiments, the attitude data includes: X-axis acceleration data, Y-axis acceleration data, Z-axis acceleration data, and one or more triaxial acceleration amplitude data; the pulse wave data includes: one or more infrared light data and green light data.

[0025] By implementing the method provided in the above embodiments, the electronic device can recognize the user's gestures using the aforementioned data.

[0026] In a second aspect, embodiments of this application provide an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more processors, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0027] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0028] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0029] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0030] It is understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a system diagram provided in the embodiments of this application;

[0032] Figure 2A This is a flowchart of a gesture interaction method provided in an embodiment of this application;

[0033] Figures 2B-2C This is a set of user gesture diagrams provided in the embodiments of this application;

[0034] Figure 3A This is a flowchart illustrating the learning process of an electronic device for learning user gestures, as provided in an embodiment of this application.

[0035] Figure 3BThis is a set of training data provided in the embodiments of this application;

[0036] Figure 4 This is a timing diagram provided in the embodiments of this application;

[0037] Figure 5 This is a flowchart of another gesture interaction method provided in an embodiment of this application;

[0038] Figure 6A , Figure 6B Here are flowcharts of two other gesture interaction methods provided in the embodiments of this application;

[0039] Figure 7 This is a flowchart of another gesture interaction method provided in an embodiment of this application;

[0040] Figure 8 This is a hardware structure diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0041] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0042] Human-computer interaction in smartwatches, fitness trackers, and other electronic devices is primarily achieved through the touchscreen of the watch face. For example, users can control these devices by tapping controls displayed on the watch face. Compared to smartphones and tablets, due to screen size limitations, smartwatches and fitness trackers struggle to support more intuitive user interface designs, leading to cumbersome operation. Furthermore, the smaller screen size of the watch face results in smaller elements within the user interface, making accidental touches more likely. This further degrades the user experience.

[0043] Gesture recognition technology can solve the problems of cumbersome operation and accidental touches mentioned above. Gesture recognition technology here refers to wearable devices worn on the user's hand, such as watches and bracelets, which identify the user's gestures by acquiring hand posture data or physiological data.

[0044] However, existing gesture recognition technologies have poor generalization capabilities. When the scene changes significantly, the recognition results are unstable and the accuracy is low. For example, gesture recognition technology based on neural conduction sensors has good recognition performance when the neural conduction sensor is in close contact with the skin. However, when the user wears the watch loosely, causing the sensor to not be in close contact with the skin, the gesture recognition performance is not ideal, or in hot and sweaty conditions, the recognition performance of this technology is also not ideal.

[0045] Here, "scene" refers to the environment in which the user makes the gesture, such as an outdoor walking environment or a well-lit playground.

[0046] Other gesture recognition technologies, such as those based on low-frequency inertial measurement units (IMUs), photoplethysmography (PPG), and ultrasonic or millimeter-wave radar, also suffer from the same problem: their recognition performance is unsatisfactory in certain scenarios. For example, when the user is not stationary, such as walking outdoors, IMU-based gesture recognition performs poorly. Under strong sunlight, PPG-based gesture recognition performs poorly.

[0047] To enhance the scenario generalization ability of gesture recognition technology and ensure good recognition performance in different specific scenarios, i.e., to improve the robustness of gesture recognition technology, embodiments of this application provide a gesture interaction method, system, and device. This method can be applied to smart wearable electronic devices such as smartwatches and wristbands.

[0048] Taking a smartwatch as an example, in implementing this method, the smartwatch can simultaneously acquire data indicating the type of gesture (gesture feature data) from multiple sensors. For example, data containing gesture posture features collected by an IMU, and data containing gesture pulse wave features collected by a pulse wave sensor. By learning the gesture feature data of different gestures sent by these multiple sensors, the smartwatch can obtain a fusion model, i.e., a gesture classifier. Through the gesture classifier, the smartwatch can recognize gesture feature data, and thus recognize the gestures performed by the user. Then, based on the association between the gesture and the service, the smartwatch can invoke the service of the application associated with that gesture, and further, the smartwatch can execute that service.

[0049] For example, suppose the finger-rubbing gesture is associated with displaying a payment QR code. When the smartwatch recognizes the finger-rubbing gesture, it can activate this function based on the aforementioned matching relationship. The user can then use the payment QR code to complete the payment. In this way, the user can control the smartwatch to display the payment QR code with just one finger-rubbing gesture, avoiding the tedious operation of repeatedly touching the screen and improving the user experience.

[0050] The embodiments of this application will be combined with Figure 1 A system that introduces gesture interaction methods.

[0051] like Figure 1As shown, taking the finger-rubbing gesture as an example, the scenario for implementing the gesture interaction method includes a user (hand) 11 and a smartwatch 12. The smartwatch 12 may include a watch face 121 and a watch strap 122.

[0052] Multiple sensors are installed within the dial 121. These sensors can be used to acquire various signals or data during the user's finger-rubbing gesture. The sensors include an inertial motion unit (IMU) 1211 and a pulse wave sensor 1212. The IMU 1211 can acquire the user's hand posture data when performing a specific gesture. Here, posture data refers to data describing the gesture's position, specific shape, and corresponding bio-vibration waves. The pulse wave sensor 1212 can acquire the user's pulse wave data when performing a specific gesture.

[0053] The inertial motion unit 1211 (IMU) includes an accelerometer 1213 and a gyroscope 1214. The accelerometer 1213 measures the acceleration data along the X, Y, and Z axes when a specific gesture is performed. The gyroscope 1214 measures the angular velocity data along the X, Y, and Z axes when a specific gesture is performed. Through these two sensors, the IMU can identify the position and shape of the user's gesture.

[0054] The pulse wave sensor 1212 includes a light-emitting diode (LED) 1215 and a photodetector 1216 (e.g., a photodiode, PD). The LED 1215 emits light (emitted light) as the user performs a specific gesture. The photodetector 1216 collects the emitted light reflected by blood vessels in the hand (reflected light). Changes in blood vessel volume and blood flow velocity cause changes in the reflected light. Therefore, the pulse wave sensor 1212 can describe changes in blood vessel volume and blood flow velocity by detecting changes in the reflected light, and thus describe changes in the pulse wave when the user performs a specific gesture.

[0055] For example, the emitted light can be green light, infrared light, etc. This application does not limit this. In some embodiments, the emitted light may also include multiple light sources, such as simultaneously using two light-emitting diodes (green light and infrared light).

[0056] During the process of the user performing the finger-rubbing gesture, the user's hand can be displayed Figure 1 The shape shown. Combined with... Figure 1 The above finger-rubbing gesture refers to the gesture of touching the thumb and index finger (or other fingers) and rubbing them back and forth 2-3 times.

[0057] During the aforementioned action, the user's hand undergoes deformation and vibration. The volume of blood vessels and the flow rate of blood in the hand change due to these vibrations and deformations, resulting in changes in the reflected light passing through the hand. At this time, the inertial motion unit 1211 can collect the signals of these vibrations and deformations and generate posture data for the finger-rubbing gesture. Simultaneously, the pulse wave sensor 1212 can collect the reflected light from the hand and generate pulse wave data for the finger-rubbing gesture. Due to the back-and-forth rubbing motion, the posture data and pulse wave data of the finger-rubbing gesture exhibit periodic changes. Therefore, the smartwatch can determine whether the user's gesture is a finger-rubbing action by observing these periodic changes.

[0058] Different gestures correspond to different posture and pulse wave data. Based on the above conclusions, smartwatches can analyze the characteristics of different posture and pulse wave data to determine which type of gesture the posture and pulse wave data indicate.

[0059] Specifically, during the finger-rubbing gesture, the posture data of the finger-rubbing gesture collected by the inertial motion unit 1211 and the pulse wave data of the finger-rubbing gesture collected by the pulse wave sensor 1212 may contain features of the finger-rubbing action. For example, the time-domain trough characteristics, vibration characteristics, peak factor, waveform factor, etc. of the above-mentioned posture data and pulse wave data are used to describe the time-domain waveform of the posture data and pulse wave data.

[0060] After inputting the aforementioned posture data and pulse wave data into the gesture classifier, the classifier can extract the finger-rubbing feature. Furthermore, the gesture classifier can determine that the gesture indicated by the posture data and pulse wave data is a finger-rubbing gesture. Thus, the smartwatch can determine that the user has performed the finger-rubbing gesture, and in response, the smartwatch can invoke services associated with the finger-rubbing gesture, such as displaying a payment QR code.

[0061] The following embodiments of this application will be combined with Figure 2A The specific process for implementing gesture interaction methods will be explained in detail.

[0062] S201: Initial state (IMU is working, pulse wave sensor is in sleep mode).

[0063] like Figure 2AAs shown, the smartwatch can be in an initial state before the user makes a specific gesture. At this time, the smartwatch's inertial motion unit (IMU) can be in an active state, while the pulse wave sensor can be in a sleep state. Here, the active state refers to the state where the IMU can detect the user's hand movements and acquire posture data in real time. Conversely, in the sleep state, the pulse wave sensor does not emit light nor detect reflected light; therefore, it cannot acquire the user's pulse wave data. The pulse wave sensor in the sleep state can be woken up, thus entering the active state.

[0064] S202: The smartwatch collects the user's posture data and recognizes the user's pre-action actions.

[0065] Before performing a specific gesture, a user will first perform a preparatory action. For example, before performing the finger-rubbing gesture, the user will first raise their naturally hanging wrist (wrist raise), which can be considered a preparatory action. Besides wrist raises, preparatory actions can also be other types of hand movements, which will not be listed here. Smartwatches can recognize these preparatory actions.

[0066] Specifically, during the preliminary movement, the user's arm and wrist may exhibit significant movement. The position and shape of the arm and wrist change accordingly. At this time, the IMU-collected hand posture data will also show significant changes. Therefore, during this preliminary movement, the IMU-collected hand posture data can contain the positional and morphological characteristics of the movement. Thus, the smartwatch can use the IMU to detect whether the user has completed the aforementioned preliminary movement.

[0067] refer to Figure 2B During the wrist-raising process, the user's upper arm and forearm can naturally hang down in a position and shape. Figure 2B ) Transform to a position and shape where the upper arm and forearm form an approximately right angle ( Figure 2C For example, the IMU's accelerometer X-axis can be determined by... Figure 2B The vertical downward change shown is Figure 2C The direction shown is horizontal to the right. At this time, the IMU-collected user hand posture data can include the aforementioned changes. When these changes are detected, the smartwatch can confirm that the user has completed the forward movement. The method described above for identifying whether the user has made a forward movement using an X-axis accelerometer can be called horizontal detection.

[0068] Beyond the methods shown in the horizontal detection, smartwatches can also determine whether a user has made a preliminary movement by detecting changes in the height of their wrist using an IMU. These height changes are preset. For example, the height change value could be 50cm. When the smartwatch detects that the wrist height has increased by 50cm or more, it can determine that the user has made a preliminary movement.

[0069] If the user does not complete the preceding action, the IMU cannot collect posture data containing the characteristics of the preceding action, and therefore the smartwatch cannot detect the preceding action. The posture data collected by the IMU is continuous, so the smartwatch can continuously identify whether the user has performed a preceding action.

[0070] S203: The smartwatch controls the pulse wave sensor to enter working mode.

[0071] After recognizing the aforementioned pre-action actions, the smartwatch can activate its pulse wave sensor. At this point, both the IMU and the pulse wave sensor become active. The smartwatch can then simultaneously acquire the user's posture data and pulse wave data.

[0072] Specifically, after recognizing a user's pre-action gesture, the smartwatch sends a control signal to the pulse wave sensor. In response to this signal, the pulse wave sensor transitions from sleep mode to active mode. In active mode, the pulse wave sensor's light-emitting diode (LED) emits green light, and a photodetector captures the green light reflected from the blood vessels, converting the reflected light signal into an electrical signal and recording it, thus obtaining the user's pulse wave data.

[0073] Before recognizing a preceding action, the smartwatch can activate only the IMU. The pulse wave sensor only becomes active after the IMU detects the user's action. This allows the smartwatch to use the pulse wave sensor only during gesture recognition. At other times, the smartwatch doesn't need to activate all sensors, thus reducing overall power consumption and extending battery life.

[0074] After recognizing the user's pre-action gesture, the smartwatch can also light up the screen, entering screen-on mode. In screen-on mode, both the IMU and pulse wave sensor are active. Simultaneously, the user can also know that the smartwatch has entered gesture recognition mode by checking the screen.

[0075] S204: The smartwatch collects posture data and pulse wave data when the user completes a gesture, and identifies the user's specific gesture through the above data.

[0076] As explained above, after recognizing the preceding action, both the smartwatch's IMU and pulse wave sensor are activated. This means that after recognizing the preceding action, the smartwatch can simultaneously acquire the user's posture data and pulse wave data after completing the action.

[0077] After completing the preliminary actions, users can make specific gestures. For example, after raising their wrist, users can immediately make a finger-rubbing gesture. Alternatively, users can also know that the smartwatch's gesture recognition function is enabled by turning on the screen, allowing them to perform specific gestures to instruct the smartwatch to complete corresponding tasks.

[0078] At this point, the posture data and pulse wave data collected by the IMU and pulse wave sensor can include features of specific gestures. Based on these features, the smartwatch can recognize the gestures made by the user. For example, the posture data and pulse wave data can include features of a finger-rubbing gesture (the periodic changes in data caused by pressing and rubbing back and forth between the thumb and index or middle finger). Furthermore, based on these features, the smartwatch can recognize the finger-rubbing gesture made by the user.

[0079] Here, specific gestures refer not only to general, typical gestures, such as clenching a fist, snapping fingers, and... Figure 1 The gestures shown include finger rubbing. Furthermore, a specific gesture indicates that it is a gesture that the gesture classifier has learned, meaning a gesture that the smartwatch can recognize through the gesture classifier.

[0080] Specifically, a smartwatch may include a gesture classifier. The gesture classifier is a model that the smartwatch learns from the feature data of different gestures. This model records the features of all the learned gestures. The model can determine whether the data input to it belongs to one of the previously learned gestures by extracting the features from the input data.

[0081] The process by which a smartwatch learns gesture feature data through a gesture recognition algorithm to obtain a gesture classifier will be described in detail in subsequent embodiments, and will not be elaborated here.

[0082] The smartwatch recognizes the user's gestures within a preset time period. It can identify posture and pulse wave data generated within that time. For example, the smartwatch might determine that 7 seconds after the pulse wave sensor becomes active is the time to recognize a specific user gesture.

[0083] During these 7 seconds, both the IMU and pulse wave sensor are active, allowing the smartwatch to continuously acquire posture and pulse wave data. While the IMU and pulse wave sensor are collecting posture and pulse wave data, they can simultaneously acquire data and send it to the gesture classifier. The posture and pulse wave data input to the gesture classifier can be referred to as input data.

[0084] Specifically, the IMU and pulse wave sensor can send the collected data to the gesture classifier in fixed-length data packets. For example, the IMU can send all 10 data points to the gesture classifier simultaneously after collecting them. The same applies to the pulse wave sensor.

[0085] After receiving 100 data points from the IMU (Initial Mutor Unit), i.e., after receiving data from the aforementioned sensors 10 times, the gesture classifier can define these 100 data points as a set of posture input data. This process can be described as: dividing the collected posture data and pulse wave data using a window of length 100. Similarly, the gesture classifier can also receive posture data from the pulse wave sensor and divide it into sets of 100 points each. A set of posture input data and a set of pulse wave input data within the same time period can be used by the gesture classifier to identify whether the user has made a specific gesture within that same time period.

[0086] It is understood that the above 10 data points and 100 data points are merely illustrative examples and should not be construed as limiting the embodiments of this application.

[0087] After acquiring the input data, the gesture classifier filters the data and calculates its features. By comparing these features with the features of gestures already learned by the classifier, it can recognize the gesture indicated by the input data.

[0088] For example, during the process of a user performing a finger-rubbing gesture, the gesture classifier receives a set of input data (100 posture data points and 100 pulse wave data points within the same time period). The gesture classifier can calculate the features of the input data, such as the peaks and troughs of the posture data. Then, the gesture classifier can determine that the features of the data match the features of the learned finger-rubbing gesture. At this point, the smartwatch confirms that the user has performed the finger-rubbing gesture.

[0089] Within the aforementioned preset time period, gesture classifiers often fail to identify whether a user has made a specific gesture based on the first input data. This is because the time unit for dividing data blocks by the machine is often milliseconds. Within the corresponding time of several (or the first few dozen data blocks), the user may not have made a gesture yet, or may have just made a gesture but not yet completed it.

[0090] Meanwhile, while the gesture classifier is recognizing the first input data, the IMU and pulse wave sensor are also collecting the user's posture data and pulse wave data. Therefore, the IMU and pulse wave sensor are continuously sending posture data and pulse wave data to the gesture classifier.

[0091] Therefore, when a user's specific gesture cannot be recognized using the first input data described above, the gesture classifier can re-determine a set of input data from the posture data and pulse wave data sent by the IMU and pulse wave sensor. Here, re-determining a set of input data can be achieved through a sliding window as described in subsequent embodiments. Figure 4 That's all for now, we won't go into details here.

[0092] Then, the gesture classifier can re-identify whether the user has made a specific gesture. That is to say, when the gesture classifier cannot identify a user's specific gesture with a set of input data, the gesture classifier can re-acquire input data from the IMU and pulse wave sensor.

[0093] Of course, the aforementioned re-entry of data is subject to a time limit. This limit is the preset duration mentioned earlier. If the smartwatch recognizes a specific gesture from the user before the preset duration expires, the smartwatch can execute the business associated with that gesture and will no longer attempt to recognize the gesture. If the smartwatch still has not recognized the user's gesture after the preset duration expires, it will also stop attempting to recognize the gesture, i.e., the timeout ends.

[0094] In addition, during the multiple data acquisition processes mentioned above, the smartwatch can also detect whether the user has performed a touchscreen operation. When a touchscreen operation is detected, the smartwatch can terminate the gesture recognition process.

[0095] S205: The smartwatch executes the corresponding business based on the recognized specific gesture.

[0096] Once the gesture classifier recognizes a specific gesture, the smartwatch can obtain the corresponding service. Then, the smartwatch can execute the aforementioned service.

[0097] For example, the finger-rubbing gesture can be associated with payment transactions. When the gesture classifier recognizes the finger-rubbing gesture, that is, when the smartwatch detects that the user has completed the finger-rubbing gesture, in response to this gesture, the smartwatch can open an application with payment functionality and then display a payment QR code. Such an application with payment functionality is, for example,...

[0098] exist Figure 2AThe gesture interaction method shown allows the smartwatch to simultaneously acquire posture data and pulse wave data of the user performing a specific gesture. Furthermore, the smartwatch can combine these two types of data to recognize the user's gestures. As the data types of the input gesture classifier increase, the smartwatch can detect a corresponding increase in the features of the indicated gestures. This is beneficial for the smartwatch to recognize the user's gestures. Therefore, combining these two types of data to recognize user gestures can improve the robustness of gesture recognition.

[0099] In this way, smartwatches can discover more features of gestures through posture data and pulse wave data. Consequently, in scenarios such as sports activities or strong sunlight, smartwatches can recognize user gestures from more features, resulting in better and more accurate gesture recognition. This solves the problem of inaccurate gesture recognition caused by changes in the environment.

[0100] The embodiments of this application will be combined with Figure 3A The flowchart shown illustrates the gesture recognition algorithm, detailing the process by which the algorithm learns various gestures and obtains a gesture classifier.

[0101] like Figure 3A As shown, the gesture recognition algorithm first acquires posture data and pulse wave data indicating the gesture from the IMU and pulse wave sensor (S301). This posture data and pulse wave data serve as training samples for the gesture recognition algorithm to learn the user's gesture. The posture data includes X-axis acceleration data, Y-axis acceleration data, Z-axis acceleration data, and the three-axis amplitude data from the accelerometer. The X, Y, and Z axes constitute a spatial coordinate system. This application does not limit the specific directions indicated by the X, Y, and Z axes. The pulse wave data includes PPG infrared light signals and PPG green light signals.

[0102] Figure 3B An illustrative set of data from the training samples above, indicating the finger-rubbing gesture, is shown. Specifically, Figure 311 shows a set of X-axis acceleration data samples for the finger-rubbing gesture; Figure 312 shows a set of Y-axis acceleration data samples for the finger-rubbing gesture; Figure 313 shows a set of Z-axis acceleration data samples for the finger-rubbing gesture; Figure 314 shows a set of three-axis amplitude data samples for the finger-rubbing gesture; Figure 315 shows a set of PPG infrared light signal samples for the finger-rubbing gesture; and Figure 316 shows a set of PPG green light signal samples for the finger-rubbing gesture.

[0103] It is understandable that the processing and calculation of attitude data and pulse wave data in subsequent embodiments are actually the processing and calculation of various data included in attitude data and pulse wave data.

[0104] First, gesture recognition algorithms can receive a large amount of gesture data (posture data and pulse wave data). A single data point of posture data and a single data point of pulse wave data can indicate the posture of the user's gesture and the pulse wave of the hand at a certain moment. A large amount of posture data and pulse wave data constitutes the changes in the user's hand gesture and pulse wave in the time domain.

[0105] Therefore, before learning gesture posture data and pulse wave data, gesture recognition algorithms can segment gesture data over a relatively long period of time by setting a sliding window.

[0106] For example, the length of the sliding window described above can be 200 data points, and the step size can be 10 data points. That is, a sliding window can include 200 data points, and the window can move along the time axis in steps of 10 data points each, thereby updating the data within the window. It is understandable that here, the length of the data segmented by the sliding window is similar to that described above. Figure 2A The data blocks for gesture recognition described are of uniform length.

[0107] After sliding window processing, the continuous posture and pulse wave data are divided into data blocks of fixed length. This fixed length is the length of the window. Then, the gesture recognition algorithm can extract features from these data blocks.

[0108] Of course, before extracting features, the gesture recognition algorithm needs to filter the aforementioned data blocks. Filtering here refers to removing noise contained in the original data, such as motion noise and heart rate noise. Motion noise refers to the noise generated when a user performs a gesture during movement. Heart rate noise is included in pulse wave data because the contraction and relaxation of blood vessels caused by heart movement affects the change in blood vessel volume caused by gestures. Therefore, the pulse wave data collected by the pulse wave sensor contains non-gesture noise such as heart rate noise. Filtering can reduce the interference caused by this noise. Therefore, the gesture recognition algorithm needs to filter the aforementioned input data before extracting features (S302).

[0109] In this embodiment, the gesture recognition algorithm may employ an Infinite Impulse Response (IIR) Chebyshev filter to filter the attitude data acquired by the IMU; and a first-order Butterworth digital filter to filter the pulse wave data acquired by the pulse wave sensor. Not limited to the filtering algorithms mentioned above, the gesture recognition algorithm may also employ other filtering algorithms to filter the input data.

[0110] S303: The gesture recognition algorithm extracts, combines, and filters features to obtain processed input data.

[0111] After completing the filtering operations described above, the gesture recognition algorithm can begin to extract features from the input data.

[0112] In this embodiment, the gesture recognition algorithm can manually design K types of features for the input data. These features include time-domain and frequency-domain features of the input data. Examples of these features include wave trough features and vibration features. Through feature construction, extraction, and combination, the K types of features can be further expanded to obtain more features.

[0113] Then, considering the real-time performance and efficiency requirements of the algorithm, the gesture recognition algorithm can select preferred features from the aforementioned additional features (such as peak factor, waveform factor, root mean square frequency, and the degree of dispersion and concentration of the spectrum). Here, preferred features refer to features that contribute significantly to the gesture recognition algorithm's recognition of gestures. In other words, by selecting preferred features, smartwatches can effectively reduce the computational overhead of the gesture recognition algorithm without sacrificing recognition accuracy.

[0114] The data corresponding to the preferred features obtained after feature extraction, combination, and filtering can be called the processed input data.

[0115] S304: The gesture recognition algorithm uses a classification learning algorithm to train the processed input data to obtain a gesture classifier.

[0116] After determining the preferred features, the gesture recognition algorithm can train the processed input data using a classification learning algorithm to obtain a gesture judgment model, i.e., a gesture classifier. The aforementioned classification learning algorithms include, but are not limited to, Support Vector Machine (SVM), Extreme Gradient Boosting (XGBoost), and Random Forest (RF).

[0117] The smartwatch can then use a gesture classifier to identify the input data indicating a specific gesture, thereby determining that the user has performed that gesture.

[0118] In the process of the gesture classifier recognizing a specific gesture, the gesture classifier can process the acquired raw input data according to the methods shown in S301-S302. This processing includes sliding window segmentation and filtering. After the above processing, the gesture classifier obtains fixed-length input data (pose data and pulse wave data) after noise reduction. Then, referring to the method shown in S303, the gesture classifier can check whether the input data meets the requirements of each of the set K types of features. When the value of the K type of feature of the input data is within the specific threshold range of the K type of feature of a certain gesture, the gesture classifier recognizes the gesture indicated by the input data as that specific gesture. For example, when the value of the K type of feature of the input data is within the specific threshold range of the K type of feature of the finger-rubbing gesture, the gesture classifier recognizes the gesture indicated by the input data as the finger-rubbing gesture.

[0119] After recognizing a specific gesture, the smartwatch can launch the application associated with that gesture and execute the application's functions.

[0120] pass Figure 3A The method shown allows the smartwatch to learn the characteristics of various gestures through a gesture recognition algorithm, including the characteristics of the data collected by the IMU and the characteristics of the data collected by the pulse wave sensor when the gesture is completed. In this way, when a user completes a specific gesture, the smartwatch can recognize the user's gesture and, in response to the gesture, execute the service associated with that gesture.

[0121] The following will combine Figure 4 The timing diagram of the IMU and pulse wave sensor acquiring input data during the user's finger-rubbing gesture is further introduced.

[0122] like Figure 4 As shown, the horizontal axis of this time series diagram represents time. Data closer to the horizontal axis represents attitude data acquired by the IMU. Data on the attitude data represents pulse wave data acquired by the pulse wave sensor. In particular, Figure 4 The attitude and pulse wave data provided are for illustrative purposes only. Refer to the aforementioned... Figure 3B The description includes attitude data such as X-axis, Y-axis, and Z-axis acceleration data; and pulse wave data such as infrared light and green light.

[0123] Where T0 can be any point in time. T1 is the start time of the pre-action. T2 is the completion time of the pre-action, i.e., the time when the smartwatch begins to recognize the user's specific gesture. T3 is the start time of the finger-rubbing gesture. Since the end time of the pre-action is not necessarily the start time of the finger-rubbing gesture, T2 and T3 are not necessarily the same time. T4 is the end time of the finger-rubbing gesture.

[0124] T0-T1 shows the posture data in the initial state. T1-T2 shows the posture data when the user completes the preceding action. T3-T4 shows the posture data and pulse wave data when the user completes the finger-rubbing gesture.

[0125] During time intervals T0-T1, as described in S201, the IMU is active, while the pulse wave sensor may be in sleep mode for power consumption reasons. Therefore, during this period, the pulse wave sensor does not collect the user's pulse wave data. At time T1, the user begins to perform a preparatory action, which is completed at time T2. Using the posture data from T1-T2, the smartwatch can detect the user's preparatory action. At this point, the smartwatch can control the pulse wave sensor to enter active mode. Thus, at time T2, the smartwatch can acquire the user's pulse wave data collected by the pulse wave sensor.

[0126] Starting from time T0, by using a sliding window to divide the data, the smartwatch can segment the attitude data stream acquired by the IMU into a discrete set of data frames, as referenced in window 401. Specifically, the smartwatch can set a window of length S, which can slide along the time axis in steps D, thereby acquiring several sets of attitude data and pulse wave data of length S. For example... Figure 3A The S301 describes a window with a length of 100 and a step size of 10. Through this window, the smartwatch can acquire several sets of posture data and pulse wave data with a length of 100.

[0127] When the window slides to time T2, the attitude data within the window is shown in window 402.

[0128] After time T2, the smartwatch can receive not only attitude data from the IMU but also pulse wave data from the pulse wave sensor. At this point, the smartwatch can also capture the user's pulse wave data using a window of the same size, such as window 403.

[0129] Based on the description of S301, a single window may not include complete gesture posture data or pulse wave data due to its initial position. To improve the accuracy and robustness of recognition, the recognition results of T consecutive sliding windows can be combined through majority voting to output the final recognition result.

[0130] For example, T3-T4 shows the user's posture and pulse wave data when performing the finger-rubbing gesture. Therefore, when the sliding window is within the T3-T4 time period, the data included in the window can be used to identify the user's gesture. That is, the gesture recognition algorithm can calculate the features of the data in the above window to identify that the user has performed the finger-rubbing gesture.

[0131] When the input data in T3-T4 fails to recognize a specific gesture, the gesture classifier can continue to acquire data from subsequent sensors and determine whether the data corresponds to the specific gesture's posture and pulse wave data. This is because, even when a specific user gesture is not recognized, and before the preset time expires, the IMU and pulse wave sensor can continuously acquire the user's posture and pulse wave data. Figure 4 As shown, after time T4, the smartwatch can also receive the user's posture data and pulse wave data.

[0132] Therefore, when the user's gesture is not recognized, the smartwatch can also use the newly received posture data and pulse wave data, namely the posture data and pulse wave data after windows 402 and 403, to re-identify whether the user has completed a specific gesture.

[0133] Once a user's specific gesture is recognized, the smartwatch executes the business associated with that gesture.

[0134] Taking energy consumption into account, the aforementioned Figure 2A The gesture interaction method described does not set the pulse wave sensor to working state at the beginning. Instead, it activates the pulse wave sensor to work state after the IMU detects the preceding action.

[0135] Therefore, without considering energy consumption, the above gesture interaction method can directly set both the pulse wave sensor and IMU to working state in the initial state. Then, the gesture classifier can directly identify the gesture by acquiring the posture data and pulse wave data of a specific gesture. For details on the method of directly acquiring the posture data and pulse wave data of a specific gesture, please refer to... Figure 5 This will not be elaborated upon here.

[0136] By implementing the method provided in this application, a smartwatch can simultaneously acquire multiple data points related to a user performing a specific gesture, such as the posture data and pulse wave data described above. By integrating different types of data to determine the user's gesture, the smartwatch can improve its robustness in recognizing user gestures.

[0137] This is because when there is abundant data on different types of gestures, the impact of significant noise interference from one type of data on the overall data becomes smaller; that is, other types of data can reduce this interference. For example, in an outdoor walking scenario, a user's arm swings. If the user makes a gesture at this time, such as rubbing their fingers, the posture data collected by the IMU will contain significant noise interference, which comes from the arm swing. Therefore, in this case, recognizing the user's gesture solely based on posture data will result in a large error, i.e., inaccurate recognition. However, the interference from arm swing does not affect the operation of the pulse wave sensor (the pulse wave sensor is only affected by light and skin color). Therefore, the pulse wave data collected by the pulse wave sensor in this situation can be used relatively well to recognize the user's gestures.

[0138] Similarly, in strong sunlight, pulse wave data contains significant noise. In such cases, gesture posture data can reduce the impact of noise in the pulse wave data on gesture recognition.

[0139] In summary, when combining posture data and pulse wave data for recognition, pulse wave data can assist in recognition and improve the accuracy of gesture recognition when the posture data contains significant noise interference. Similarly, when the pulse wave data contains significant noise interference, posture data can assist in recognition and improve the accuracy of gesture recognition. Therefore, the gesture recognition method that combines posture data and pulse wave data has better robustness and is more adaptable to different scenarios.

[0140] When a gesture corresponds to multiple services, the smartwatch can also combine location data to determine which of the multiple services should be executed. Figure 6A A flowchart of the above method is shown.

[0141] S601: Smartwatches acquire user location data.

[0142] First, before recognizing the user's gestures, the smartwatch can acquire the user's location data. This location data can be obtained through a Wireless Fidelity (Wi-Fi) network, a Global Positioning System (GPS) network, or a cellular network to which the device is connected. The smartwatch is not limited to these methods; it can also determine the user's location through other means. This application does not impose such limitations.

[0143] After acquiring the user's location data, the smartwatch can determine whether the user is in the area where a payment will be made. For example, the location data acquired by the smartwatch can indicate that the user is in a shopping mall.

[0144] S602: The smartwatch recognizes a gesture made by the user that is associated with multiple services.

[0145] After determining the user's location, the smartwatch can detect whether the user has performed a specific gesture. The process of identifying whether the user has performed a specific gesture can be referred to the description in the foregoing embodiments, and will not be repeated here.

[0146] In this embodiment of the application, a single gesture can be associated with multiple services. For example, services associated with the finger-rubbing gesture may include: displaying a payment QR code, displaying a schedule reminder interface, etc.

[0147] S603: The smartwatch determines the services that need to be activated based on location data.

[0148] When a smartwatch recognizes a gesture associated with multiple services, it can determine which of those services should be executed based on the aforementioned location data.

[0149] For example, when a smartwatch recognizes a finger-rubbing gesture, in response to that gesture, the smartwatch can perform actions such as displaying a payment QR code or displaying a schedule reminder. In this case, referring to the location data mentioned above (the user is in a shopping mall), the smartwatch can determine to perform the action of "displaying the payment QR code".

[0150] S604: Smartwatch displays payment QR code.

[0151] Beyond the payment scenarios described above, the method of determining which of multiple services to execute based on location data can be applied to other scenarios as well. For example, in a smart home scenario, a smartwatch can remotely control the operating status of multiple electronic devices in the home. In this case, upon recognizing a gesture to turn on multiple electronic devices, the smartwatch can determine which electronic device should be turned on based on the distance between the user's location (i.e., the watch's location) and the aforementioned electronic devices.

[0152] Implementation Figure 6A The method shown allows users to control multiple services with a single gesture. This way, as the number of different services increases, users don't need to worry about adding new gestures to match the growing number of services. For smartwatches, based on recognizing user gestures, they can further determine the user's intent, thereby providing a better user experience.

[0153] Rather than simply recognizing a user's gesture as another gesture and then executing the corresponding service, users would prefer a smartwatch to obtain a more accurate recognition result through two or three attempts. This is because, when misrecognition occurs, the smartwatch, in response to the recognition result (i.e., the gesture), will execute the service associated with that gesture. If a service not intended by the user is executed, the user will have to close the application that was misrecognized and reopen the application they actually want to use. This significantly impacts the user experience.

[0154] Therefore, this application also provides a gesture recognition method. In this method, using the location data described in the above embodiments, the smartwatch can verify whether the recognized gesture is correct. The specific process of this method can be found in [reference needed]. Figure 6B .

[0155] like Figure 6B As shown, firstly, before recognizing a user's specific gesture, the smartwatch can determine that the user will make a gesture A (S611) based on the user's location data.

[0156] Specifically, in its initial state, the smartwatch can obtain the user's location data through the Wi-Fi network, GPS, or cellular network it is connected to. Using this location data, the smartwatch can determine that the user will perform a specific gesture (gesture A). This association between location data and gestures is pre-stored in the smartwatch.

[0157] refer to Figure 2A and Figure 5 According to the introduction, the pulse wave sensor of the smartwatch in its initial state can be in either sleep or working state.

[0158] Similarly, taking payment scenarios as an example, a smartwatch can determine that the user is in a commercial location such as a shopping mall based on the user's location data. In a shopping mall, the user is highly likely to use the smartwatch for payment transactions. This means the user is highly likely to make a gesture associated with a payment transaction. Therefore, based on the aforementioned location data, once the smartwatch records a specific gesture associated with a payment transaction, it can determine that the user will complete that gesture.

[0159] For example, the finger-rubbing gesture can correspond to the "display payment QR code" service. In this case, once it is determined that the user is in a commercial location such as a shopping mall, the smartwatch can determine that the user will complete the finger-rubbing gesture.

[0160] S612: The smartwatch recognizes the user's gesture B.

[0161] After completing the steps shown in S612, the smartwatch can detect whether the user has performed a specific gesture. The smartwatch can recognize that the user has made gesture A. The process of recognizing whether the user has performed a specific gesture can be referred to the description in the foregoing embodiments, and will not be repeated here.

[0162] S613: The smartwatch determines that gesture B and gesture A are the same gesture, and executes the service associated with gesture A.

[0163] After recognizing the user making gesture B, the smartwatch first determines whether gesture B is the same as gesture A. If gesture B is the same as gesture A, the smartwatch can execute the service associated with gesture A in response to that gesture.

[0164] For example, gesture A is a finger-rubbing gesture associated with the "display payment QR code" function. When gesture B is a finger-rubbing gesture, the smartwatch can confirm that the user has made the finger-rubbing gesture. At this time, the smartwatch can execute the "display payment QR code" function.

[0165] S614: When it is determined that gesture B is a different gesture from gesture A, the smartwatch can obtain new input data and re-identify the user's gesture.

[0166] When gesture B is a different gesture from gesture A, the smartwatch can first refrain from executing the service corresponding to gesture B, and instead reacquire the user's posture data and pulse wave data, and then re-identify the user's gesture.

[0167] Here, the smartwatch can re-acquire input data because: when the user is not performing the desired action, they will repeatedly complete the gestures corresponding to that action. (Reference) Figure 4 The posture and pulse wave data shown indicate that after time T4, if the smartwatch does not respond to the user's gesture from the previous process, the user will repeat the gesture two or three times. Therefore, after time T4, the gesture classifier can continue to receive user posture and pulse wave data sent by the sensors.

[0168] Optionally, when the smartwatch detects that gesture B is a different gesture from gesture A, it can also display a message on the screen reminding the user to complete the gesture again. Based on this prompt, the user will then complete the gesture again. In this way, the smartwatch can also acquire new posture and pulse wave data.

[0169] Based on the reacquired posture and pulse wave data, the smartwatch can re-recognize the user's gesture. If the re-recognition result is still gesture B, the smartwatch can execute the service corresponding to gesture B. Of course, the smartwatch can re-recognize one more time. This application does not limit the number of re-recognition attempts.

[0170] If the result of the re-identification is still gesture A, the smartwatch can perform the service corresponding to gesture A.

[0171] For example, in the payment scenario described above, if the smartwatch recognizes a clenched fist gesture (gesture B) from the user, which is not the finger-rubbing gesture (gesture A) associated with the payment service, the smartwatch can initially refrain from executing the service corresponding to the clenched fist gesture. If, however, the clenched fist gesture corresponds to the "play next song" service, the smartwatch can then delay executing the "play next song" service and instead re-identify the user's gesture. Therefore, the smartwatch can re-acquire the user's posture data and pulse wave data. Based on this data, the smartwatch can continue to identify the user's gestures.

[0172] If the recognition result is still a clenched fist gesture, the smartwatch can perform the associated function, such as "play the next song," in response to this gesture. If the recognition result is a finger-rubbing gesture, the smartwatch can perform a payment function (displaying a payment QR code) in response to this gesture.

[0173] Implementation Figure 6B The method shown allows the smartwatch to anticipate likely gestures based on the user's location. If a gesture is easily misidentified as another, the smartwatch can preemptively disable that other gesture. This way, even if the gesture is misidentified, the smartwatch won't immediately execute the corresponding service, minimizing the risk of response errors. The smartwatch can then re-identify the gesture.

[0174] Implementation Figure 6B The gesture interaction method shown reduces response errors caused by misrecognition. This provides a foundation for multiple recognitions. Consequently, when a user performs the same gesture multiple times consecutively, the smartwatch is more likely to correctly recognize the user's gesture and thus make the correct response, improving the user experience.

[0175] In other implementations, the smartwatch can also collect the sound signal of the gesture during the recognition of a specific gesture. By combining the posture data, pulse wave data, and sound signal described above, the smartwatch can more accurately recognize the user's gestures. The following will combine... Figure 7 A flowchart illustrating a gesture interaction method that incorporates sound signals.

[0176] like Figure 7As shown, firstly, the smartwatch in the initial state (S701) can recognize the user's pre-action (S702). Here, the initial state refers to the state where the IMU is active, the pulse wave sensor is in sleep (or in a low-power operation mode), and the microphone is in sleep (or in a low-power operation mode). In response to the user completing the pre-action, the smartwatch can control the pulse wave sensor and microphone to enter the active state (S703). The specific process of the smartwatch recognizing the pre-action and controlling the pulse wave sensor to enter the active state can be referred to the description in the aforementioned embodiments, and will not be repeated here.

[0177] Similar to a pulse wave sensor, once the smartwatch detects that the user has completed a preliminary action, it can send a control signal to the microphone. In response to this control signal, the microphone can then enter working mode. The microphone in working mode can then collect sound signals from the environment.

[0178] Optionally, between steps S702 and S703, the smartwatch can also acquire the user's location data to determine the user's environment. The above process can be referenced. Figure 6A or Figure 6B The method shown will not be elaborated here.

[0179] After the pulse wave sensor and microphone are activated, the smartwatch can acquire input data from the user's specific gesture (S704). Here, the input data includes the posture data, pulse wave data, and sound signal of the specific gesture. The posture data can be acquired from the IMU; the pulse wave data can be acquired from the pulse wave sensor; and the sound signal can be acquired from the microphone.

[0180] Posture data and pulse wave data have been described in detail in the foregoing embodiments and will not be repeated here. In the embodiments of this application, the sound signal includes the sound emitted by the fingers and other joints when a gesture is completed. For example, in the process of completing a finger snap gesture, the finger snap gesture itself includes sound. The smartwatch can learn from the above sounds to identify whether a specific gesture performed by the user is a finger snap gesture.

[0181] In this embodiment, the microphone sampling frequency can be 16kHz. The length of the sliding window can be 320 points, and the step size can be 160 points. In this way, the gesture recognition algorithm can receive sound signals of several sliding window sizes.

[0182] Using the aforementioned input data (posture data, pulse wave data, and sound signal), the smartwatch can recognize the user's gestures (S705). Specifically, after acquiring the aforementioned data and signals, the smartwatch can send the posture data and pulse wave data from the input data to a gesture classifier; and input the sound signal from the input data to a sound recognition module. Combining the recognition results from the gesture classifier and the sound recognition module, the smartwatch can identify which gesture the user made.

[0183] In the process of recognizing sound signals, the sound recognition module first divides the sound signal collected by the microphone into several data blocks of the preset length using a sliding window with a preset length and preset step size. The preset length and preset step size of this sliding window can differ from the length and step size used in the gesture classifier.

[0184] Then, the voice recognition module can use a one-dimensional convolutional neural network to extract frequency features, such as Mel-frequency cepstral coefficients, from the aforementioned voice signal data blocks. Based on these frequency features, the voice recognition module can obtain a recognition result. Then, combined with the recognition result from the gesture classifier, the smartwatch can confirm that the user has performed a specific gesture.

[0185] For example, when recognizing a finger snap gesture, a smartwatch can use posture data, pulse wave data, and the sound signal generated by the finger snap to identify that the user has completed the finger snapping action.

[0186] (S706) After recognizing the user's gesture through posture data, pulse wave data, and sound signals, the smartwatch can perform the service associated with that gesture. For example, if a finger snap gesture is associated with answering a phone call, the smartwatch can answer the call when the finger snap gesture is recognized.

[0187] Implementation Figure 7 The gesture interaction method shown allows the smartwatch to recognize user gestures not only through posture data and pulse wave data, but also through sound signals. This makes the smartwatch's gesture recognition more accurate.

[0188] In another gesture interaction method that incorporates sound signals, the smartwatch can operate without distinguishing between sound signals in the input data. That is, gesture data, pulse wave data, and sound signals in the input data can be identified by a classifier.

[0189] refer to Figure 3AWhen a gesture recognition algorithm learns a user's gestures, the learning samples can include not only gesture posture data and pulse wave data, but also sound signals. Thus, when recognizing gestures containing sound signals, the algorithm can determine the user's gesture based on these three types of data. This will not be elaborated further here.

[0190] In the embodiments of this application, after the preceding action is identified, the attitude data collected by the IMU can be referred to as the first attitude data; the attitude data used to identify the preceding action can be referred to as the second attitude data.

[0191] The first gesture is, for example, a finger-rubbing gesture; the first operation is, for example, a payment operation associated with the finger-rubbing gesture (i.e., displaying a user interface containing a payment QR code); the first location is, for example, a location where payment activities frequently occur, including shopping malls, convenience stores, etc. Figure 2A The window with a length of 100 and a step size of 10 described in the text can be called the first window, where the first length is 100 data points.

[0192] Taking a smartwatch as an example of a device that implements gesture interaction, the following will combine... Figure 8 The hardware diagram shown illustrates the hardware structure of a smartwatch. However, this device is not limited to smartwatches; it can also be a smart bracelet, etc.

[0193] like Figure 8 As shown, a smartwatch may include a processor 811, a memory 812, a wireless communication module 813A, a mobile communication module 813B, a power switch 814, a display screen 815, an audio module 816, an inertial motion unit 817, and a pulse wave sensor 818.

[0194] The processor 811 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0195] In this embodiment, the application processor (AP) can support the smartwatch in running various applications, such as those involving payment services. app, Applications, and other applications. The graphics processing unit (GPU) works with the display screen 815 to display images and user interfaces, such as displaying payment QR codes, etc.

[0196] The controller can serve as the central nervous system and command center of the smartwatch. Based on instruction opcodes and timing signals, the controller generates operation control signals to control instruction fetching and execution. In this embodiment, the smartwatch requires controller support for processes such as recognizing user gestures, coordinating the operation of various sensors, controlling the logical transitions between various hardware and software modules, and controlling the working state of the application.

[0197] The processor 811 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 811 is a cache memory. This memory can store instructions or data that the processor 811 has just used or that are used repeatedly. If the processor 811 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 811, and thus improves the efficiency of the system.

[0198] The memory 812 is coupled to the processor 811 and is used to store various software programs and / or multiple sets of instructions. The memory 812 can be used to store computer-executable program code, which includes instructions. The processor 811 executes various functional applications and data processing of the smartwatch by running the instructions stored in the memory 812. The memory 812 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the smartwatch (such as audio data, image data to be displayed, etc.). Furthermore, the memory 812 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0199] In this embodiment, the software program and / or multiple sets of instructions that enable the smartwatch to learn and recognize specific user gestures and invoke specific services based on those gestures can be stored in the memory 812. The aforementioned software program and / or multiple sets of instructions also include various applications (APPs) installed on the smartwatch, such as... The memory 812 also stores user data corresponding to the aforementioned applications.

[0200] The wireless communication module 813A can provide solutions for wireless communication applications in smartwatches, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 813A can be one or more devices integrating at least one communication processing module. The wireless communication module 813A receives electromagnetic waves via antenna 1, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 811. The wireless communication module 813A can also receive signals to be transmitted from processor 811, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 1.

[0201] In some embodiments, antenna 2 of the smartwatch is coupled to mobile communication module 813B, and antenna 1 is coupled to wireless communication module 813A, enabling the smartwatch to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0202] In the gesture interaction method for obtaining location data provided in this application embodiment, the smartwatch can obtain its own location data through wireless communication module 813A, mobile communication module 813B, and other wireless communication technologies. These other wireless communication technologies include, for example, GNSS as described above. When using a WLAN network, the smartwatch can obtain its current location through wireless communication module 813A. When using mobile cellular data communication, the smartwatch can obtain its current location through mobile communication module 813B. The smartwatch can also obtain its current location through other positioning systems such as BDS and GPS.

[0203] The power switch 814 can be used to control the power supply to the smartwatch. In some embodiments, the power switch 814 can be used to control the power supply to the smartwatch from an external power source.

[0204] Display screen 815 can be used to display images, videos, etc. Display screen 815 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0205] In this embodiment, the display screen 815 provides the smartwatch with the function of displaying the user interface of payment applications and payment QR codes. Furthermore, the display screen 815 and the touch sensor 815A together form a touchscreen, also known as a "touchscreen". In touch-interaction control methods, in response to user touch operations, such as clicking or swiping, the touchscreen can correspondingly switch the user interface, display or close the user interface of the application process, etc.

[0206] The audio module 816 can be used to convert digital audio signals into analog audio signals for output, and can also be used to convert analog audio input into digital audio signals. The audio module 816 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 816 can be located in the processor 811, or some functional modules of the audio module 816 can be located in the processor 811. The audio module 816 can transmit audio signals to the wireless communication module 813 via a bus interface (e.g., a UART interface, etc.) to enable the playback of audio signals through a Bluetooth speaker.

[0207] The smartwatch may also include a microphone 816A, also known as a "microphone" or "voice transducer," used to convert sound signals into electrical signals. When a voice control command is given, the user can speak, inputting the sound signal into the microphone 816A. In this embodiment, the microphone 816A can acquire the sound signal of a specific gesture when the user completes that gesture. Furthermore, the gesture recognition algorithm can recognize the gesture by learning its sound signal.

[0208] An inertial motion unit 817 (IMU) can be used to measure an object's three-axis attitude angles (or angular rates) and acceleration. The inertial motion unit 817 may include an accelerometer sensor 817A and a gyroscope sensor 817B.

[0209] The accelerometer 817A comprises a mass block, a damper, an elastic element, a sensing element, and an adaptation circuit. During acceleration, the accelerometer 817A obtains the acceleration value by measuring the inertial force acting on the mass block and applying Newton's second law. The accelerometer 817A can detect the magnitude of acceleration in various directions (typically three axes) of electronic devices. When the electronic device is stationary, it can detect the magnitude and direction of gravity.

[0210] The gyroscope sensor 817B can determine the angular velocity of an object around three axes (i.e., the x, y, and z axes), and therefore, it is commonly used to determine the motion posture of an object. In some embodiments, the gyroscope sensor 817B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 817B detects the angle of vibration of the electronic device, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the vibration of the electronic device through reverse movement, thus achieving image stabilization. The gyroscope sensor 817B can also be used in navigation and motion-sensing gaming scenarios.

[0211] In this embodiment, the accelerometer 817A and / or the gyroscope 817B can be used to detect whether the user's hand is in a horizontal position (horizontal detection). Horizontal detection can be used by the smartwatch to determine whether the user will make a specific gesture. During the process of the user completing a specific gesture, the accelerometer 817A and / or the gyroscope 817B can also collect the user's gesture posture data, thereby assisting the smartwatch in recognizing the user's gesture from the posture data.

[0212] The pulse wave sensor 818 includes a light-emitting diode (LED) and a photodetector, such as a photodiode. The LED can be a monochromatic LED, an infrared LED, etc. Monochromatic LEDs include, for example, yellow LEDs, green LEDs, etc. The pulse wave sensor 818 acquires the user's pulse wave signal through photoplethysmography (PPG). PPG refers to the method by which a photodetector acquires the human pulse wave signal by detecting the energy of reflected light after it has passed through tissues such as the skin, muscles, and blood vessels.

[0213] In this embodiment, the pulse wave sensor 818 can enable the smartwatch to acquire the pulse wave signal when the user makes a specific gesture, thereby assisting the smartwatch in recognizing the user's gesture from the user's pulse wave signal.

[0214] This invention primarily provides a gesture interaction method. This method can be applied to electronic devices such as smartwatches and fitness trackers where touch operation is inconvenient. By implementing this method, smartwatches and other electronic devices can more accurately recognize user gestures as the user performs specific actions. In particular, smartwatches and other electronic devices can accurately recognize user gestures even when the user's environment changes significantly.

[0215] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (GTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, GTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0216] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0217] As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the listed items. As used in the above embodiments, depending on the context, the term "when" can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)."

[0218] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0219] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A gesture interaction method, applied to an electronic device, the method comprising: receiving a gesture input from a user; determining a gesture type of the gesture input; and performing a corresponding operation according to the gesture type. The method comprises: collecting first gesture data through an inertial motion unit; collecting pulse wave data through a pulse wave sensor while collecting the first gesture data; obtaining position data of a user; determining that the user makes a first gesture through the first gesture data and the pulse wave data; confirming a first operation in a plurality of operations corresponding to the first gesture according to the position data, and executing the first operation.

2. The method of claim 1, wherein, Before collecting the first gesture data through the inertial motion unit, the method further comprises: collecting second gesture data; determining that the user makes a pre-action according to the second gesture data; in response to determining that the user makes the pre-action, turning on the pulse wave sensor.

3. The method of claim 2, wherein, The method further comprises: in response to determining that the user makes the pre-action, lighting up a screen.

4. The method of claim 1, wherein, Determining that the user makes a first gesture through the first gesture data and the pulse wave data specifically comprises: using a first window to obtain a first gesture data block and a first pulse wave data block from the first gesture data and the pulse wave data, the window having a first length; filtering the first gesture data block and the first pulse wave data block to obtain a second gesture data block and a second pulse wave data block; calculating a first feature of the second gesture data block and the second pulse wave data block; using the feature to determine that the user makes a first gesture.

5. The method according to any one of claims 1-3, characterized in that, The method further comprises: when confirming that the user is at a first position according to the position data, but not identifying a first gesture corresponding to the first position, displaying a first interface, the first interface being used to prompt the user to repeat the previous gesture.

6. The method of claim 1, wherein executing the first operation specifically comprises: displaying a user interface containing a payment QR code.

7. The method of claim 2, wherein, The method further comprises: in response to determining that the user makes the pre-action, turning on a microphone; collecting a sound signal through the microphone; Determining that the user makes a first gesture through the first gesture data and the pulse wave data further comprises: judging whether the user makes a first gesture through the first gesture data, the pulse wave data and the sound signal.

8. The method of claim 7, wherein, Judging whether the user makes a first gesture through the first gesture data, the pulse wave data and the sound signal specifically comprises: judging whether the user makes a first gesture through a first feature of the first gesture data, the pulse wave data and a frequency feature of the sound signal.

9. The method according to claim 4 or 8, characterized in that, The first feature comprises: a trough feature, a vibration feature, a peak factor, a waveform factor, a root mean square frequency, and two or more features representing the dispersion and concentration of a frequency spectrum.

10. The method of claim 1, wherein the gesture data comprises: one or more of X-axis acceleration data, Y-axis acceleration data, Z-axis acceleration data, and three-axis acceleration amplitude data; the pulse wave data comprises: one or more of infrared light data and green light data.

11. An electronic device, comprising: comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories configured to store computer program code comprising computer instructions that, when executed by the one or more processors, cause performance of the method of any of claims 1-10.

12. A chip system for use in an electronic device, the chip system comprising one or more processors configured to invoke computer instructions to cause performance of the method of any of claims 1-10.

13. A computer program product comprising instructions that, when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-10.

14. A computer-readable storage medium comprising instructions, wherein: when the instructions are executed on an electronic device, cause performance of the method of any of claims 1-10.

Images