Gesture recognition method and electronic equipment

Abstract

The invention provides a gesture recognition method and electronic equipment, and relates to the technical field of man-machine interaction. The gesture recognition method comprises the steps that in response to user operation, gesture sensing information is obtained, and the user operation is used for triggering gesture control; acquiring a first gesture according to the gesture sensing information; information used for describing the first gesture is reported to the first application, so that the first application executes a first operation corresponding to the first gesture, and the first application is an application subscribing to the first gesture. Compared with the method that the applications perform gesture recognition by themselves, the first system service in the electronic equipment performs gesture recognition in a unified manner, so that collision of multiple applications in the gesture recognition process is avoided, the number of times of repeated gesture recognition in the electronic equipment is reduced, and the calculation power waste of the electronic equipment is reduced.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a gesture recognition method and electronic device. Background Technology

[0002] Functions supported by applications (such as native system applications and third-party applications) can be triggered by gestures. The gesture recognition process is used to identify the gestures corresponding to user operations.

[0003] Taking a third-party application as an example, during gesture recognition, the electronic device acquires gesture sensing information and sends it to the running third-party application. The gesture recognition logic within the third-party application predicts the corresponding gesture based on the gesture sensing information and executes the control event corresponding to that gesture.

[0004] However, the gesture recognition processes of applications may conflict with each other, wasting the computing power of electronic devices. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a gesture recognition method and an electronic device. The technical solution provided by this application unifies gesture recognition through a first system service within the electronic device. By processing gesture sensing information, a first gesture is obtained, and applications subscribed to the first gesture are notified to execute the operation corresponding to the first gesture. By having the system service manage the gesture recognition process within the electronic device, redundancy caused by multiple applications performing gesture recognition independently can be minimized, thus helping to conserve the computing power of the electronic device.

[0006] Firstly, the gesture recognition method can be applied to electronic devices. Here, we will take the execution of the gesture recognition method provided in this embodiment by a first system service in an electronic device as an example. The gesture recognition method includes: in response to a user operation, acquiring gesture sensing information, whereby the user operation triggers gesture control; acquiring a first gesture based on the gesture sensing information; and reporting information describing the first gesture to a first application, so that the first application performs a first operation corresponding to the first gesture, wherein the first application is an application that subscribes to the first gesture.

[0007] In this embodiment, arbitrary gesture recognition is handled by a unified system service. Compared to applications performing gesture recognition themselves, this helps avoid conflicts arising from multiple applications simultaneously performing gesture recognition on the same target gesture. For example, related technologies may result in multiple applications performing gesture recognition based on the same gesture sensing information, causing the electronic device to perform multiple gesture recognitions for the same user operation. However, according to common usage habits, a user operation is performed on one application. Therefore, unnecessary gesture recognition processes exist in multiple gesture recognitions, meaning there is redundancy between the gesture recognition performed by different applications.

[0008] In this embodiment, gesture recognition is uniformly completed by a fixed system service, and there is no redundancy in gesture recognition for the same gesture. This solves the problem that conflicts are likely to occur when applications perform gesture recognition independently, helps to reduce the repeated gesture recognition process, thereby reducing the computing power occupation of electronic devices by gesture recognition and reducing the lag of electronic devices.

[0009] On the other hand, after recognizing a gesture based on gesture sensing information, the electronic device executes the corresponding operation through the application that has subscribed to that gesture. By separating the subject of gesture recognition from the subject that executes the corresponding operation of gesture control, the user operation used to trigger gesture control has minimal impact on other applications in the electronic device that have not subscribed to that gesture, thus helping to reduce interference from gesture recognition to other unrelated applications in the electronic device.

[0010] In one possible implementation, after obtaining the first gesture based on at least one type of gesture factor, the method further includes: obtaining updated gesture sensing information in response to a user operation; obtaining motion parameters of the first gesture based on the updated gesture sensing information; and reporting the motion parameters of the first gesture to a first application so that the first application continues to perform the first operation.

[0011] Thus, after recognizing the corresponding gesture, the motion parameters of the first gesture are determined through gesture sensing information, allowing the application to adjust the implementation effect of the first operation based on the motion parameters. This method helps to achieve the desired effect of the first operation, flexibly changing with changes in user operation and facilitating more precise gesture control.

[0012] In one possible implementation, the motion parameters of the first gesture are related to the gesture control type to which the first gesture belongs; if the first gesture belongs to the swipe control type, the motion parameters of the first gesture are used to indicate changes in the swipe distance; or, if the first gesture belongs to the zoom control type, the motion parameters of the first gesture are used to indicate changes in the zoom ratio; or, if the first gesture belongs to the rotation control type, the motion parameters of the first gesture are used to indicate changes in the rotation angle. The motion parameters are used to control the effect of the first operation during the continuous operation of the user.

[0013] In one possible implementation, before acquiring gesture sensing information in response to a user action, the method further includes: acquiring subscription information sent by a first application, the subscription information indicating at least one gesture supported by the first application, the at least one gesture including a first gesture.

[0014] In this way, the application pre-notifies the first system service of at least one supported gesture via subscription information, enabling the electronic device to respond normally to gesture control after the functional module takes over gesture recognition. Upon recognizing a gesture subscribed to by the application, the functional module notifies the application to complete the corresponding control event. This achieves the effect of isolating gesture recognition and gesture control through the functional module, helping to reduce interference from the gesture recognition process to the application.

[0015] In one possible implementation, obtaining a first gesture based on gesture sensing information includes: obtaining at least one type of gesture factors based on the gesture sensing information, wherein each type of gesture factor is used to describe the gesture parameters of the corresponding dimension; and obtaining the first gesture based on the at least one type of gesture factors.

[0016] Thus, gesture factors can be used to distinguish gestures of different control types. In the embodiment corresponding to this implementation, the gesture factors used to distinguish each gesture control type overlap, thereby integrating the gesture recognition process of different gesture control types into a single gesture recognition logic, enabling the gesture recognition logic to adapt to the gesture recognition needs of various gesture control types.

[0017] Furthermore, during gesture recognition, at least one type of gesture factor is first acquired based on gesture sensing information, and the first gesture is determined using this at least one type of gesture factor. The acquired gesture factors include those shared by multiple gesture control types. In other words, multiple gesture control types share at least one type of gesture factor in the gesture recognition method. Compared to designing separate gesture recognition logic for each type of gesture, or for each gesture alone, this embodiment reuses gesture factors during the gesture recognition process, which helps reduce the computational load on the electronic device during gesture recognition, thereby improving the efficiency of gesture recognition.

[0018] Moreover, this method has good compatibility with recognizing different gestures. After introducing a new gesture, this method supports determining the gesture control type to which the new gesture belongs and adding the new gesture to that gesture control type, thus expanding the gesture recognition logic and helping to improve the adaptability of the gesture recognition logic to different use scenarios.

[0019] In one possible implementation, obtaining a first gesture based on at least one type of gesture factor includes: obtaining a first gesture attribute based on at least one type of gesture factor, the first gesture attribute being used to characterize a first gesture control type to which the first gesture belongs; obtaining at least one second gesture attribute based on at least one type of gesture factor, the second gesture attribute being used to characterize attribute parameters associated with the first gesture control type; and obtaining the first gesture based on the first gesture attribute and at least one second gesture attribute.

[0020] Thus, by determining the gesture control type and specific gesture parameters through at least one gesture factor, the first gesture can be identified. In the embodiment corresponding to this implementation, the factors determining the first gesture are broken down, and the determination processes for each gesture attribute obtained from the breakdown are independent of each other. After applying a new gesture, this gesture recognition method only needs to adjust and expand the method branches for determining some gesture attributes to be compatible with the new gesture, thereby helping to improve the gesture recognition method's compatibility with more gestures and enhance its universality.

[0021] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities, where the competing priorities are used to characterize the order in which the multiple types of gesture factors participate in gesture recognition; obtaining a first type based on at least one type of gesture factor includes: obtaining a first gesture factor from the multiple types of gesture factors, where the first gesture factor is the gesture factor with the highest competing priority among the multiple types of gesture factors; and obtaining a first gesture attribute based on the gesture control type corresponding to the first gesture factor if the first gesture factor satisfies the corresponding judgment condition.

[0022] Thus, by using the competition priority indicator to guide the order in which gesture factors are used in determining the gesture control type, the orderly progress of the gesture control type determination process is ensured. In this embodiment, the branch of the gesture control type is determined by whether the gesture factor meets its corresponding judgment condition. Gesture factors with higher competition priority are used first to obtain the gesture control type to which the first gesture belongs. If the gesture factor with higher competition priority meets its corresponding judgment condition, other gesture factors with lower competition priority do not need to be used, which helps to simplify the logic of determining the gesture control type.

[0023] Furthermore, selecting gesture factors based on competition priority allows the gesture recognition process to acquire gesture factors in stages. For example, if a gesture factor with higher competition priority does not meet the corresponding judgment condition, a gesture factor with lower competition priority is then selected, reducing the total number of gesture factors that need to be acquired and thus improving the efficiency of gesture recognition.

[0024] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities, where the competing priorities characterize the order in which the multiple types of gesture factors participate in gesture recognition. Obtaining a first gesture attribute based on at least one type of gesture factor includes: obtaining a first gesture factor from the multiple types of gesture factors, where the first gesture factor is the gesture factor with the highest competing priority among the multiple types of gesture factors; if the first gesture factor does not meet its corresponding judgment condition, obtaining a second gesture factor from the multiple types of gesture factors, where the second gesture factor is the gesture factor with the highest competing priority among the gesture factors that did not participate in gesture recognition among the multiple types of gesture factors; and if the second gesture factor meets its corresponding judgment condition, determining the first gesture attribute based on the gesture control type corresponding to the second gesture factor.

[0025] In this way, each gesture factor is ensured to participate in gesture recognition only once, and the same gesture factor will not be judged repeatedly. This avoids the repeated use of gesture factors, which could lead to the inability to determine the gesture control type, and thus ensures the stability of the gesture recognition logic.

[0026] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities; based on at least one type of gesture factor, a first gesture attribute is obtained, including: obtaining a target gesture factor, which is the gesture factor with the highest competing priority among the multiple types of gesture factors; determining whether the target gesture factor meets the corresponding judgment condition; if the target gesture factor meets the corresponding judgment condition, determining the first gesture attribute based on the gesture control type corresponding to the target gesture factor, wherein the first gesture attribute is used to characterize the gesture control type corresponding to the target gesture factor; if the target gesture factor does not meet the corresponding judgment condition, updating the target gesture factor to the gesture factor with the highest competing priority among the gesture factors that have not participated in gesture recognition among the multiple types of gesture factors, and repeatedly performing the operation of determining whether the target gesture factor meets the corresponding judgment condition until the first gesture control type is determined, or all gesture factors among the multiple types of gesture factors have participated in gesture recognition.

[0027] Thus, based on the competition priority among multiple gesture factors, each gesture factor is verified to ensure it meets its corresponding judgment condition. If the target gesture factor meets the judgment condition, the first gesture attribute can be determined. This method helps reduce the number of checks required to verify whether a gesture factor meets the judgment condition, thereby improving the efficiency of determining the first gesture attribute.

[0028] In one possible implementation, the multiple gesture factors include a direction factor and a center-of-gravity factor; the competition priority of the direction factor is higher than that of the center-of-gravity factor; the gesture control type corresponding to the direction factor is a sliding control type, and the gesture control type corresponding to the center-of-gravity factor is a rotation control type; or, the gesture control type corresponding to the direction factor is a sliding control type, and the gesture control type corresponding to the center-of-gravity factor is a scaling control type; wherein, the direction factor is used to characterize the movement direction of at least one gesture recognition point indicated by the gesture sensing information, and the center-of-gravity factor is used to characterize the distance between at least one gesture recognition point and the geometric center of gravity when the total number of gesture recognition points is greater than 2, and the geometric center of gravity is the center of gravity of the geometric figure with at least one gesture recognition point as its vertex. Optionally, the determination condition corresponding to the direction factor includes: there are multiple gesture recognition points greater than or equal to a first number, and the direction factor corresponding to each of the multiple gesture recognition points is the same; and / or, the determination condition corresponding to the center-of-gravity factor includes: there are a second number of gesture recognition points, and for each of the second number of gesture recognition points, the change in the center-of-gravity factor corresponding to the gesture recognition point is greater than or equal to a first distance threshold.

[0029] Therefore, since obtaining the orientation factor is simpler than obtaining the center of gravity factor, setting the competition priority of the orientation factor higher than that of the center of gravity factor helps increase the usage rate of the orientation factor in determining the gesture control type and reduce the usage rate of the center of gravity factor (e.g., for gestures belonging to the swipe control type, where the orientation factor is sufficient to determine that the gesture belongs to the swipe control type, the electronic device does not need to obtain the center of gravity factor separately), thus helping to reduce the computational burden on the electronic device.

[0030] In one possible implementation, obtaining a first gesture attribute based on at least one type of gesture factor further includes: obtaining a third gesture factor from at least one type of gesture factor, the third gesture factor being used to verify the validity of the user operation; and if the third gesture factor satisfies the validity condition, obtaining a first gesture attribute to characterize the gesture control type corresponding to the selected gesture factor, the selected gesture factor being either the first gesture factor or the second gesture factor.

[0031] The validity of user operations is verified by a third gesture factor, allowing gesture recognition to proceed only when the user operation involves a certain displacement or speed. This third gesture factor filters out interference from invalid user operations with small movements, thus improving the accuracy of the gesture recognition process. Furthermore, this method helps prevent electronic devices from misinterpreting user touches as user operations, thereby preventing situations where a first system service notifies a first application to perform a first operation when it is unnecessary, thus avoiding abnormal operation of the electronic device.

[0032] In one possible implementation, the third gesture factor includes at least one of the following: a distance factor, used to characterize the movement distance of at least one gesture recognition point; and a speed factor, used to characterize the movement speed of at least one gesture recognition point.

[0033] Optionally, the validity conditions for the speed factor include: the speed factor corresponding to at least one gesture factor is greater than or equal to a speed threshold, and / or, the validity conditions for the distance factor include: the distance factor corresponding to at least one gesture factor is greater than or equal to a second distance threshold.

[0034] Speed ​​and distance factors can accurately reflect the movement of a user during a certain period of operation. Therefore, setting speed and / or distance factors as the third gesture factor helps to reflect whether a valid user operation has occurred in a timely and accurate manner.

[0035] In one possible implementation, obtaining the first gesture based on the first gesture attribute and at least one second gesture attribute includes: during the continuous process of user operation, continuously obtaining n predicted gestures based on the corresponding first gesture attribute and at least one second gesture attribute, where n is a positive integer; and obtaining the first gesture based on the n predicted gestures if the n predicted gestures satisfy the accuracy condition.

[0036] If the gesture to be recognized is repeatedly identified as the same gesture, then designating that gesture as the first gesture helps improve the accuracy of gesture recognition.

[0037] In one possible implementation, at least one second gesture attribute corresponding to the first gesture control type includes at least one of the following: the total number of gesture recognition points indicated by the gesture sensing information, and motion attributes; the first gesture attribute includes at least one of the following: swipe control type, scaling control type, and rotation control type; the motion attribute corresponding to the swipe control type includes a swipe direction attribute, the motion attribute corresponding to the scaling control type includes a scaling trend attribute, and the motion attribute corresponding to the rotation control type includes a rotation direction attribute; wherein, the total number of gesture recognition points takes at least one of the following values: single finger, two fingers, three fingers, four fingers, and five fingers; the swipe direction attribute is at least one of the following: up, down, left, right, upper left, upper right, lower left, and lower right; the scaling trend attribute is at least one of the following: open and pinch; and the rotation direction attribute is at least one of the following: clockwise and counterclockwise.

[0038] In this way, a variety of gestures can be described by the first gesture attribute and at least one second gesture attribute, which helps to improve the gesture recognition method's ability to recognize different gestures.

[0039] In one possible implementation, the information used to describe the first gesture includes first information and second information, wherein the first information is determined based on the first gesture attributes and motion attributes, and the second information is the total number of gesture recognition points.

[0040] In this way, combining the motion attributes from the first gesture attribute and the second gesture attribute to obtain the first information helps to reduce the types of parameters that need to be transmitted to the first application.

[0041] In one possible implementation, the competition priority of the gesture factor is positively correlated with the total number of gesture control types associated with the gesture factor, and the competition priority of the direction factor is higher than that of the center of gravity factor; wherein, the gesture control type corresponding to the direction factor is the sliding control type, and the gesture control type corresponding to the center of gravity factor is the rotation control type.

[0042] Thus, controlling the direction of each decision branch in the gesture control process by setting the number of gesture control types associated with each gesture control type helps improve the systematic nature of the gesture control type acquisition process. Acquiring the direction factor is simpler than acquiring the center of gravity factor. By setting the competition priority of the direction factor higher than that of the center of gravity factor, the usage rate of the direction factor in determining the gesture control type is increased, while the usage rate of the center of gravity factor is reduced (e.g., for a gesture belonging to the swipe control type, the direction factor is sufficient to determine that the gesture belongs to the swipe control type, and the electronic device does not need to acquire the center of gravity factor separately), thus helping to reduce the computational burden on the electronic device.

[0043] In one possible implementation, at least one type of gesture factor includes at least one of the following: a quantity factor, used to characterize the total number of at least one gesture recognition point indicated by gesture sensing information; a direction factor, used to characterize the movement direction of at least one gesture recognition point; a distance factor, used to characterize the movement distance of at least one gesture recognition point; a center of gravity factor, used to characterize the distance between at least one gesture recognition point and the geometric center of gravity when the total number of gesture recognition points is greater than 2, the geometric center of gravity being the center of gravity of a geometric figure with at least one gesture recognition point as its vertex; and a speed factor, used to characterize the movement speed of at least one gesture recognition point.

[0044] In this way, the above-mentioned gesture factors can describe the movement trend of gestures from different aspects. Gestures of the same gesture control type show a certain consistency in gesture factors, while gestures of different gesture control types differ in the same gesture factor. By setting multiple types of gesture factors, it is helpful to determine the correct gesture control type based on the gesture factors, and then recognize the correct gesture, which helps to improve the accuracy of gesture recognition.

[0045] In one possible implementation, the gesture sensing information includes position information of at least one gesture recognition point; based on the gesture sensing information, at least one type of gesture factor is obtained, including: if the at least one type of gesture factor includes a centroid factor, determining the position information of the geometric centroid based on the position information of at least one gesture recognition point; and determining the centroid factor corresponding to each gesture recognition point among the at least one gesture recognition point based on the position information of the gesture recognition point and the position information of the geometric centroid.

[0046] Thus, by using the centroid factor to reflect the distance between the gesture recognition point and the geometric centroid, the relative scaling trend of the gesture can be determined in subsequent steps. In this embodiment, the geometric centroid is used as a reference point to calculate the distance between each gesture recognition point and the geometric centroid. Compared to determining the distance between any two gesture recognition points at once, using the centroid position as a reference point in this embodiment helps reduce the computational load in the process of obtaining the centroid factor by the electronic device and simplifies the gesture recognition logic corresponding to the scaling control type of gesture. At the same time, it also helps to accurately characterize the scaling trend of the gesture.

[0047] In one possible implementation, the gesture sensing information includes the position information of at least one gesture recognition point; based on the gesture sensing information, at least one type of gesture factor is obtained, including: if the at least one type of gesture factor includes a relative factor, for each relative factor, based on the position information and position reference information of at least one gesture recognition point, determining the relative factor corresponding to each gesture recognition point among the at least one gesture recognition point, wherein the relative factor is at least one of a direction factor, a speed factor, and a distance factor.

[0048] Thus, by using the location information and location reference information of the gesture recognition point to determine the direction factor, velocity factor and distance factor, it is helpful to determine the direction factor, velocity factor and distance factor of the electronic device in a short period of time, thereby improving the timeliness of the motion trend represented by the direction factor, velocity factor and distance factor.

[0049] In one possible implementation, the first system service resides in the system framework layer of the electronic device.

[0050] By placing the gesture recognition function at the system framework layer, the recognition results are not affected by the types of applications installed on the electronic device.

[0051] In a second aspect, this application provides an electronic device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory to cause the electronic device to perform the methods in the first aspect and any possible implementation thereof.

[0052] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform: in response to a user operation, acquire updated gesture sensing information; based on the updated gesture sensing information, acquire motion parameters of a first gesture; and report the motion parameters of the first gesture to a first application so that the first application continues to perform the first operation.

[0053] In one possible implementation, the motion parameters of the first gesture are related to the gesture control type to which the first gesture belongs; if the first gesture belongs to the sliding control type, the motion parameters of the first gesture are used to indicate the change in sliding distance; if the first gesture belongs to the scaling control type, the motion parameters of the first gesture are used to indicate the change in scaling ratio; if the first gesture belongs to the rotation control type, the motion parameters of the first gesture are used to indicate the change in rotation angle.

[0054] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform: obtaining subscription information for a first application, the subscription information indicating at least one gesture supported by the first application, the at least one gesture including the first gesture.

[0055] In one possible implementation, the electronic device acquires a first gesture based on gesture sensing information, including: acquiring at least one type of gesture factors based on the gesture sensing information, wherein the at least one type of gesture factors is used to describe gesture parameters of one dimension; and acquiring the first gesture based on the at least one type of gesture factors.

[0056] In one possible implementation, the electronic device acquires a first gesture based on at least one type of gesture factors, including: acquiring a first gesture attribute based on at least one type of gesture factors, the first gesture attribute being used to characterize a first gesture control type to which the first gesture belongs; acquiring at least one second gesture attribute based on at least one type of gesture factors, the at least one second gesture attribute being used to characterize at least one attribute parameter corresponding to the first gesture control type; and acquiring the first gesture based on the first gesture attribute and the at least one second gesture attribute.

[0057] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities, where the competing priorities are used to characterize the order in which the multiple types of gesture factors participate in gesture recognition; the electronic device obtains a first gesture attribute based on at least one type of gesture factor, including: obtaining a first gesture factor from the multiple types of gesture factors, wherein the first gesture factor is the gesture factor with the highest competing priority among the multiple types of gesture factors; and determining the first gesture attribute based on the gesture control type corresponding to the first gesture factor when the first gesture factor satisfies the corresponding determination condition.

[0058] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities, where the competing priorities characterize the order in which the multiple types of gesture factors participate in gesture recognition. The electronic device obtains a first gesture attribute based on at least one type of gesture factor, including: obtaining a first gesture factor from the multiple types of gesture factors, wherein the first gesture factor is the gesture factor with the highest competing priority among the multiple types of gesture factors; if the first gesture factor does not meet the corresponding judgment condition, obtaining a second gesture factor from the multiple types of gesture factors, wherein the second gesture factor is the gesture factor with the highest competing priority among the gesture factors that did not participate in gesture recognition among the multiple types of gesture factors; and if the second gesture factor meets the corresponding judgment condition, determining the first gesture attribute based on the gesture control type corresponding to the second gesture factor.

[0059] In one possible implementation, at least one type of gesture factor includes multiple types of gesture factors with competing priorities; based on at least one type of gesture factor, a first gesture attribute is obtained, including: obtaining a target gesture factor, which is the gesture factor with the highest competing priority among the multiple types of gesture factors; determining whether the target gesture factor meets the corresponding judgment condition; if the target gesture factor meets the corresponding judgment condition, determining the first gesture attribute based on the gesture control type corresponding to the target gesture factor, wherein the first gesture attribute is used to characterize the gesture control type corresponding to the target gesture factor; if the target gesture factor does not meet the corresponding judgment condition, updating the target gesture factor to the gesture factor with the highest competing priority among the gesture factors that have not participated in gesture recognition among the multiple types of gesture factors, and repeatedly performing the operation of determining whether the target gesture factor meets the corresponding judgment condition until the first gesture control type is determined, or all gesture factors among the multiple types of gesture factors have participated in gesture recognition.

[0060] In one possible implementation, the multiple gesture factors include a direction factor and a center-of-gravity factor; the competition priority of the direction factor is higher than that of the center-of-gravity factor; the gesture control type corresponding to the direction factor is a sliding control type, and the gesture control type corresponding to the center-of-gravity factor is a rotation control type; or, the gesture control type corresponding to the direction factor is a sliding control type, and the gesture control type corresponding to the center-of-gravity factor is a scaling control type; wherein, the direction factor is used to characterize the movement direction of at least one gesture recognition point indicated by the gesture sensing information, and the center-of-gravity factor is used to characterize the distance between at least one gesture recognition point and the geometric center of gravity when the total number of gesture recognition points is greater than 2, and the geometric center of gravity is the center of gravity of the geometric figure with at least one gesture recognition point as its vertex. Optionally, the determination condition corresponding to the direction factor includes: there are multiple gesture recognition points greater than or equal to a first number, and the direction factor corresponding to each of the multiple gesture recognition points is the same; and / or, the determination condition corresponding to the center-of-gravity factor includes: there are a second number of gesture recognition points, and for each of the second number of gesture recognition points, the change in the center-of-gravity factor corresponding to the gesture recognition point is greater than or equal to a first distance threshold.

[0061] In some possible implementations, the electronic device obtains a first gesture attribute based on at least one type of gesture factor, and further includes: obtaining a third gesture factor from at least one type of gesture factor, the third gesture factor being used to verify the validity of the user operation; and if the third gesture factor satisfies the validity condition, obtaining a first gesture attribute to characterize the gesture control type corresponding to the selected gesture factor, the selected gesture factor being either the first gesture factor or the second gesture factor.

[0062] In one possible implementation, the third gesture factor includes at least one of the following: a distance factor, used to characterize the movement distance of the at least one gesture recognition point; and a speed factor, used to characterize the movement speed of the at least one gesture recognition point. Optionally, the validity condition corresponding to the speed factor includes: the speed factor corresponding to each of the at least one gesture factor is greater than or equal to a speed threshold, and / or, the validity condition corresponding to the distance factor includes: the distance factor corresponding to each of the at least one gesture factor is greater than or equal to a second distance threshold.

[0063] In one possible implementation, obtaining the first gesture based on a first gesture attribute and at least one second gesture attribute includes: during the continuous process of user operation, continuously obtaining n predicted gestures based on the corresponding first gesture attribute and at least one second gesture attribute, where n is a positive integer; and obtaining the first gesture based on the n predicted gestures if the n predicted gestures satisfy the accuracy condition.

[0064] In one possible implementation, at least one second gesture attribute corresponding to the first gesture control type includes at least one of the following: the total number of gesture recognition points indicated by the gesture sensing information, and motion attributes; the first gesture attribute includes at least one of the following: swipe control type, scaling control type, and rotation control type; the motion attribute corresponding to the swipe control type includes a swipe direction attribute, the motion attribute corresponding to the scaling control type includes a scaling trend attribute, and the motion attribute corresponding to the rotation control type includes a rotation direction attribute; wherein, the total number of gesture recognition points takes at least one of the following values: single finger, two fingers, three fingers, four fingers, and five fingers; the swipe direction attribute is at least one of the following: up, down, left, right, upper left, upper right, lower left, and lower right; the scaling trend attribute is at least one of the following: open and pinch; and the rotation direction attribute is at least one of the following: clockwise and counterclockwise.

[0065] In one possible implementation, the information used to describe the first gesture includes first information and second information, wherein the first information is determined based on the first gesture attributes and the total number of gesture recognition points, and the second information is motion attributes.

[0066] In one possible implementation, at least one type of gesture factor includes at least one of the following: a quantity factor, used to characterize the total number of at least one gesture recognition point indicated by gesture sensing information; a direction factor, used to characterize the movement direction of at least one gesture recognition point; a distance factor, used to characterize the movement distance of at least one gesture recognition point; a center of gravity factor, used to characterize the distance between at least one gesture recognition point and the geometric center of gravity when the total number of gesture recognition points is greater than 2, the geometric center of gravity being the center of gravity of a geometric figure with at least one gesture recognition point as its vertex; and a speed factor, used to characterize the movement speed of at least one gesture recognition point.

[0067] In one possible implementation, the gesture sensing information includes position information of at least one gesture recognition point; the electronic device acquires at least one type of gesture factor based on the gesture sensing information, including: if the at least one type of gesture factor includes a center of gravity factor, determining the position information of the geometric center of gravity based on the position information of at least one gesture recognition point; and determining the center of gravity factor corresponding to each of the at least one gesture recognition points based on the position information of at least one gesture recognition point and the position information of the geometric center of gravity.

[0068] In one possible implementation, the gesture sensing information includes the position information of at least one gesture recognition point; the electronic device obtains at least one type of gesture factors based on the gesture sensing information, including: when the at least one type of gesture factors includes a relative factor, for each relative factor, based on the position information and position reference information of at least one gesture recognition point, determining the relative factor corresponding to each gesture recognition point among the at least one gesture recognition point, wherein the relative factor is at least one of a direction factor, a speed factor, and a distance factor.

[0069] In one possible implementation, the first system service resides in the system framework layer of the electronic device.

[0070] Thirdly, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processing circuit, implement the method as described in any possible embodiment of the first aspect.

[0071] Fourthly, a chip system is provided, the chip system including a processing circuit and a storage medium, the storage medium storing computer instructions; when the computer instructions are executed by the processing circuit, they implement the method as described in any possible embodiment of the first aspect.

[0072] Fifthly, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the method as described in any of the possible embodiments of the first aspect.

[0073] The technical effects corresponding to any of the implementation methods in aspects two through five can be found in the first aspect and the technical effects corresponding to any of the implementation methods in the first aspect, and will not be repeated here. Based on the implementation methods provided in the above aspects, this application can further combine them to provide more implementation methods. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0075] Figure 2A A schematic diagram of the functional modules provided in the embodiments of this application;

[0076] Figure 2B A software structure block diagram of an electronic device provided in an embodiment of this application;

[0077] Figure 3 A flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0078] Figure 4 A schematic diagram illustrating the functional gesture subscription process provided in the embodiments of this application;

[0079] Figure 5 This is a schematic diagram of a loop for the gesture recognition method provided in the embodiments of this application;

[0080] Figure 6 This is a schematic diagram showing the distribution of directional intervals provided in an embodiment of this application;

[0081] Figure 7 A schematic diagram of the centroid distribution of a geometric figure provided in an embodiment of this application;

[0082] Figure 8 A schematic diagram of the kneading operation provided in an embodiment of this application;

[0083] Figure 9 A global schematic diagram of the gesture recognition method provided in the embodiments of this application;

[0084] Figure 10 This is a schematic diagram of the trajectory of a swipe gesture provided in an embodiment of this application;

[0085] Figure 11 This is a schematic diagram of the trajectory of an opening gesture provided in an embodiment of this application;

[0086] Figure 12 This is a schematic diagram of the trajectory of a rotation gesture provided in an embodiment of this application;

[0087] Figure 13 The interaction flowchart of the gesture recognition method provided in the embodiments of this application is shown.

[0088] Figure 14 This is a diagram illustrating the responsiveness of a swipe gesture provided in an embodiment of this application.

[0089] Figure 15 This is a diagram illustrating the responsiveness of the pinching gesture provided in an embodiment of this application.

[0090] Figure 16 This is a diagram illustrating the changing effects of the pinching gesture provided in the embodiments of this application;

[0091] Figure 17 An interactive flowchart of another gesture recognition method provided in this application embodiment;

[0092] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0093] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0094] The terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0095] Hereinafter, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0096] In this application, the terms "exemplarily" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0097] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0098] As described in the background section, the functions supported by applications (such as system tools and third-party applications) can be triggered by gestures. Gestures can be understood as actions performed by the user's hands to control electronic devices and implement preset function control commands.

[0099] After detecting a user action, the sensors in the electronic device determine the location information of at least one gesture recognition point based on the user's action, thus obtaining gesture sensing information. The user action corresponds to a gesture to be recognized, and the electronic device recognizes the gesture based on the gesture sensing information. Taking contact gestures as an example, the gesture recognition point refers to the contact point formed when the user's hand touches the touchscreen. The contact point remains for a certain period of time during the gesture. The electronic device can obtain multiple gesture sensing information by periodically detecting the location information of the contact point; the movement trajectory of the gesture recognition point can be analyzed through the gesture sensing information to identify the gesture corresponding to the user's action.

[0100] In one possible implementation, taking the gesture recognition process as an example, after the electronic device acquires gesture sensing information, it sends this information to a running third-party application. The application has its own gesture recognition logic, which transmits the gesture sensing information to this logic. The gesture recognition logic analyzes the movement trajectory of the gesture recognition point using the gesture sensing information, outputs the gesture recognition result, and thus determines the functional gesture that matches the gesture to be recognized. Subsequently, the application executes the function triggered by that gesture.

[0101] As application functionality continues to expand, more and more applications support gesture control. Currently, applications support gesture control types including, but not limited to, at least one of the following: swipe control, zoom control, and rotation control. Since the total number of single gestures is limited, as the types of application functions continue to increase, multi-finger gestures and combined gestures (such as swiping then rotating, pinching then rotating, or swiping then pinching) are also gradually being added to applications. The gesture recognition logic within an application is based on the gesture design supported by the application, in order to respond to user operations and perform gesture recognition.

[0102] Because different gestures use different gesture recognition logic, adding a new gesture to an application requires updating the application's gesture recognition logic to ensure accurate recognition. This forces application developers to dynamically adjust and add gesture recognition logic as functionality expands. Clearly, this gesture recognition method has poor scalability and incurs significant manpower costs.

[0103] Furthermore, conflicts may occur during the gesture recognition process of various applications in this embodiment. For example, at a certain moment, the electronic device responds to the user's operation, acquires gesture sensing information, and reports the gesture recognition information to Application 1 and Application 2; Application 1 and Application 2 simultaneously perform gesture recognition based on the gesture sensing information, resulting in multiple applications repeatedly performing gesture recognition. Repeated gesture recognition wastes the computing power of the electronic device and can easily lead to lag.

[0104] Furthermore, due to differences in gesture recognition logic across different applications, the gestures recognized by each application based on the same gesture sensing information may be inconsistent, leading to lower accuracy in gesture control. Therefore, the above method is prone to gesture control anomalies, affecting the accuracy and smoothness of the gesture interaction process.

[0105] To address the aforementioned issues, this application provides a gesture recognition method executed by a first system service in an electronic device. After acquiring gesture sensing information, the first system service obtains a first gesture using this information. Subsequently, the first system service reports information describing the first gesture to a first application subscribed to the first gesture, enabling the first application to perform a first operation corresponding to the first gesture. Compared to applications independently performing gesture recognition, having the first system service uniformly handle gesture recognition—after recognizing the first gesture, the first system service notifies the applications subscribed to that gesture to perform the corresponding operation—helps avoid conflicts between gesture recognition methods across multiple applications and conserves the computing power of the electronic device.

[0106] Moreover, this method unifies the recognition logic of different types of gestures, thereby ensuring that the same gesture can be recognized based on the same gesture sensing information, thus guaranteeing the reliability of the gesture recognition process.

[0107] In addition, this embodiment supports extending the gesture recognition logic when new gestures are added to the application (e.g., the process of obtaining gesture factors through gesture sensing information is not affected), which helps to improve the scalability of the gesture recognition method.

[0108] In some embodiments, the gesture recognition method can be applied to an electronic device. The electronic device is also referred to as: access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent, or user equipment.

[0109] The electronic devices in the embodiments of this application include, but are not limited to: mobile phones, personal computers, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, vehicle-mounted terminals, and roadside units (RSUs) with terminal functions.

[0110] Optionally, the operating system installed on the electronic device includes, but is not limited to, Or other operating systems.

[0111] For easier understanding, please refer to Figure 1 , Figure 1 A schematic diagram of the electronic device is shown as follows: Figure 1 As shown, the electronic device 100 includes at least: a processor 110, a memory 120, a communication module 130, a display screen 140, and a sensor 150.

[0112] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0113] Memory 120 includes external memory and internal memory. The internal memory can be used to store executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. Memory 120 can be located within processor 110 and is used to store instructions and data. External memory refers to a memory card connected to electronic device 100 via an external memory interface.

[0114] The communication module 130 is used to implement the communication function of the electronic device 100. Optionally, the communication module 130 includes functional modules such as an antenna, a mobile communication module, a wireless communication module, a modem processor, and a baseband processor. Optionally, at least some of the functional modules of the mobile communication module can be housed within the processor. The wireless communication module is used to support the electronic device in wireless communication.

[0115] The electronic device 100 implements its display function through functional modules such as a GPU and a display screen 140. The display screen 140 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include one or N display screens 140, where N is a positive integer greater than 1. The GPU and display screens 140 can be used to support the electronic device 100 in displaying interactive interfaces for system tools or third-party applications, where the interactive interface displays content for user interaction. Users can use gestures to change or adjust the displayed content in the interactive interface.

[0116] The sensor 150 may include a pressure sensor 150A, a touch sensor 150B, etc. The pressure sensor 150A senses pressure signals and converts them into electrical signals. In some embodiments, the pressure sensor 150A may be positioned below the display screen 140. The electronic device 100 may also calculate the touch position based on the detection signal from the pressure sensor 150A. The touch sensor 150B is also called a "touch panel." The touch sensor 150B may be positioned on the display screen 140, and the touch sensor 150B and the display screen 140 together form a touchscreen, also called a "touch screen." The touch sensor 150B detects touch operations applied to or near it. The touch sensor 150B can transmit the detected touch operation to the application processor to determine the touch event type and execute the corresponding instructions, thereby enabling interaction with the user.

[0117] The sensor module 150 may also include an optical sensor. The optical sensor may include an optical camera, etc. Optionally, the optical camera may be positioned below the display screen 140 to capture hand gestures upon sensing user interaction.

[0118] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0119] like Figure 2A As shown, the electronic device 100 provided in this application embodiment may include the following software functional modules: a gesture sensing module 210 and a contention and recognition module 220. Optionally, the first system service includes the contention and recognition module 220. Exemplarily, the first system service may also include the gesture sensing module 210.

[0120] The gesture sensing module 210 is used to acquire gesture sensing information of the gesture to be recognized when a gesture to be recognized is detected. The gesture sensing module 210 sends the gesture sensing information to the contention and recognition module 220. Optionally, the gesture recognition point of the gesture to be recognized is recognized by the sensor of the electronic device; the gesture sensing module 210 determines the position information of the gesture recognition point and generates gesture sensing information.

[0121] The gesture sensing information is also called a gesture raw event. For example, the gesture sensing module 210 sends a gesture raw event to the contention and recognition module 220 so as to transmit the gesture sensing information to the contention and recognition module 220.

[0122] The contention and recognition module 220 is used to obtain the first gesture based on the gesture sensing information. Optionally, the contention and recognition module 200 is also used to report information describing the execution of the first gesture to a first application, so that the first application performs the first operation corresponding to the first gesture, wherein the first application is an application that subscribes to the first gesture.

[0123] In some embodiments, the competition and recognition module 220 is equipped with a fusion recognition algorithm and a multi-finger competition mechanism. The fusion recognition algorithm integrates the recognition logic of multiple types of gestures, and the multi-finger competition mechanism adapts to gestures performed by different numbers of fingers. Please refer to the embodiments below for details.

[0124] In other embodiments, the gesture sensing module 210 and the contention and recognition module 220 are deployed within the operating system of the electronic device 100. In the field of software development, an operating system can be divided into a system side and an application side. The system side provides basic services, while the application side interfaces with user requirements. For example... Figure 2AAs shown, the gesture sensing module 210 and the contention and recognition module 220 are built into the operating system. In this case, the gesture sensing module 210 and the contention and recognition module 220 are located on the system side of the operating system. Optionally, native system tools belong to the system side, while third-party applications belong to the application side.

[0125] In one example, the application on the business side notifies the contention and recognition module 220 of at least one gesture it supports via a business subscription. After the gesture sensing module 210 acquires the gesture sensing information, the contention and recognition module 220 acquires a first gesture based on the gesture sensing information. This first gesture is the gesture that was notified to the contention and recognition module 220 via the business subscription.

[0126] Optionally, the competition and identification module 220 determines a first application from at least one application based on the subscription information in the above-mentioned business subscription process, and notifies the first application to perform the first operation corresponding to the first gesture, so that the application side can start the business processing process to realize gesture control.

[0127] In some embodiments, the software system of the electronic device 100 adopts a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100. Figure 2B This is a software structure block diagram of an electronic device provided in an embodiment of this application. The electronic device is... Figure 1 The electronic device 100 in the system. A layered architecture divides the software into several layers, each with a clear role and function. Optionally, layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the TS (arkts runtime) runtime and system libraries, and the kernel layer.

[0128] The application layer can include a series of application packages.

[0129] like Figure 2B As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, wireless fidelity (WLAN), Bluetooth, music, video, and SMS.

[0130] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. Optionally, applications in the application layer architecture layer support gesture control.

[0131] like Figure 2BAs shown, the application framework layer may include a window manager, content providers, a view system, a phone manager, a resource manager, a notification manager, and primary system services. The application framework layer is also known as the system framework layer.

[0132] The window manager is used to manage window programs. It can be used to obtain screen size, determine if a status bar is present, lock the screen, capture screenshots, and more.

[0133] Content providers store and retrieve data, making that data accessible to applications. This data includes: videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0134] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon can include views for displaying text and views for displaying images.

[0135] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0136] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and so on.

[0137] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. Examples include notifications of download completion and message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0138] The first system service provides a unified gesture recognition service. Optionally, the first system service includes a contention and recognition module. Optionally, the first system service also includes a gesture sensing module.

[0139] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0140] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0141] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. The system libraries include multiple functional modules, such as the surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), and 2D graphics engines (such as SGL).

[0142] The Surface Manager is used to manage the display subsystem and provides fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0143] The media library supports playback and recording of various commonly used audio and video formats, as well as still image files. It supports multiple audio and video coding formats, such as: Moving Picture Experts Group 4 (MPEG4), Advanced Video Coding (H.264), Moving Picture Experts Group Audio Layer III (MP3), Advanced Audio Coding (AAC), Adaptive Multi Rate (AMR), Joint Photographic Experts Group (JPG), and Portable Network Graphics (PNG).

[0144] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0145] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0146] The gesture recognition method provided in this application will be specifically described below using an electronic device as the execution subject. Optionally, this embodiment is... Figure 1 The first system service is implemented in the electronic device 100. This embodiment uses an electronic device including a gesture sensing module and a contention and recognition module as an example for explanation. Please refer to the relevant documentation. Figure 3 , Figure 3 This is a flowchart illustrating the gesture recognition method provided in an embodiment of this application. Figure 3As shown, the gesture recognition method provided in this application embodiment includes at least steps S310 to S340.

[0147] S310, the first system service responds to user operations and acquires gesture sensing information.

[0148] In this system, user actions trigger gesture control. In some embodiments, the user action is a gesture to be recognized sensed by the electronic device, used to instruct the electronic device to perform a preset function. The preset function is implemented by an application installed on the electronic device performing a corresponding operation. For example, the preset function includes at least one of the following: refreshing the displayed content, adjusting the display effect of the displayed content, opening / closing the application, calling a background application, or moving an application to the background.

[0149] Optionally, the default function is implemented by an application in the electronic device. For example, the application includes at least one of the following: a native system application and a third-party application.

[0150] The system's native applications include system tools and applications that cannot be uninstalled from the operating system. System tools are used to service and maintain the operating system. System functions include, but are not limited to, system management, performance optimization, hardware management, data cleanup, and security maintenance. Examples include the command prompt, task manager, and disk cleaner. Applications that cannot be uninstalled from the operating system include the app store, notepad, photo album, and calling applications. For example, the system's native applications are developed by the operating system's development team.

[0151] Third-party applications are provided by operating system developers or other developers. Optionally, the electronic device obtains the installation package of the third-party application from the content distribution platform and installs the third-party application onto the electronic device by decompressing the installation package.

[0152] In some embodiments, depending on the application type, gestures include at least one of the following: gestures for triggering functions provided by native system applications, and gestures for triggering functions provided by third-party applications.

[0153] Optionally, the gesture to be recognized is made by a user of the electronic device. The user controls the electronic device to perform a corresponding operation through the gesture in order to complete a preset function. For example, after detecting a user operation, a first system service in the electronic device recognizes a first gesture corresponding to the user operation; subsequently, the first system service notifies the application to perform the operation corresponding to the gesture. The process of "determining the gesture corresponding to the user operation" mentioned here is the gesture recognition process.

[0154] In some embodiments, the gestures supported by the electronic device are gestures subscribed to by the application. Optionally, before performing step S310, at least one gesture supported by the application is obtained through subscription information from the application corresponding to the user's operation.

[0155] Optionally, the gesture recognition process is completed by a contention and recognition module. For example, the first system service includes a contention and recognition module. For example, the contention and recognition module is provided with a subscription interface, through which the application notifies the contention and recognition module of at least one supported gesture. For instance, the application reports subscription information to the first system service through the subscription interface.

[0156] like Figure 4 As shown, the competition and recognition module obtains subscription information from the application; the competition and recognition module then notifies the application of the subscription result. The subscription information represents at least one gesture supported by the application, and the subscription result notifies the application that the gesture subscription was successful.

[0157] For example, after the application obtains the subscription result, at least one gesture supported by the application is called the application's subscription gesture, and the gesture recognition process for the subscription gesture is completed by the competition and recognition module. For example, for each of the at least one gesture, the subscription information includes a type identifier and attribute parameters. The type identifier is used to characterize the gesture control type to which the gesture belongs, and the attribute parameters are used to characterize the specific implementation of the gesture within its respective gesture control type. The attribute parameters include at least one of the following: movement direction, rotation direction, movement distance, total number of gesture recognition points, etc.

[0158] Optionally, after a new application is installed on an electronic device or an application adds a gesture, the application notifies the user of subscription information about the new gesture through a subscription interface.

[0159] For example, an electronic device stores and maintains a gesture mapping table, which records the mapping relationship between gestures and applications that subscribe to those gestures. For instance, for a given gesture, the gesture mapping table stores the gesture identifier and the application identifiers of the applications that subscribe to that gesture. By setting up the gesture mapping table, after the first system service recognizes the first gesture, it can quickly determine the applications that can subscribe to that first gesture.

[0160] In some embodiments, the gesture control type to which the gesture belongs includes at least one of the following: swipe control type, zoom control type, rotation control type, etc. In the embodiments of this application, swipe control type can also be abbreviated as swipe type, zoom control type can also be abbreviated as zoom type, and rotation control type can also be abbreviated as rotation type.

[0161] The swipe control type includes at least one gesture achieved by extending the hand and moving the hand through the elbow and / or arm. The gestures included in the swipe control type can perform operations including, but not limited to: refreshing displayed content, switching display pages, adjusting brightness, adjusting volume, adjusting video playback progress, launching background applications, and moving applications to the background.

[0162] The zoom control type includes at least one gesture achieved by fixing the wrist and contracting or opening the knuckles. The gestures included in the zoom control type can perform operations including, but not limited to: adjusting the display size of the content (such as zooming in or out of an image or text), adjusting camera shooting parameters, cropping the display content, and calling system tools (such as accessing desktop layout tools by pinching the fingers).

[0163] The rotation control type includes at least one gesture achieved by rotating the wrist while keeping the relative position of the fingers constant. The gestures included in the rotation control type can perform operations such as adjusting the display angle of the displayed content.

[0164] In some embodiments, a gesture control type includes multiple gestures, which have similar overall movement trends but differ in their specific implementations. In this application embodiment, each of the multiple gestures possesses attribute parameters, which can distinguish the various gestures included under the same gesture control type. For example, the attribute parameters include a gesture attribute dimension (total number of gesture recognition points) and a motion attribute dimension. For instance, multiple gestures belonging to the same gesture control type may have different total numbers of gesture recognition points.

[0165] Optionally, the gesture attributes include at least one of the following: the total number of gesture recognition points and motion attributes. The total number of gesture recognition points can be at least one of the following: single finger, two fingers, three fingers, four fingers, or five fingers. The motion attributes describe the overall motion trajectory of the finger corresponding to each of at least one gesture recognition point.

[0166] For example, the motion attributes corresponding to the slide control type include a slide direction attribute. In one example, the slide direction attribute is at least one of the following: up, down, left, right, upper left, upper right, lower left, and lower right.

[0167] For example, the motion attributes corresponding to the scaling control type include scaling trend attributes. In one example, the scaling trend attribute is at least one of the following: open, pinch. That is, the scaling control type can be further subdivided into open control type and pinch control type.

[0168] For example, the motion attributes corresponding to the rotation control type include a rotation direction attribute. In one example, the rotation direction attribute is at least one of the following: clockwise or counterclockwise.

[0169] For example, if a gesture belongs to the swipe control type, the swipe direction attribute is right, and the total number of gesture recognition points is 2, then this gesture is a two-finger swipe operation. As another example, if a gesture belongs to the zoom control type, the zoom trend attribute is pinch, and the total number of gesture recognition points is 5, then this gesture is a five-finger pinch operation. And as yet another example, if a gesture belongs to the rotation control type, the rotation direction attribute is clockwise, and the total number of gesture recognition points is 4, then this gesture is a four-finger clockwise rotation operation.

[0170] In some embodiments, the gesture is a single gesture. For example, the gesture to be recognized is a two-finger swipe gesture.

[0171] In other embodiments, the gesture is a compound gesture. A compound gesture is obtained by combining at least two individual gestures. For example, a gesture might involve first pinching with three fingers, then switching to a three-finger pinch. Or, a gesture might involve first sliding with three fingers, then switching to rotating with two fingers. In this case, there is at least one shared recognition point between the at least two individual gestures included in the compound gesture.

[0172] In this context, a shared recognition point refers to the same gesture recognition point shared by at least two individual gestures included in a composite gesture. That is, when the first individual gesture in a composite gesture switches to the second individual gesture, the sensing signal corresponding to at least one shared recognition point among the at least one gesture recognition point of the first electronic gesture does not disappear. In this case, during the continuous execution of user operations, the first system service in the electronic device performs gesture recognition every detection interval, which helps to detect gesture changes promptly.

[0173] In some embodiments, gestures are classified into at least two types according to the acquisition method: contact gestures and non-contact gestures. Contact gestures refer to actions performed by a user's hand after contact with an electronic device; in this case, the gesture to be recognized can be sensed by a pressure sensor in the electronic device. The gesture to be recognized refers to the first gesture to be recognized; before the first gesture is identified and determined by the gesture recognition method, it can be called the gesture to be recognized.

[0174] Non-contact gestures refer to actions performed by a user without physical contact with an electronic device. For non-contact gestures, the gesture to be recognized can be captured by an optical camera within the electronic device. The electronic device performs image recognition on the captured image of the gesture to be recognized, obtaining at least one gesture recognition point; the electronic device determines the gesture sensing information based on the display position of at least one gesture recognition point in the image.

[0175] In some embodiments, gesture sensing information is used to describe the posture of the gesture to be recognized. Optionally, the gesture sensing information includes the position information of at least one gesture recognition point. The gesture recognition point is a key point of the gesture to be recognized. For example, during the continuous execution of the gesture to be recognized, the position information of the gesture recognition point changes as the gesture moves.

[0176] Optionally, for contact-based gestures, the gesture recognition point includes at least one contact point between the user's hand and the display screen of the electronic device. For example, the gesture recognition point is the contact point between the fingertip and the display screen of the electronic device. For example, assuming there are four contact points between the gesture to be recognized and the display screen of the electronic device, all four contact points are gesture recognition points, and the gesture sensing information is used to describe the position information of the above four contact points.

[0177] Optionally, for non-contact gestures, gesture recognition points include points on the edges of the gesture to be recognized. For example, after detecting a gesture to be recognized, an optical camera in the electronic device captures a gesture image, and the gesture recognition points are located on the contour lines of the gesture displayed in the gesture image. Gesture recognition points include the distal ends of the fingers. In this case, gesture recognition points can be obtained by performing feature point recognition on the gesture image. Feature point recognition methods include, but are not limited to, real-time object detection algorithms (such as YOLO).

[0178] For example, the location information of gesture recognition points is represented by coordinates. For instance, the gesture sensing information includes s gesture recognition points, where the location information of gesture recognition point p1 is represented as (x1, y1), the location information of gesture recognition point p2 is represented as (x2, y2), and so on, where s is a positive integer. For example, s is a positive integer less than or equal to 5.

[0179] For example, for contact-based gestures, the position information of the gesture recognition point is determined using a display coordinate system, which is two-dimensional. The display coordinate system is a two-dimensional coordinate system used to locate the displayed content. For instance, for a gesture to be recognized that belongs to the non-contact category, the position of the gesture recognition point can be determined using an image coordinate system, which is two-dimensional. The image coordinate system is a two-dimensional coordinate system used to locate pixels in an image.

[0180] In some embodiments, when the gesture to be recognized corresponds to multiple gesture recognition points, the gesture sensing information also includes a recognition point index. The recognition point index identifies a gesture recognition point, and different gesture recognition points have different gesture indices. Here, "multiple" refers to two or more.

[0181] Optionally, the recognition point index is represented by a number, such as multiple gesture recognition points having recognition point indices that are progressively increasing natural numbers (e.g., progressively increasing by a step size of 1). For example, for a gesture to be recognized that belongs to the contact type, the recognition point index is called the finger index or index, which is used to indicate the finger number corresponding to the gesture recognition point.

[0182] Figure 5 This is a schematic diagram illustrating a loop of the gesture recognition method provided in an embodiment of this application. Figure 5 As shown, the first system service acquires the gesture sensing information of the gesture to be recognized; if the gesture to be recognized meets the competition recognition conditions, the first system service executes the subsequent steps of gesture recognition (please refer to the description of steps S320, S330 and S340 below); if the gesture to be recognized does not meet the competition recognition conditions, the first system service reacquires the gesture sensing information.

[0183] Optionally, the competition recognition condition is used to determine whether the gesture recognition process is complete. For example, the competition recognition condition may include either "the functional gesture matching the gesture to be recognized is not determined" or "the gesture to be recognized has moved." Whether the gesture to be recognized has moved can be determined by multiple gesture sensing information. For example, the electronic device acquires two gesture sensing information sequentially, and can determine whether the gesture to be recognized has moved based on these two pieces of information. For instance, if the two gesture sensing information are the same, the electronic device can determine that the gesture to be recognized has not moved. Or, if the two gesture sensing information are different, the electronic device can determine that the gesture to be recognized has moved.

[0184] In one example, the competition recognition condition is that the functional gesture matching the gesture to be recognized is not determined, and the gesture to be recognized has moved. That is, no gesture to be recognized is currently recognized, and the user operation has not ended. Exemplarily, when the functional gesture matching the gesture to be recognized is not determined, and the gesture to be recognized has moved, the first system service in the electronic device performs subsequent steps of gesture recognition; when it is determined that the functional gesture matching the gesture to be recognized is determined, or when the gesture to be recognized has not moved, the electronic device re-executes step S310.

[0185] If the gesture to be recognized has not moved, the movement trend of the gesture cannot be determined based on the gesture sensing information. In this case, the subsequent gesture recognition steps are not performed, which helps to avoid the electronic device from performing invalid calculations.

[0186] In some embodiments, step S310 is performed by the gesture sensing module. Optionally, from the time the gesture to be recognized is detected until the gesture recognition ends, the gesture sensing module performs step 310 multiple times to acquire gesture sensing information multiple times.

[0187] For example, the gesture sensing module acquires gesture sensing information every first time interval. This first time interval is preset, such as 0.011 seconds, 0.017 seconds, or 0.034 seconds. The specific value of the first time interval depends on factors such as the hardware components and power consumption mode of the electronic device, and is not limited here. For instance, if the gesture to be recognized is a contact gesture, the electronic device detects the user's finger touching the touchscreen and triggers the acquisition of gesture sensing information every first time interval. Then, when the user's finger leaves the display screen, the electronic device stops acquiring gesture sensing information. During this process, the electronic device can acquire multiple gesture sensing information entries.

[0188] In one example, after acquiring gesture sensing information, the first system service in the electronic device determines a first gesture that matches the gesture to be recognized based on the gesture sensing information. If a gesture matching the gesture to be recognized is recognized in a subsequent step, the gesture recognition ends; if no gesture matching the gesture to be recognized can be recognized, the first system service in the electronic device re-executes step S310 until a first gesture matching the gesture to be recognized is determined, or until the number of times the gesture sensing information is acquired exceeds a threshold, indicating that the gesture recognition has failed. The threshold for the number of acquisitions is preset, such as 3 times, 5 times, etc.

[0189] Compared to related technologies that wait to acquire gesture sensing information at multiple moments or only start executing subsequent gesture recognition steps after detecting the end of the gesture to be recognized, this embodiment starts executing subsequent gesture recognition steps immediately after acquiring gesture sensing information each time. This helps to shorten the duration of the gesture recognition process, so that the gesture matching the gesture to be recognized can be determined before the gesture to be recognized ends. This helps to improve the response speed of electronic devices to user operations, reduce response latency, and improve the smoothness of using electronic devices.

[0190] S320, the first system service obtains at least one type of gesture factor based on gesture sensing information.

[0191] In this system, each of at least one type of gesture factor is used to describe gesture parameters in a corresponding dimension, and different types of gesture factors are used to describe the motion trend of the gesture to be recognized from different dimensions. Optionally, gesture parameters refer to physical quantities that can be sensed by the electronic device during the continuous operation of the user. For example, gesture parameters can also be calculated from the sensed physical quantities. Different dimensions of gesture parameters are used to characterize different physical quantities. For example, the dimensions of gesture parameters are not limited to at least one of the following: quantity dimension, direction dimension, center of gravity dimension, velocity dimension, and distance dimension.

[0192] Optionally, gesture factors are classified according to their determination method, including at least two types: relative gesture factors and absolute gesture factors. For example, absolute gesture factors are determined by the electronic device based on gesture sensing information, while relative gesture factors are determined by the electronic device based on gesture sensing information and other reference information. These other reference information include, but are not limited to, gesture sensing information acquired at other times. Relative gesture factors are also called relative factors, and absolute gesture factors are also called absolute factors.

[0193] In some embodiments, the type of gesture factor includes, but is not limited to, at least one of the following: quantity factor, direction factor, center of gravity factor, distance factor, and velocity factor. Optionally, the quantity factor is used to describe gesture parameters in the quantity dimension, the direction factor is used to describe gesture parameters in the direction dimension, the center of gravity factor is used to describe gesture parameters in the center of gravity dimension, the velocity factor is used to describe gesture parameters in the velocity dimension, and the distance factor is used to describe gesture parameters in the distance dimension.

[0194] The quantity factor is used to characterize the total number of gesture recognition points included in the gesture to be recognized. Optionally, the quantity factor is used to determine the gesture that matches the gesture to be recognized.

[0195] In some embodiments, the quantity factor is an absolute gesture factor. Optionally, the first system service in the electronic device determines the quantity factor based on gesture sensing information, including: the first system service counts the number of gesture recognition points included in the gesture sensing information to obtain the quantity factor. For example, when the gesture sensing information includes recognition point numbers, the electronic device subtracts the recognition point number with the smallest value from the recognition point number with the largest value, and adds one to the resulting value to obtain the quantity factor.

[0196] A direction factor is used to characterize the movement direction of at least one gesture recognition point. Optionally, the direction factor is used to determine the gesture control type to which the gesture to be recognized belongs. Furthermore, the direction factor can also be used to determine attribute parameters, as detailed in the examples below.

[0197] Optionally, the direction factor includes the direction factor corresponding to each gesture recognition point. The direction factor corresponding to a certain gesture recognition point is used to characterize the movement direction of that gesture recognition point. By comparing the direction factors corresponding to each gesture recognition point, the similarities and differences in the movement directions of each gesture recognition point can be determined.

[0198] It is understandable that the movement trajectory of the gesture to be recognized is unstable during its continuous execution. For the same gesture recognition point, the direction factor determined based on gesture recognition information at different times may also be different. Therefore, the direction factor is time-sensitive. For example, the direction factor obtained from the gesture sensing information at the first moment is used to characterize the movement direction of the gesture to be recognized within the first time period, and the direction factor corresponding to each gesture recognition point is used to characterize the movement direction of each gesture recognition point within the first time period.

[0199] The first time period refers to a period including the first moment. For example, the first time period is a period ending at the first moment. For instance, if an electronic device acquires gesture sensing information 2 at the second moment and gesture sensing information 1 at the first moment, and determines at least one type of gesture factor based on gesture sensing information 1, the first time period is a period starting at the second moment and ending at the first moment. The second moment is earlier than the first moment.

[0200] The orientation factor is a relative gesture factor. Optionally, the first system service obtains the orientation factor based on the gesture sensing information, including: for each gesture recognition point among at least one gesture recognition point, the first system service obtains the orientation factor corresponding to the gesture recognition point based on the position information of the gesture recognition point and the position reference information of the gesture recognition point.

[0201] The first moment refers to the moment when the gesture sensing information was acquired during the most recent execution of step 310 before this execution of step 320.

[0202] Optionally, the location reference information is used to represent the historical location of the gesture recognition point during the user's operation. For example, the location information is obtained from the gesture sensing information at a first moment, and the location reference information is obtained from the gesture sensing information at a second moment. The gesture sensing information at the second moment belongs to the other reference information mentioned above, and the second moment is earlier than the first moment.

[0203] Optionally, the location reference information corresponds to the starting position of the gesture recognition point during a segment of movement of the gesture to be recognized; the location information corresponds to the ending position of the gesture recognition point during the same segment of movement of the gesture to be recognized. In the embodiments of this application, the location reference information is also called reference information, and the two have the same meaning; the location information is also called first location information, and the two have the same meaning.

[0204] For example, multiple different first moments correspond to the same second moment. For instance, moment 1 and moment 2 are both first moments, and the second moments corresponding to moment 1 and moment 2 are both moment 3. Time 3 can be the starting moment of the gesture recognition process. In one example, after detecting the gesture to be recognized, the electronic device acquires the gesture sensing information at the initial moment and stores the gesture sensing information at the initial moment so that, during the execution of step S330, the position reference information of each gesture recognition point can be obtained based on the gesture sensing information at the initial moment.

[0205] For example, each first moment corresponds to a different second moment. For instance, there is a k-time interval between the first and second moments. The gesture sensing information at the first moment is the gesture sensing information at moment i, and the second moment is moment ik*t, where k is a positive number, i is a positive number greater than k, and t represents the duration of the first time interval. If k = 1, then the second moment is the moment when the gesture sensing information was previously acquired.

[0206] In some embodiments, the first system service obtains the direction factor corresponding to the gesture recognition point based on the location information of the gesture recognition point and the location reference information of the gesture recognition point, including: the first system service obtains a direction vector based on the location reference information and the location information; the first system server obtains the direction factor corresponding to the gesture recognition point based on the direction vector.

[0207] Optionally, the first system service obtains the direction factor corresponding to the gesture recognition point based on the direction vector, including: the first system service determines the direction angle of the direction vector as the direction factor corresponding to the gesture recognition point. Here, the direction angle refers to the angle of deflection of the direction vector relative to the x-axis in the display coordinate system or image coordinate system, around the counterclockwise direction.

[0208] For example, the display coordinate system can be a two-dimensional coordinate system with the lower left corner as the origin, the horizontal direction to the right as the positive x-axis, and the horizontal direction upward as the positive y-axis, in the portrait mode of the electronic device.

[0209] In one example, the first position information of gesture recognition point 1 is (endX, endY), the reference position information of gesture recognition point 1 is (startX, startY), and the formula for calculating the direction angle θ of the direction vector is:

[0210] Math.atan2 is used to calculate the planar angle between the direction vector and the positive x-axis.

[0211] Optionally, the first system service obtains the direction factor corresponding to the gesture recognition point based on the direction vector, including: the electronic device determines the direction factor corresponding to the gesture recognition point based on the direction interval to which the direction vector belongs. For example, if the direction angle of the direction vector belongs to the first direction interval among N direction intervals, the first system service represents the direction factor corresponding to the gesture recognition point through the first direction interval, where N is a positive integer greater than 1 and less than 360.

[0212] In this context, the range [0, 360°] on the two-dimensional plane is divided into N directional intervals, each corresponding to an angle range [a°, b°), where a and b are real numbers within the range [0, 360].

[0213] In one example, such as Figure 6 As shown, the two-dimensional plane is divided into 8 directional intervals, each with an angle span of 45°. The 8 directional intervals are: right interval [-22.5°, 22.5°), lower right interval [22.5°, 67.5°), lower interval [67.5°, 112.5°), lower left interval [112.5°, 157.5°), left interval [157.5°, -157.5°), upper left interval [-157.5°, -112.5°), upper interval [-112.5°, -67.5°), and upper right interval [-67.5°, -22.5°]. If the direction angle of a certain directional vector is 10°, then the directional vector belongs to the right directional interval, and the direction factor corresponding to the gesture recognition point is: right.

[0214] In this embodiment, on the one hand, the direction factor corresponding to the gesture recognition point is determined by the direction interval to which the direction vector belongs, thus limiting the selectable values ​​of the direction factor. This helps to quickly compare whether the direction factors corresponding to at least one gesture recognition point are the same. This approach helps to accelerate the efficiency of gesture recognition.

[0215] On the other hand, the movement direction of each gesture recognition point is deviated during the movement process. This method helps to offset the impact of the movement deviation of each gesture recognition point on the accuracy of gesture recognition, thereby helping to improve the accuracy of the gesture recognition process.

[0216] A distance factor is used to characterize the movement distance of at least one gesture recognition point. Optionally, the distance factor is used to verify whether the gesture sensing information was generated by the gesture control process. That is, the first system service can determine whether gesture control has occurred, or whether the user operation is effective, through the distance factor, as detailed in the following embodiments.

[0217] The distance factor is a relative gesture factor. The distance factor, obtained from the gesture sensing information at the first moment, is used to characterize the movement distance of at least one gesture recognition point within the first time period. Optionally, each of the at least one gesture recognition point corresponds to a distance factor. Specifically, the distance factor corresponding to a particular gesture recognition point is used to characterize the movement distance of that gesture recognition point within the first time period.

[0218] Optionally, the first system service in the electronic device obtains a distance factor based on the gesture sensing information, including: for each gesture recognition point among at least one gesture recognition point, the first system service obtains the distance factor corresponding to the gesture recognition point based on the location information and the location reference information of the gesture recognition point. For a description of the location information and location reference information, please refer to the above introduction, which will not be repeated here.

[0219] In one example, the location information of the gesture recognition point is (endX, endY), and the location reference information is (startX, startY). The formula for calculating the distance factor L corresponding to this gesture recognition point is:

[0220] When the total number of gesture recognition points is greater than 2, the centroid factor is used to characterize the distance between the geometric centroid of the gesture to be recognized and at least one gesture recognition point.

[0221] The geometric centroid of the gesture to be recognized is the centroid of a geometric figure with at least one gesture recognition point as its vertex. For example... Figure 7 As shown, the gesture to be recognized corresponds to five gesture recognition points, namely P1, P2, P3, P4 and P5. The geometric centroid refers to the centroid G of the geometric figure 700 with P1, P2, P3, P4 and P5 as vertices.

[0222] Optionally, the center of gravity factor is used to determine the gesture control type to which the gesture to be recognized belongs. Furthermore, the center of gravity factor can also be used to determine attribute parameters, as detailed in the examples below.

[0223] The centroid factor is a relative gesture factor. Optionally, the centroid factor includes a centroid factor corresponding to at least one gesture recognition point. The centroid factor corresponding to a given gesture recognition point is used to characterize the distance between that gesture recognition point and the geometric centroid.

[0224] Optionally, the first system service in the electronic device obtains the center of gravity factor based on the gesture sensing information, including: the first system service obtains the position information of at least one gesture recognition point based on the gesture sensing information; the first system service obtains the position information of the geometric center of gravity based on the position information of the at least one gesture recognition point; for each of the at least one gesture recognition point, the first system service obtains the center of gravity factor corresponding to the gesture recognition point based on the position information of the gesture recognition point and the position information of the geometric center of gravity.

[0225] For example, assuming the total number of at least one gesture recognition point is p, where p is a positive integer greater than 2; the first system service obtains the position information of the geometric centroid based on the position information of at least one gesture recognition point, including: the first system service divides the p gesture recognition points into p-2 recognition point sets based on the position information of the p gesture recognition points; the first system service obtains the abscissa of the geometric centroid based on the area of ​​the p-2 triangles and the abscissa of each gesture recognition point; the first system service obtains the ordinate of the geometric centroid based on the area of ​​the p-2 triangles and the ordinate of each gesture recognition point in the recognition point set.

[0226] Each of the p-2 sets of recognition points includes 3 gesture recognition points, which can form a triangle. The p-2 sets of recognition points are used to divide the geometric figure into p-2 non-intersecting triangles. For example, at least two of the p-2 sets of recognition points include at least one identical gesture recognition point.

[0227] like Figure 8 As shown, the five gesture recognition points are A(x1, y1), B(x2, y2), C(x3, y3), D(x4, y4), and E(x5, y5). These five points form a geometric shape 800, which is divided into three sets of recognition points: Set 1 (including gesture recognition points A, B, and C), Set 2 (including gesture recognition points A, D, and C), and Set 3 (including gesture recognition points A, E, and D). The area of ​​triangle 810 formed by Set 1 is s1.

[0228] In one example, the x-coordinate of the geometric centroid is determined by the formula: get.

[0229] Among them, s i Let x represent the area of ​​the triangle formed by the set of the i-th identification points. i0 x i1 x i2Let represent the x-coordinate of each of the three gesture recognition points included in the i-th recognition point set, where i is a positive integer less than or equal to p-2.

[0230] In one example, the y-coordinate of the geometric centroid is obtained through the formula: get.

[0231] Among them, y i0 y i1 y i2 This represents the ordinate of each of the three gesture recognition points included in the i-th recognition point set. For explanations of other parameters, please refer to the above example.

[0232] For example, the first system service obtains the centroid factor corresponding to the gesture recognition point based on the position information of the gesture recognition point and the position information of the geometric centroid. This includes: the first system service processing the position information of the gesture recognition point and the position information of the geometric centroid using a distance formula to obtain a distance result; and the first system service determining the distance result as the centroid factor corresponding to the gesture recognition point. The formula used to calculate the distance result is the same as the formula used to calculate the distance factor in the above embodiment; please refer to the above for details, which will not be repeated here.

[0233] A velocity factor is used to characterize the movement speed of at least one gesture recognition point. Optionally, the velocity factor is used to determine the gesture control type to which the gesture to be recognized belongs.

[0234] The velocity factor is a relative gesture factor. The velocity factor is time-sensitive; it is obtained from the gesture sensing information at the first moment and is used to characterize the movement speed of at least one gesture recognition point over a period of time, including the first moment.

[0235] Optionally, the speed factor includes speed factors corresponding to at least one gesture recognition point. The speed factor corresponding to a particular gesture recognition point characterizes the movement speed of that gesture recognition point. For example, for a given gesture recognition point, the speed factor obtained based on the gesture sensing information at a first moment is used to characterize the movement speed of that gesture recognition point during a first time period.

[0236] For example, the first system service in the electronic device obtains a speed factor based on gesture sensing information, including: the electronic device obtaining a distance factor; for each gesture recognition point among at least one gesture recognition point, the electronic device determines the speed factor corresponding to the gesture recognition point based on the distance factor corresponding to the gesture recognition point and the duration of a first time period.

[0237] In one example, the velocity factor corresponding to a gesture recognition point is positively correlated with its corresponding distance factor and negatively correlated with the duration of the first time segment. For instance, the velocity factor corresponding to a gesture recognition point is equal to the distance factor divided by the duration of the first time segment.

[0238] In some embodiments, the acquisition of at least one type of gesture factor based on gesture recognition information by the electronic device is accomplished by a competition and recognition module within the electronic device. Optionally, the competition and recognition module is a system-level functional module within the operating system of the electronic device.

[0239] For example, after the gesture sensing module acquires the gesture sensing information at the first moment, the gesture sensing module sends the gesture sensing information at the first moment to the competition and recognition module so that the competition and recognition module can determine at least one type of gesture factor at the first moment based on the gesture sensing information at the first moment.

[0240] In some embodiments, at least one type of gesture factor is an essential gesture factor in the gesture recognition process. Optionally, the electronic device stores and maintains a set of essential factors, in which at least one type of gesture factor is included.

[0241] For example, if the gesture control types supported by the electronic device include swipe control, then the essential factor set includes gesture factors essential for determining the swipe control type. For example, if the gesture control types supported by the electronic device include swipe control and zoom control, then the gesture factors determined based on gesture sensing information include: gesture factors for determining the swipe control type and gesture factors essential for determining the zoom control type.

[0242] In some embodiments, the required gesture factors are related to the gestures supported by the electronic device, and the required factor set is stored and maintained by the contention and recognition module. Optionally, after receiving subscription information from the application, the contention and recognition module determines the required gesture factors corresponding to the new gesture. If the required factor set includes the required gesture factors corresponding to the new gesture, then there is no need to update the required factor set; if the required factor set does not include the required gesture factors corresponding to the new gesture, then the contention and recognition module adds the required gesture factors corresponding to the new gesture to the required factor set, thus obtaining an updated required factor set.

[0243] Optionally, if all applications supporting a certain gesture control type have been uninstalled, the competition and recognition module determines at least one essential gesture factor corresponding to the gesture control type, and deletes the at least one essential gesture factor from the essential factor group to obtain an updated essential factor group.

[0244] This embodiment helps reduce the total number of gesture factors that need to be acquired during the gesture recognition process. After the electronic device determines the necessary gesture factors based on the gesture sensing information, it can perform subsequent gesture recognition steps. Compared to calculating all types of gesture factors, calculating the necessary gesture factors consumes less computation. Therefore, this embodiment helps reduce the computational load required for the gesture recognition process, helps shorten the time spent acquiring gesture factors, and thus improves the efficiency of gesture recognition.

[0245] In some embodiments, there is a competition priority among gesture factors, which indicates the order in which each gesture factor in at least one class of gesture factors participates in the gesture recognition process. Optionally, gesture factors with higher competition priority are given priority in the recognition process of the gesture to be recognized. Optionally, if the gesture control type to which the gesture to be recognized belongs is determined based on a certain gesture factor, then other gesture factors in at least one class of gesture factors with lower competition priority will no longer participate in the step of determining the gesture control type to which the gesture to be recognized belongs.

[0246] Optionally, the various types of gesture factors can be arranged in descending order of competition priority, for example: quantity factor, direction factor, and center of gravity factor. For instance, speed factor and distance factor can be used if the direction factor meets the corresponding factor determination condition.

[0247] For example, when the direction factor meets the determination condition corresponding to the direction factor, the first system service in the electronic device verifies whether the gesture to be recognized belongs to the swipe control type based on the speed factor and the distance factor.

[0248] In one example, if both the speed factor and distance factor are greater than or equal to the speed threshold, the gesture to be recognized belongs to the swipe control type; if either the speed factor or the distance factor is less than the speed threshold, or the distance factor is less than the distance threshold, the gesture to be recognized does not belong to the swipe control type. The speed threshold and distance threshold are preset.

[0249] For example, the electronic device determines the calculation order of at least one type of gesture factor based on the competition priority, and determines at least one type of gesture factor according to the gesture sensing information in the calculation order. For a certain type of gesture factor, the higher the competition priority of the gesture factor, the earlier the calculation order of that type of gesture factor can be.

[0250] For example, if at least one type of gesture factor includes gesture factor 1, gesture factor 2, and gesture factor 3, and these three gesture factors are arranged in descending order of competition priority as: gesture factor 1, gesture factor 3, gesture factor 2, then the three gesture factors are ordered in ascending order of calculation as: gesture factor 1, gesture factor 3, gesture factor 2. In this example, during step S320, the electronic device first obtains gesture factor 1 based on the gesture sensing information, and finally obtains gesture factor 2 based on the gesture sensing information.

[0251] The above embodiments describe the process of the electronic device executing step S320 to obtain at least one type of gesture factor. After determining at least one type of gesture factor, the electronic device executes step S330 to determine the gesture control type to which the gesture to be recognized belongs based on the at least one type of gesture factor, and obtains the gesture recognition result of the gesture to be recognized. This step is described below through several embodiments.

[0252] S330, the first system service obtains the first gesture corresponding to the gesture sensing information based on at least one type of gesture factor.

[0253] The first gesture refers to the gesture that matches the gesture to be recognized. In other words, the first gesture is the gesture recognized by the electronic device that corresponds to the user's operation. Optionally, if gesture recognition is successful, a first system service in the electronic device obtains information describing the first gesture. For example, this information describing the first gesture is also called the gesture recognition result; the two have the same meaning.

[0254] In some implementations, the information describing the first gesture is used to trigger the execution of the first operation corresponding to the first gesture. Optionally, the information describing the first gesture includes a gesture identifier for the first gesture. For example, the gesture identifier may be the name, sequence number, etc., of the first gesture.

[0255] Optionally, the information used to describe the first gesture includes a type identifier and gesture attributes. For information on the type identifier and gesture attributes, please refer to the above description. Taking the first gesture as a three-finger right swipe gesture as an example, the information used to describe the first gesture includes a type identifier for the swipe control type, a movement direction attribute of: right, and a total number of gesture recognition points of: 3.

[0256] Optionally, the information used to describe the first gesture includes first information and second information, wherein the first information is determined based on the first gesture attributes and motion attributes, and the second information is the total number of gesture recognition points.

[0257] The first gesture attribute is used to characterize the first gesture control type to which the first gesture belongs. For example, the first gesture attribute is a type identifier for the first gesture control type. For the method of determining the first gesture attribute, please refer to the following text.

[0258] For example, the first information is used to indicate the motion type of the first gesture. For example, the first information can consist of a first gesture attribute and a motion attribute. For instance, taking a three-finger right swipe gesture as an example, the first information is a rightward swipe motion type.

[0259] Optionally, in the event of gesture recognition failure, the electronic device's first system service generates a gesture recognition result to indicate that the gesture to be recognized is invalid. For example, the gesture recognition result includes a failure message. This failure message indicates that the gesture recognition failed. The failure message can be expressed as "error" or "fail," etc.

[0260] In one example, if gesture recognition fails, the first system service in the electronic device will not respond to the gesture to be recognized. For instance, if gesture recognition fails, the contention and recognition module will not send the gesture recognition result to any application to reduce interference from invalid gesture recognition results.

[0261] In another example, in the event of gesture recognition failure, the competition and recognition module sends a set of sensing information to at least one application, which includes gesture sensing information acquired at least once in response to user actions; for a second application in the at least one application, after receiving the set of sensing information, the second application begins to execute the gesture recognition process of the application content based on the set of sensing information.

[0262] At least one of the applications includes gesture recognition logic. For example, the electronic device determines whether an application has independent gesture recognition logic. If the application has gesture recognition logic, the electronic device sends a set of sensing information to the application; if the application does not have gesture recognition logic, there is no need to send a set of sensing information to the application.

[0263] Optionally, at least one application includes an application that is running in the electronic device. For example, for a certain application, if a running process of the application exists in the CPU of the electronic device, the electronic device sends a set of sensing information to the application. The second application can be any one of the at least one applications. For example, the second application may be a third-party application among the at least one applications. The gesture recognition logic on which the second application performs gesture recognition can be designed by the developer of the second application.

[0264] The following examples illustrate the process of determining the first gesture based on gesture factors.

[0265] In some embodiments, a first system service in an electronic device obtains a first gesture based on at least one type of gesture factor, including: obtaining a first gesture attribute based on at least one type of gesture factor; obtaining at least one second gesture attribute corresponding to the first gesture control type based on at least one type of gesture factor; and obtaining the first gesture based on the first gesture attribute and at least one second gesture attribute.

[0266] The first gesture control type refers to the gesture control type to which the first gesture belongs, and the first gesture attribute is used to characterize the first gesture control type to which the first gesture belongs. For an introduction to the types of gesture control types, please refer to the above embodiment. The second gesture attribute is used to characterize the specific implementation of the first gesture. The second gesture attributes corresponding to different types of gesture control types are not entirely the same.

[0267] Optionally, the second gesture attribute includes at least one of the following: the total number of gesture recognition points and motion attributes. The motion attributes differ for different gesture control types. For example, the motion attributes for a swipe control type include a swipe direction attribute, the motion attributes for a zoom control type include a zoom trend attribute, and the motion attributes for a rotation control type include a rotation direction attribute. Please refer to the above embodiments for selectable values ​​for the motion attributes.

[0268] In some embodiments, at least one type of gesture factor includes multiple types of gesture factors with competing priorities, where the competing priorities are used to characterize the order in which each type of gesture factor participates in gesture recognition.

[0269] In some embodiments, the first system service obtains a first gesture attribute based on at least one type of gesture factor, including: the first system service obtains a target gesture factor, which is the gesture factor with the highest competition priority among the multiple types of gesture factors. Subsequently, the first system service obtains the discrimination condition corresponding to the target gesture factor, and determines whether the target gesture factor meets the corresponding discrimination condition. If the target gesture factor meets the corresponding discrimination condition, the first system service determines the first gesture attribute based on the gesture control type corresponding to the target gesture factor. If the target gesture factor does not meet the corresponding discrimination condition, the first system service updates the target gesture factor to the gesture factor with the highest competition priority among the gesture factors that have not yet participated in gesture recognition among the multiple types of gesture factors, and repeatedly executes the operation of obtaining the discrimination condition corresponding to the target gesture factor to determine whether the target gesture factor meets the corresponding discrimination condition, until it is determined that the first gesture attribute or all gesture factors among the multiple types of gestures have participated in gesture recognition.

[0270] Wherein, multiple gesture factors refer to at least two types of gesture factors that have competing priorities among at least one gesture factor. Optionally, when there are competing priorities among multiple gesture factors, the target gesture factor is the gesture factor with the highest competing priority among the multiple gesture factors. For example, the competing priority of the target gesture factor is higher than or equal to the competing priorities of other gesture factors among the multiple gesture factors. For instance, the target gesture factor is the gesture factor with the highest competing priority among the optional gesture factors.

[0271] For example, the target gesture factor is selected only once in step S330. For instance, the multiple gesture factors include gesture factor 1, gesture factor 2, and gesture factor 3. The three gesture factors are ordered from high to low according to their competition priority as follows: gesture factor 1, gesture factor 2, gesture factor 3. Since gesture factor 1 has already been selected once in step S330, and gesture factor 2 has not been selected in step S330, gesture factor 2 is determined as the target gesture factor.

[0272] This helps ensure that each gesture factor participates once in the process of determining the gesture recognition result in step S330, thereby avoiding repeated judgment of the same type of gesture factor and helping to improve the efficiency of the gesture recognition process.

[0273] Optionally, the selection counts of each gesture factor at different times are independent and do not affect each other. For the same type of gesture factor, the selection counts of the gesture factors corresponding to the same gesture recognition point obtained from gesture sensing information at different times do not accumulate.

[0274] For example, the number of times direction factor 1 is selected does not affect the number of times direction factor 2 is selected. Direction factor 1 is obtained based on the gesture sensing information at time 1, and direction factor 2 is determined based on the gesture sensing information at time 2, with time 2 lagging behind time 1.

[0275] The determination criteria corresponding to the target gesture factor are used to determine whether the gesture parameters represented by the target gesture factor meet the conditions. In this embodiment, the determination criteria are also called factor determination criteria. Optionally, when the total number of types of multiple gesture factors is greater than or equal to 2, there are at least two gesture factors corresponding to different determination criteria.

[0276] The determination criteria in this application utilize the similarities and differences across various gesture factors to distinguish different gesture control types. For example, a certain gesture control type shares the same characteristics as other gesture control types on the first gesture factor, but differs from the characteristics of other gesture control types on the first gesture factor. Through the determination criteria corresponding to the first gesture factor, the gesture recognition process can identify this gesture control type.

[0277] In some embodiments, a first system service in an electronic device obtains a first gesture attribute based on the gesture control type corresponding to the target gesture factor, including: the first system service obtaining a first gesture attribute used to characterize the gesture control type corresponding to the target gesture factor.

[0278] The gesture control type corresponding to the target gesture factor is related to the type of the target gesture factor. Optionally, different types of gesture factors correspond to different gesture control types. For example, the gesture control type corresponding to the target gesture factor is preset in the gesture recognition algorithm.

[0279] In some embodiments, the multiple gesture factors include a direction factor and a center of gravity factor. Optionally, the competition priority of the direction factor is higher than that of the center of gravity factor. For example, the gesture control type corresponding to the direction factor is a swipe control type, and the gesture control type corresponding to the center of gravity factor is a rotation control type. For example, the gesture control type corresponding to the direction factor is a swipe control type, and the gesture control type corresponding to the center of gravity factor is a zoom control type.

[0280] For example, if the target gesture factor is a direction factor, and the gesture parameters described by the direction factor satisfy the judgment conditions corresponding to the direction factor, then the first gesture attribute is obtained according to the gesture control type corresponding to the direction factor. In this case, the first gesture attribute is used to characterize the sliding control type.

[0281] For example, the competition priority of the direction factor is higher than that of the center of gravity factor, the gesture control type corresponding to the direction factor is the swipe control type, and the gesture control type corresponding to the center of gravity factor is the zoom control type.

[0282] Optionally, the competition priority of the center of gravity factor is higher than that of the direction factor, the gesture control type corresponding to the direction factor is the sliding control type, and the gesture control type corresponding to the center of gravity factor is the rotation control type.

[0283] In some embodiments, the plurality of gesture factors include a direction factor and a center of gravity factor, wherein the competition priority of the direction factor is higher than that of the center of gravity factor.

[0284] Optionally, the competition priority of a gesture factor is positively correlated with the type of gesture control associated with that gesture factor during gesture recognition. For example, when the competition priority of the direction factor is set higher than that of the center of gravity factor, the judgment condition corresponding to the direction factor can determine whether the first gesture control type belongs to a swipe control type or a non-swipe control type. That is, regardless of whether the gesture belongs to a swipe control type or a non-swipe control type, the direction factor must be used during gesture recognition. Therefore, the gesture control types associated with the direction factor include: swipe control type and non-swipe control type (such as zoom control type and rotation control type).

[0285] The judgment condition corresponding to the center of gravity factor is used to identify which type of non-slide control the first gesture control belongs to. Therefore, the gesture control types associated with the center of gravity factor include: scaling control type and rotation control type. It can be seen that the direction factor is associated with 3 types of gesture control types, while the center of gravity factor is associated with 2 types. Therefore, the competition priority of the direction factor is higher than that of the center of gravity factor.

[0286] As can be seen from the above embodiments regarding the method for determining gesture factors, the computational load in the direction factor acquisition process is smaller than that in the center-of-gravity factor acquisition process. Setting the competition priority of the direction factor higher than that of the center-of-gravity factor ensures that for a first gesture of the sliding control type, after determining the direction factor, the first gesture control type can be identified as sliding control type based on the corresponding judgment conditions. Therefore, in this competition priority setting method, for a first gesture belonging to the sliding control type, it is not necessary to calculate the center-of-gravity factor based on gesture sensing information, thereby helping to reduce the computational load of gesture recognition and improve the efficiency of the gesture device.

[0287] The following describes the judgment conditions corresponding to each type of gesture factor.

[0288] Optionally, the determination condition corresponding to the direction factor is: there are multiple gesture recognition points greater than or equal to a first number, and the direction factors corresponding to these multiple gesture recognition points are the same; the gesture control type corresponding to the direction factor is: swipe control type. For example, when the direction factors corresponding to the first number of gesture recognition points are the same, the first system service in the electronic device obtains a first gesture attribute used to describe the swipe control type; when the first number of direction factors are the same, the first system service determines that the gesture to be recognized does not belong to the swipe control type.

[0289] The first quantity is related to the total number of gesture recognition points of the gesture to be recognized. The first system service in the electronic device determines the first quantity through a quantity factor of the gesture to be recognized. For example, the first quantity is greater than or equal to 1 / g times the total number of gesture recognition points, and less than or equal to the total number of gesture recognition points, where g is a positive number greater than 1. For instance, the first quantity is equal to the total number of gesture recognition points of the gesture to be recognized.

[0290] In one example, for a certain gesture to be recognized, the gesture includes 3 gesture recognition points, and the direction factor corresponding to each of the 3 gesture recognition points is left. The 3 direction factors are the same. Then the first system service determines that the gesture parameter corresponding to the direction factor satisfies the corresponding factor determination condition of the direction factor.

[0291] Optionally, the determination condition corresponding to the center of gravity factor is: there is a second number of gesture recognition points, and for each gesture recognition point in the second number of gesture recognition points, the change in the center of gravity factor corresponding to the gesture recognition point is greater than or equal to the first distance threshold; the gesture control type corresponding to the center of gravity factor is the scaling control type.

[0292] For example, when there are a second number of gesture recognition points, and for each of these second number of gesture recognition points, the change in the centroid factor corresponding to that gesture recognition point is greater than or equal to a first distance threshold, the first system service in the electronic device acquires a first gesture attribute characterizing the scaling control type. When there are a fourth number of gesture recognition points, and for each of these fourth number of gesture recognition points, the change in the centroid factor corresponding to that gesture recognition point is greater than or equal to the first distance threshold, the electronic device determines that the gesture to be recognized does not belong to the scaling control type. For example, when the centroid factor does not meet its corresponding determination condition, the electronic device acquires a first gesture attribute characterizing the rotation control type.

[0293] Among them, the fourth quantity is less than the second quantity (that is, there is no second quantity of gesture recognition points), the second quantity is a positive integer, and the third quantity is a natural number.

[0294] Specifically, for any gesture recognition point among at least one gesture recognition point, the change in the centroid factor is used to characterize the change in distance between the gesture recognition point and the centroid of the geometric shape. For example, for a given gesture recognition point, the change in the centroid factor is related to two factors: the centroid factor of the gesture recognition point at a first time moment and the centroid factor of the gesture recognition point at a second time moment. The centroid factor at the first time moment is obtained based on the gesture sensing information at the first time moment, and the centroid factor at the second time moment is obtained based on the gesture sensing information at the second time moment.

[0295] For example, the change in the centroid factor is equal to the centroid factor at the second time step minus the centroid factor at the first time step. For an explanation of the first time step, the second time step, and the centroid factor, please refer to the description of step S320.

[0296] The second quantity is related to the total number of gesture recognition points. Optionally, the electronic device determines the second quantity by a factor of the number of gestures to be recognized. For example, the second quantity is greater than or equal to 1 / h times the total number of gesture recognition points, and less than or equal to the total number of gesture recognition points, where h is a positive number greater than 1. For example, the second quantity is equal to the first quantity, or the second quantity is unrelated to the first quantity.

[0297] Optionally, the determination condition for the center of gravity factor is: there exists a third number of gesture recognition points, and for each of these third number of gesture recognition points, the change in the center of gravity factor is less than or equal to a second distance threshold. In this case, the gesture control type corresponding to the center of gravity factor is a rotation control type.

[0298] The third quantity is related to the total number of gesture recognition points corresponding to the gesture to be recognized. For example, the third quantity is greater than or equal to half the total number of gesture recognition points. For instance, the second quantity equals the third quantity. The second distance threshold is preset. For instance, the second distance threshold equals the first distance threshold, or the second distance threshold is any positive number unrelated to the first distance threshold.

[0299] For example, if a third number of gesture recognition points exist, and the change in the centroid factor of each of these third number of gesture recognition points is less than or equal to a second distance threshold, then the electronic device determines that the gesture to be recognized belongs to the rotation control type. If the third number of centroid factors does not exist, then the first system service obtains a first gesture attribute to characterize the rotation control type. For example, if the centroid factor does not meet its corresponding determination condition, the electronic device determines that the gesture to be recognized does not belong to the scaling control type.

[0300] In some embodiments, for a target gesture factor among multiple types of gesture factors, if the target gesture factor meets its corresponding determination condition, the first system service in the electronic device will determine the gesture attribute used to characterize the gesture control type corresponding to the target gesture factor as the first gesture attribute. If the target gesture factor of the gesture to be recognized does not meet its corresponding determination condition, the first system service will update the target gesture factor according to the multiple types of gesture factors, or re-execute step S310, or determine that the gesture recognition process has failed.

[0301] In one possible implementation, a first system service in an electronic device obtains a first gesture attribute based on at least one type of gesture factor, including: the first system service obtaining a first gesture factor from multiple types of gesture factors, wherein the first gesture factor is the gesture factor with the highest priority among the multiple types of gesture factors; and, if the first gesture factor satisfies the corresponding determination condition, the first system service determines the first gesture attribute based on the gesture control type corresponding to the first gesture factor.

[0302] The determined first gesture attribute is used to characterize the gesture control type corresponding to the first gesture factor.

[0303] For example, the first gesture factor is the one with the highest competition priority among the multiple gesture factors. For instance, if the multiple gesture factors include a direction factor and a center of gravity factor, and the direction factor has a higher competition priority than the center of gravity factor, then the first gesture factor is the direction factor.

[0304] In another possible implementation, a first system service in an electronic device obtains a first gesture attribute based on at least one type of gesture factor, including: the first system service obtaining a first gesture factor from multiple types of gesture factors, wherein the first gesture factor is the gesture factor with the highest competitive priority among the multiple types of gesture factors; if the first gesture factor does not meet the corresponding determination condition, the first system service obtaining a second gesture factor from multiple types of gesture factors, wherein the second gesture factor is the gesture factor with the highest competitive priority among the gesture factors that did not participate in gesture recognition among the multiple types of gesture factors; if the second gesture factor meets the corresponding determination condition, the first system service determines the first gesture attribute based on the gesture control type corresponding to the second gesture factor.

[0305] The determined first gesture attribute is used to characterize the gesture control type corresponding to the second gesture factor.

[0306] Optionally, the competition priority of the second gesture factor is lower than that of the first gesture factor. In one example, the multiple gesture factors include a direction factor and a center of gravity factor. The competition priority of the direction factor is higher than that of the center of gravity factor. If the direction factor has been selected once, then the second gesture factor is the center of gravity factor.

[0307] Optionally, the electronic device may also repeatedly execute steps 320 and 330. During each execution of step 320, the electronic device determines a subset of gesture factors from at least one type of gesture factors based on gesture sensing information. For example, during the execution of step 320, the electronic device determines the gesture factor with the higher competition priority based on the gesture sensing information according to competition priority; subsequently, the electronic device executes step S330; if step 330 cannot identify the gesture control type of the gesture to be recognized based on the gesture factor, the electronic device reselects a new gesture factor according to competition priority, and the electronic device executes steps S320 and S330. The electronic device repeatedly executes steps S320 and S330 in the above manner until the recognition of the gesture to be recognized is completed, or the loop exceeds the maximum number of loops.

[0308] Because there is competition for priority among multiple types of gesture factors, if the type of gesture to be recognized can be determined by the gesture factor with higher competition priority, then other gesture factors with relatively lower competition priority will not participate in the gesture recognition process. Therefore, executing step S320 in stages, calculating at least some gesture factors from one type of gesture factor each time, helps to ensure that all calculated gesture factors can participate in the gesture recognition process, thereby avoiding the situation where all gesture factors are calculated at once, but some gesture factors do not participate in gesture recognition.

[0309] This approach helps reduce the total number of gesture factors that need to be calculated, thereby improving the computational efficiency of electronic devices in the gesture recognition process. Since gesture factors with higher competition priority among at least one type participate in the gesture recognition process first, during the phased execution of step S320, the gesture factors with higher competition priority are calculated first. This not only does not affect the normal progress of the gesture recognition process, but also helps avoid calculating useless gesture factors, thus improving the gesture recognition efficiency of electronic devices and increasing the response speed of gesture interaction.

[0310] In some embodiments, before acquiring the first gesture based on the first gesture attribute and at least one second gesture attribute, the first system service in the electronic device further includes: acquiring a third gesture factor from at least one type of gesture factors; and, if the third gesture factor satisfies a validity condition, acquiring the first gesture attribute according to the gesture control type corresponding to the selected gesture factor. Optionally, if the third gesture factor does not satisfy the validity condition, the first system service repeatedly executes the step of acquiring gesture sensing information in response to a user operation until the first gesture is acquired, or the user operation disappears.

[0311] The third gesture factor is used to verify the validity of the user's operation. Optionally, the third gesture factor can be used to verify whether the user's operation is intended to implement gesture control. For example, the type of the third gesture factor includes at least one of the following: speed factor, distance factor.

[0312] In one example, the validity condition for the velocity factor is that the velocity factor of at least one gesture recognition point is greater than or equal to a velocity threshold, and the validity condition for the distance factor is that the distance factor of at least one gesture recognition point is greater than or equal to a second distance threshold.

[0313] Here, the selected gesture factor refers to the gesture factor selected from multiple gesture factors during the process of obtaining the first gesture attribute based on multiple gesture factors. For example, the selected gesture factor is the currently selected target gesture factor. For example, the selected gesture factor includes the first gesture factor and the second gesture factor in the above embodiments.

[0314] Optionally, if the velocity factor is greater than or equal to a velocity threshold and the distance factor is greater than or equal to a second distance threshold, the first system service in the electronic device continues to execute the step of acquiring the first gesture based on the first gesture attribute and at least one second gesture attribute; if the velocity factor is less than the velocity threshold, or the distance factor is less than the second distance threshold, it indicates that the user's operation has not resulted in movement within a certain period of time, or the degree of movement is negligible. In this case, the first system service acquires the gesture sensing information collected at the next detection moment.

[0315] If the third gesture factor meets the validity condition, the first gesture attribute is obtained according to the gesture control type corresponding to the selected gesture factor, including: obtaining the first gesture attribute used to characterize the gesture control type corresponding to the selected gesture factor.

[0316] The validity of user actions is verified through a third gesture factor, allowing gesture recognition to proceed only when the user's action involves a certain displacement or speed. This third gesture factor filters out invalid user actions with small movements, thus improving the accuracy of the gesture recognition process. Furthermore, this method helps prevent electronic devices from misinterpreting user touches as user actions, thereby avoiding situations where the first application is notified to perform an action when the user does not need to, leading to abnormal operation of the electronic device.

[0317] In some embodiments, a first system service in an electronic device acquires a first gesture based on a first gesture attribute and at least one second gesture attribute, including: during the continuous operation of a user, the first system service continuously acquires n predicted gestures based on the corresponding first gesture attribute and at least one second gesture attribute, where n is a positive integer; if the n predicted gestures meet an accuracy condition, the first system service acquires the first gesture based on the predicted gestures.

[0318] The corresponding first gesture attribute and at least one second gesture attribute are determined by the same gesture sensing information. The corresponding first gesture attribute and at least one second gesture attribute can determine a predicted gesture.

[0319] Here, the n predicted gestures represent the predicted gestures determined based on each of the n gesture sensing information before the first gesture is determined.

[0320] Optionally, in determining the first gesture, the first system service acquires gesture sensing information every first time interval during the continuous user operation. Before determining the first gesture, for each sensed gesture sensing information, the first system service acquires at least one gesture factor based on the gesture sensing information. Subsequently, the first system service acquires a first gesture attribute and at least one second gesture attribute based on the at least one gesture factor. The first gesture attribute and at least one second gesture attribute mentioned here are corresponding first gesture attributes and at least one second gesture attributes. The first system service acquires a predicted gesture based on the corresponding first gesture attribute and at least one second gesture attribute. The predicted gesture is a specific gesture.

[0321] For example, the predicted gesture is a gesture subscribed to by an application. For instance, if the corresponding first gesture attribute and at least one second gesture attribute indicate: zoom control type, pinch, and 3 fingers, then the 3-finger pinch is the predicted gesture determined in this instance.

[0322] For example, n predicted gestures are obtained based on n consecutively collected gesture sensing information. For example, the consecutively collected n gesture sensing information refers to the n gesture sensing information collected at each first time interval.

[0323] The accuracy condition relates to the consistency among the n predicted gestures. Optionally, the accuracy condition includes that the n predicted gestures include m identical predicted gestures, where m is a positive integer less than or equal to n. For example, the m identical predicted gestures are obtained based on m consecutively acquired gesture sensing information. For example, the m predicted gestures are the most recently determined consecutive m predicted gestures among the n predicted gestures. m is a preset positive integer, such as m equal to 3, 4, 5, 6, etc.

[0324] For example, if m of the n predicted gestures are the same, the first system service determines any one of the m predicted gestures as the first gesture; if m of the n predicted gestures are not the same, the first system service re-executes the steps of obtaining gesture sensing information and obtaining the predicted gesture based on the gesture sensing information until the first gesture is determined or the user operation disappears.

[0325] In one example, during the duration of a user operation, the first system service determines a predicted gesture based on a first gesture attribute and a second gesture attribute; the first system service updates the prediction count of the predicted gesture; if the prediction count is less than a duration threshold, the first system service repeats the step of acquiring gesture sensing information in response to the user operation until the first gesture is acquired or the user operation disappears. If the prediction count equals the duration threshold, the first system service determines the predicted gesture as the first gesture.

[0326] The prediction count is used to characterize the number of times the same predicted gesture occurs during the continuous user operation. The prediction count represents the cumulative value of gestures continuously identified as the same gesture during multiple executions of step S330. Optionally, after acquiring a predicted gesture, the first system service increments the prediction count of that gesture by 1. For example, if the prediction count of the predicted gesture is less than the duration threshold, the first system service resets the prediction counts of other predicted gestures besides that gesture.

[0327] For example, the first gesture is the subscription gesture that first reaches the duration threshold among at least one subscription gesture.

[0328] Optionally, the duration threshold is preset. For example, the duration threshold can be 3, 4, 5, 6, etc. By setting the duration threshold, fluctuations in the movement trajectory of the gesture to be recognized can be avoided, which could cause the electronic device to match the gesture to be recognized with an incorrect gesture, thus leading to errors in gesture control.

[0329] In one example, during the gesture recognition process, the electronic device executes step S330 three times; the first execution of step S330 determines the first gesture as swipe gesture 1, the second execution of step S330 determines the first gesture as zoom gesture 1, and the third execution of step S330 determines the first gesture as zoom gesture 1; after the three executions of step S330 are completed, the prediction count for swipe gesture 1 is equal to 1, and the prediction count for zoom gesture 1 is equal to 2.

[0330] This embodiment ensures that the gestures recognized continuously based on gesture sensing information are consistent by setting and updating the prediction count, thereby helping to eliminate interference caused by user tremors in gesture recognition and improving the accuracy of gesture recognition.

[0331] In some embodiments, taking touchscreen gestures as an example, the gesture to be recognized needs to go through at least three types of events from start to finish: a Down event, a Move event, and an Up event. The Down event describes the gesture to be recognized contacting the sensing device (such as a touchscreen) of the electronic device; the Move event describes the movement of the gesture on the touchscreen; and the Up event describes the gesture leaving the touchscreen. Optionally, step S310 is performed during the Down event and / or the Move event, and steps S320 and S330 are performed during the Move event. For example, each Move event corresponds to one gesture raw event; that is, in each Move event, the first system service acquires gesture sensing information once and determines the corresponding predicted gesture based on the acquired gesture sensing information. If the predicted gestures corresponding to multiple Move events are consistent, the first system service determines the predicted gesture as the first gesture.

[0332] In this embodiment, when a gesture to be recognized is detected, the electronic device performs a gesture recognition step during the Down and Move events. This helps to determine the gesture recognition result before the Up event, which helps to quickly determine the gesture to be recognized, thereby improving the response speed of the electronic device to the gesture to be recognized.

[0333] The following example illustrates the logic diagram of at least one gesture recognition method. In this embodiment, at least one type of gesture factor competes for priority, and the gesture factors are arranged from highest to lowest priority as follows: quantity factor, direction factor, and center of gravity factor. For example... Figure 9 As shown, the gesture recognition method includes at least the following steps, and the execution of these steps is performed by the first system service in the electronic device.

[0334] In step S910, after detecting the Down event of the gesture to be recognized, the first system service acquires and stores the gesture sensing information at a second moment. The second moment can be the initial moment after the gesture to be recognized is detected. Optionally, the gesture sensing information includes the location information of at least one gesture recognition point.

[0335] Optionally, the electronic device acquires gesture sensing information in response to user operation.

[0336] In step S920, the first system service obtains and stores the center of gravity factor at the second moment based on the gesture sensing information at the second moment. Please refer to the above embodiment for the method of obtaining the center of gravity factor.

[0337] In step S930, after detecting the Move event of the gesture of the device to be used, the electronic device acquires and stores the gesture sensing information at the first moment.

[0338] Step S940: The first system service detects whether the gesture sensing information meets the competition recognition conditions.

[0339] Optionally, the competition recognition condition is that the functional gesture that matches the gesture to be recognized is not determined, and the gesture to be recognized moves. For example, if the functional gesture that matches the gesture to be recognized is not determined, and the gesture to be recognized moves, the first system service executes step S950. For example, if the gesture to be recognized does not move, the first system service executes step S930; if the functional gesture that matches the gesture to be recognized is determined, the first system service determines that gesture recognition is complete.

[0340] In one example, once a first gesture matching the gesture to be recognized has been determined, the first system service obtains updated gesture sensing information; based on the updated gesture sensing information, the first system service obtains the motion parameters of the first gesture. The motion parameters of the first gesture are used to indicate the motion of the first gesture; please refer to the description of S350 below for details.

[0341] Step S950: The first system service obtains the direction factor at the first moment based on the gesture sensing information at the first moment.

[0342] Optionally, the first system service obtains the orientation factor at the first moment based on the gesture sensing information and the reference sensing information at the first moment. For example, the reference sensing information is the gesture sensing information at the second moment. The reference sensing information includes location reference information used to characterize the historical position of the gesture recognition point.

[0343] Optionally, during step S950, the first system service also obtains the quantity factor and speed factor at the first moment based on the gesture sensing information at the first moment.

[0344] Step S960: The first system service determines whether the direction factor at the first moment meets the determination condition corresponding to the direction factor.

[0345] Optionally, if the direction factor at the first moment meets the determination condition, the first system service executes step S970; if the direction factor at the first moment does not meet the determination condition, the first system service executes step S980.

[0346] Optionally, before executing step S960, the first system service first determines the type of multi-finger gesture operation to be recognized based on a quantity factor; after determining the type of multi-finger gesture operation to which the gesture belongs, the first system service executes step S960. For example, when the quantity factor is 4, the first system service determines that the gesture to be recognized belongs to a four-finger gesture operation. As another example, when the quantity factor is 3, the first system service determines that the gesture to be recognized belongs to a three-finger gesture operation.

[0347] Figure 10 As shown, when the gesture to be recognized is a swipe control gesture, the movement direction of the gesture recognition point remains basically consistent throughout the duration of the gesture. Therefore, gestures belonging to the swipe control type can be identified by using the judgment conditions corresponding to the swipe factor.

[0348] Step S970: The first system service obtains a first gesture attribute that describes the type of swipe control.

[0349] Optionally, during the execution of step S970, after determining that the direction factor meets its corresponding judgment condition, the first system service obtains the speed factor and / or distance factor at a first moment; if the speed factor at the first moment is greater than or equal to a speed threshold, and / or the distance factor at the first moment is greater than or equal to a second distance threshold, the first system service determines that the first gesture control type is a sliding control type; otherwise, the first system service re-executes step S930. For example, the speed threshold and the second distance threshold are preset.

[0350] Because user actions are not always smooth, jitter in the gestures being recognized can easily provide interference to the gesture recognition process, leading to errors. Setting speed and distance thresholds can help eliminate interference data caused by jitter, thus improving the accuracy of gesture recognition.

[0351] In step S980, the first system service obtains the center of gravity factor at the first moment based on the gesture sensing information at the first moment.

[0352] Step S990: The first system service determines the change in the center of gravity factor based on the center of gravity factor of each gesture recognition point at the first moment and the center of gravity factor at the second moment.

[0353] Step S991: The first system service determines whether the centroid factor at the first moment meets the judgment condition corresponding to the centroid factor.

[0354] Optionally, the determination condition corresponding to the centroid factor is that there is a second number of gesture recognition points, and for each gesture recognition point in the second number of gesture recognition points, the change in the centroid factor corresponding to the gesture recognition point is greater than or equal to the first distance threshold. Optionally, if the centroid factor at the first moment satisfies the determination condition corresponding to the centroid factor, the electronic device executes step S992; if the centroid factor at the first moment does not satisfy the determination condition corresponding to the centroid factor, the electronic device executes step S993.

[0355] Please refer to Figure 11 and Figure 12 Compare zoom control gestures with rotation control gestures. For example... Figure 11 As shown, when the gesture belongs to the opening control type, the center of gravity of the geometric shape remains basically unchanged during the user's operation, while the distance between the gesture recognition point and the center of gravity gradually increases. The gesture to be recognized (belonging to the scaling control type, such as opening or pinching gestures) can be identified by using the factor determination conditions corresponding to the center of gravity factor. Figure 12 As shown, when the gesture to be recognized belongs to the rotation control type, the center of gravity of the geometric shape remains basically unchanged during the continuous operation of the user, and the distance between the gesture recognition point and the center of gravity also remains basically unchanged.

[0356] Step S992, the first system service obtains a first gesture attribute to characterize the zoom control type.

[0357] Optionally, in step S992, the first system service determines whether the first type is a pinch gesture or an open gesture based on the direction of change of the center of gravity factor. For example, if the direction of change of the center of gravity factor is an increase in value, the first system service determines that the first type is an open gesture; if the direction of change of the center of gravity factor is a decrease in value, the electronic device determines that the first type is a pinch gesture.

[0358] Step S993: The first system service obtains a first gesture attribute to characterize the type of rotation control.

[0359] Optionally, in step S993, the electronic device determines whether the first type is a clockwise rotation gesture or a counterclockwise rotation gesture based on the orientation factor.

[0360] Step S994: ​​The first system service obtains at least one second gesture attribute based on at least one gesture factor.

[0361] Optionally, at least one second attribute includes the total number of gesture recognition points and a motion attribute. For example, the electronic device determines the total number of gesture recognition points based on a quantity factor among at least one gesture factor.

[0362] For example, when the first gesture attribute is used to characterize the sliding control type, the electronic device obtains the sliding direction attribute based on the direction factor. For example, when the first gesture attribute is used to characterize the scaling control type, the electronic device obtains the scaling trend attribute based on the change in the center of gravity factor. For example, when the first gesture attribute is used to characterize the rotation control type, the electronic device obtains the rotation direction attribute based on the direction factor.

[0363] In step S995, the first system service obtains the predicted gesture based on the first gesture attribute and at least one second gesture attribute.

[0364] Optionally, the first system service combines the first gesture attribute and at least one second gesture attribute to obtain the predicted gesture.

[0365] In step S996, the first system service determines whether the predicted number of times the predicted gesture is equal to the duration threshold. Optionally, if the predicted number of times the first gesture is equal to the duration threshold, the first system service determines the predicted gesture as the first gesture; if the predicted number of times the first gesture is less than the duration threshold, the first system service re-executes step S930.

[0366] Optionally, before step S996, the first system service increments the prediction count of the predicted gesture by one. If, in step S996, it is determined that the prediction count of the predicted gesture is less than the duration threshold, the first system service resets the prediction count of other predicted gestures.

[0367] For example, before the first system service re-executes step S930, the first system service updates the first moment and the second moment. For instance, the first system service uses the first moment as the new second moment and the next moment for acquiring gesture sensing information as the new first moment.

[0368] Step S997: After detecting the Up event of the gesture to be recognized, the first system service cleans up the data required during the gesture recognition process.

[0369] Optionally, the data required during gesture recognition includes at least one of the following: gesture sensing information at at least one moment, and gesture factors at at least one moment.

[0370] This embodiment determines at least one type of gesture factor through gesture sensing information, and then determines the gesture attribute to which the gesture to be recognized belongs based on at least one type of gesture factor, thereby obtaining the first gesture. By setting a contention priority, the gesture recognition logic for different gestures is unified, which facilitates subsequent expansion of the gesture recognition method. Furthermore, unifying various gestures into a single gesture recognition logic allows the electronic device to complete gesture recognition through a unified system service (such as a contention and recognition module) after detecting the gesture to be recognized. Compared to each application performing gesture recognition independently, this embodiment helps avoid conflicts between applications during the gesture recognition process.

[0371] In this embodiment, all gesture factors required for gesture recognition are not acquired all at once, but rather in multiple stages. For a given gesture factor, it is acquired through gesture sensing information before it is needed. This approach helps reduce the number of gesture factors that the electronic device needs to determine without affecting the gesture recognition process, thereby reducing the computational burden on the electronic device.

[0372] Furthermore, by setting a duration threshold in this embodiment, interference from incidentally detected data during the continuous execution of the gesture can be avoided, thus improving the reliability of the gesture recognition results.

[0373] In some embodiments, step S330 is performed by a competition and recognition module in the electronic device. In step S330, the competition and recognition module determines a first gesture factor from multiple types of gesture factors; if the first gesture factor meets the corresponding judgment condition, the competition and recognition module determines a first gesture attribute according to the gesture control type corresponding to the first gesture factor, and determines at least one second gesture attribute according to at least one type of gesture factor; and obtains a first gesture according to the first gesture attribute and the second gesture attribute; if the first gesture factor does not meet the corresponding judgment condition, the competition and recognition module determines a new first gesture factor from multiple types of gesture factors, or, if at least one type of gesture factor has been selected, the competition and recognition module executes step S310, or determines that the gesture recognition process has failed.

[0374] In step S340, the first system service reports information describing the first gesture to the first application so that the first application can perform the first operation corresponding to the first gesture.

[0375] The first application is the application that subscribes to the first gesture. Please refer to the above embodiment for the gesture subscription process.

[0376] In some embodiments, the first application is an application in the electronic device that supports the first gesture. For example, if the first gesture is a right swipe gesture, and application 1 has a right swipe gesture function, then application 1 can be used as the first application. Optionally, the first application is an application displayed in the user interface of the electronic device, or an application running in the background. Exemplarily, the electronic device determines the first application through a gesture mapping table; for details of the gesture mapping table, please refer to the embodiments described above.

[0377] Optionally, the first application is the application that subscribes to the first gesture. For example, before step S340, the first application notifies the first system service of the subscription information in order to subscribe to the first gesture from the first system service.

[0378] Optionally, if there are multiple candidate applications subscribing to the first gesture installed on the electronic device, the first system service selects one of the multiple candidate applications as the first application. In one example, the electronic device includes an application priority list, which indicates the order in which gesture recognition results are sent to applications on the electronic device; the first system service selects the candidate application with the highest application priority from the multiple candidate applications according to the priority list, as the first application.

[0379] In some embodiments, after receiving subscription information from an application, the first system service determines whether there is an overlapping gesture in the gesture mapping table based on the at least one gesture supported by the application according to the subscription information. The overlapping gesture is a functional gesture supported by at least two applications. If there is an overlapping gesture, the first system service sets an application priority among the at least two applications that support the overlapping gesture and stores the application priority in a priority list.

[0380] Optionally, the first operation refers to the gesture control triggered by the first gesture. The first operation includes, but is not limited to, at least one of the following: refreshing the displayed content, switching the displayed page, adjusting the brightness, adjusting the volume, adjusting the video playback progress, calling up a background application, moving an application to the background, adjusting the display size of the displayed content (such as enlarging or shrinking an image or text), adjusting the camera's shooting parameters, cropping the displayed content, calling system tools (such as accessing the desktop layout tool by pinching the user), adjusting the display angle of the displayed content, etc.

[0381] In this embodiment, the system service notifies the applications that have subscribed to the first gesture to perform the corresponding operation, without the system service notifying other applications in the electronic device of the first gesture, thus avoiding interference from gesture recognition to other applications.

[0382] The following example illustrates the various modules within an electronic device and their interaction flow in the gesture recognition process. Figure 13As shown, this embodiment mainly includes the following steps:

[0383] S1310, the first application sends subscription information to the competition and identification module.

[0384] Optionally, the subscription information is used to indicate at least one gesture supported by the first application. For example, for each gesture supported by the first application, the subscription information includes a type identifier of the gesture control type to which the gesture belongs and attribute parameters of the gesture.

[0385] S1320, the contention and identification module sends the subscription results to the first application.

[0386] S1330: After the user makes a gesture to be recognized, the gesture sensing module responds to the user's operation and acquires gesture sensing information.

[0387] S1340, the gesture sensing module sends gesture sensing information to the contention and recognition module.

[0388] S1350, the competition and recognition module acquires the first gesture based on the gesture sensing information.

[0389] S1360, the contention and recognition module reports information describing the first gesture to the first application.

[0390] S1370, the first application performs a first operation corresponding to the first gesture based on the information used to describe the first gesture.

[0391] In this embodiment, the competition and recognition module takes over the gesture recognition requirements and obtains the gesture recognition result based on the gesture sensing information. On the one hand, compared to sending the gesture sensing information to each application and having the application complete the gesture recognition itself based on the gesture sensing information, this embodiment helps to avoid the phenomenon of multiple applications simultaneously performing gesture recognition on the same gesture to be recognized, and helps to solve the duplication and conflict in the gesture recognition process of multiple applications.

[0392] On the other hand, the contention and recognition module sends gesture recognition results to the application only when the application supports gestures that indicate gesture recognition results. It does not send gesture recognition results to the application when the application does not support gestures that indicate gesture recognition results. This helps reduce the impact of gesture recognition results on unrelated applications. The contention and recognition module separates application and user gesture operations, thereby reducing interference from unrelated gestures on the application.

[0393] In some embodiments, the gesture recognition process provided in this embodiment may further include the following steps:

[0394] In step S350, if the user operation has not ended, the first system service in the electronic device responds to the user operation and obtains the updated gesture sensing information.

[0395] Optionally, after determining the first gesture, the electronic device detects whether the user operation has ended; if the user operation has not ended, the electronic device acquires updated gesture sensing information. If the user operation has ended, it is not necessary to acquire gesture sensing information again. For example, if at least one gesture recognition point has not disappeared, the electronic device acquires updated gesture sensing information.

[0396] For example, after acquiring the first gesture, if the gesture to be recognized has not ended (e.g., the gesture recognition point has not disappeared), the gesture sensing module in the electronic device acquires the gesture sensing information at the third moment (i.e., the updated gesture sensing information).

[0397] For example, the third time step lags behind the first time step in step 310 when acquiring gesture sensing information, and there is a difference between the gesture sensing information at the third time step and the gesture sensing information at the first time step. For example, the first time step and the third time step differ by one first time interval. For example, the time when the gesture sensing module acquires gesture sensing information for the jth time is the first time step, and the time when it acquires gesture sensing information for the (j+1)th time step is the third time step, where j is a natural number.

[0398] Optionally, step S350 is executed after step S340. After sending the gesture recognition result to the first application, the electronic device determines whether the gesture to be recognized meets the follow-up condition. If the gesture to be recognized meets the follow-up condition, the electronic device executes step S350; if the gesture to be recognized does not meet the follow-up condition, the gesture recognition process is completed.

[0399] For example, the follow-up condition is related to the first gesture. For instance, the follow-up condition includes: the first gesture is a follow-up gesture. Here, a follow-up gesture refers to a gesture whose movement or change determines the effect of a control event. In one example, if the first gesture is a follow-up gesture, the electronic device executes S350; if the first gesture is not a follow-up gesture, the electronic device does not execute S350.

[0400] For example, the follow-up condition includes at least one of the following: the first gesture is a follow-up gesture, and the gesture recognition point has not disappeared. Here, the follow-up gesture is a gesture whose effect changes with the user's operation.

[0401] In one example, the follow-up condition can be that the first gesture is a follow-up gesture and the gesture recognition point has not disappeared. If the first gesture is a follow-up gesture and the gesture recognition point has not disappeared, the electronic device executes step S340 and then executes step S350; if the first gesture is not a follow-up gesture or the gesture recognition point has disappeared, step S350 is not executed.

[0402] In step S360, the first system service in the electronic device obtains the motion parameters of the first gesture based on the updated gesture sensing information.

[0403] Optionally, the motion parameters of the first gesture are used to indicate the effect of performing the first operation. Different motion parameters of the first gesture can enable different gesture control operations in the first operation. For example, the motion parameters of the first gesture are also called effect control information. The effect control information can be used to control the fineness of the implementation of the first operation.

[0404] Optionally, the motion parameters of the first gesture are related to the gesture control type to which the first gesture belongs. For example, the motion parameters of the first gesture may include different parameters for different gesture control types.

[0405] In some embodiments, when the first gesture is a sliding control type, the motion parameters of the first gesture are used to indicate changes in the sliding distance; when the first gesture is a scaling control type, the motion parameters of the first gesture are used to indicate changes in the scaling ratio; when the first gesture is a rotation control type, the operation parameters of the first gesture are used to indicate changes in the rotation angle.

[0406] The change in sliding distance is represented by a unit sliding distance. The unit sliding distance indicates the sliding distance of a swipe-controlled gesture within a unit of time. For example, unit time refers to the time interval for acquiring gesture sensing information, that is, the first time interval in the above embodiment.

[0407] The scaling ratio represents the ratio between the intended display size of the content and a reference display size. For example, the reference display size is the display size of the content before the user operation begins, or the display size of the content before a first time interval. For zoom gestures, the effect control information includes a change in unit distance. This change in unit distance represents the change in distance between the gesture recognition point and the center of gravity within a unit of time.

[0408] The change in rotation angle is used to characterize the angle between the intended display direction of the displayed content and the reference display direction. For example, for a rotation gesture, the effect control information includes a unit rotation angle. This unit rotation angle characterizes the angle of rotation per unit time between the gesture recognition point and the center of gravity, with the center of gravity as the center of rotation.

[0409] In some embodiments, the electronic device obtains the gesture factor at the third moment based on the gesture sensing information at the third moment; the electronic device obtains the motion parameter effect control information of the first gesture based on the gesture factor at the third moment.

[0410] Optionally, if the first gesture is a sliding control type, the electronic device obtains the distance factor at the third moment based on the gesture sensing information at the third moment; the electronic device weights and averages the distance factors corresponding to each gesture recognition point at the third moment to obtain the unit sliding distance.

[0411] Optionally, if the first gesture belongs to the scaling control type, the electronic device obtains the center of gravity factor at the third moment based on the gesture sensing information at the third moment; the electronic device obtains the change in the center of gravity factor corresponding to each gesture recognition point at the third moment based on the center of gravity factor at the third moment; subsequently, the electronic device weighted averages the changes in the center of gravity factors at the third moment to obtain the motion parameters. The method for determining the change in the center of gravity factor is described in the above embodiment and will not be repeated here.

[0412] For example, the electronic device obtains the changes in each centroid factor at the third time point based on the centroid factor at the third time point and the reference centroid factor. The reference centroid factor can be the centroid factor at the fourth time point, which is earlier than the third time point. In one example, the fourth time point is separated from the third time point by a first time interval.

[0413] Optionally, if the first gesture is a rotation control gesture, the electronic device obtains the rotation unit angle based on the gesture sensing information at the third moment, the gesture sensing information at the fourth moment, and the position information of the geometric center of gravity.

[0414] For example, for any one of the at least one gesture recognition points, the electronic device determines a first straight line based on the position information of the gesture recognition point at a third time and the position information of its geometric center of gravity; it determines a second straight line based on the position information of the gesture recognition point at a fourth time and the position information of its geometric center of gravity; it obtains the change angle of the gesture recognition point based on the first and second straight lines; and it calculates a weighted average of the change angles of all gesture recognition points to obtain a unit rotation angle. For an explanation of the fourth time, please refer to the previous embodiment.

[0415] In some embodiments, the process of determining effect control information based on gesture sensing information is implemented by a competition and recognition module. The competition and recognition module receives gesture sensing information sent by the gesture sensing module and executes step S350, using the corresponding calculation method based on the functional gesture indicated by the gesture recognition result to obtain the motion parameters of the first gesture based on the gesture sensing information.

[0416] After determining the motion parameters of the first gesture, the first system service reports the motion parameters of the first gesture to the first application.

[0417] Step S370: The first operation is continued to be performed by the first application according to the motion parameters of the first gesture.

[0418] Optionally, if the first gesture is a swipe control type, the first application follows the user's operation based on the motion parameters of the first gesture, refreshing the display position of the displayed content, or refreshing the displayed content itself. For example... Figure 14 As shown, the first application moves the displayed content from the display area 1410 to the display area 1420 according to the motion parameters of the first gesture.

[0419] Optionally, if the first gesture is a scaling control type, the first application changes the display size of the displayed content according to the motion parameters of the first gesture. For example... Figure 15 As shown, the first application follows the user's operation based on the motion parameters of the first gesture, shrinking the display area 1510 of the displayed content to the display area 1520. This is equivalent to reducing the display size of the displayed content while keeping the content unchanged.

[0420] like Figure 16 As shown, at a certain moment, the first gesture corresponds to five gesture recognition points P1, P2, P3, P4, and P5, and the centroid of the geometric figure is G; the gesture to be recognized at another moment corresponds to five gesture recognition points P1, P2, P3, P4, and P5. ′ P2 ′ P3 ′ P4 ′ and P5 ′ The centroid of the geometric figure is G. ′ During the kneading operation, the geometric center of gravity G and G' ′ The positions between them do not change much, or do not change at all. The center of gravity of the geometric shape is aligned with the center of the display area, and the display area is reduced according to the change in the center of gravity factor, so that the display area can continuously shrink as the pinch operation continues.

[0421] Optionally, if the function gesture indicated by the gesture recognition result is a rotation gesture, the first application controls the display angle of the displayed content according to the effect control information.

[0422] In step S380, the electronic device repeats steps 350 to 370 until the gesture to be recognized disappears. Optionally, before repeating step 350, the electronic device uses the third moment as the new fourth moment and the moment to which the next acquired gesture sensing information belongs as the new third moment.

[0423] Steps S350-S380 are used to complete the follow-up operation after determining the gesture recognition result, so that the user can achieve more flexible function control through gestures.

[0424] Figure 17 This is an interactive diagram illustrating the follow-up process provided for an exemplary embodiment of this application. For example... Figure 17 As shown, this embodiment includes at least the following steps; for steps S1310 to S1370, please refer to the embodiment above. Optionally, if the gesture to be detected continues after step S1370 and the gesture recognition point detected by the electronic device does not disappear, the electronic device executes step S1380; if the gesture recognition point disappears, the electronic device determines that the gesture recognition process has ended. In this embodiment, the first system service is implemented as a contention and recognition module.

[0425] Step S1380: The gesture sensing module acquires the updated gesture sensing information.

[0426] In step S1390, the gesture sensing module sends updated gesture sensing information to the contention and recognition module. The contention and recognition module obtains the updated gesture sensing information from the gesture sensing module.

[0427] In step S1391, the competition and recognition module obtains the motion parameters of the first gesture based on the updated gesture sensing information. Optionally, after step S1391, the competition and recognition module re-recognizes the gesture based on the updated gesture sensing information (e.g., by executing steps S920 to S996). If the newly recognized gesture is the same as the first gesture obtained in step S1350, the competition and recognition module executes step S1392; if the newly recognized gesture is different from the first gesture obtained in step S1350, the competition and recognition module obtains a new first gesture based on the newly recognized gesture and starts execution from step S1360. This method enables the recognition of combined gestures, allowing the gesture recognition method to adapt to an increasing number of gestures.

[0428] In step S1392, the competition and recognition module reports the motion parameters used for the first gesture to the first application.

[0429] In step S1393, the first application continues to execute the first operation based on the motion parameters of the first gesture.

[0430] By controlling the effect information, the responsiveness of the first application can be improved, which helps to enhance the effectiveness of flexibly adjusting preset functions through functional gestures.

[0431] The above combination Figures 3-17 The gesture recognition method provided in the embodiments of this application is described in detail below. Figure 18 This application provides a detailed description of the electronic device provided in its embodiments. All or part of any feature in any embodiment of this application can be freely combined. The resulting combined technical solutions also fall within the scope of this application.

[0432] In one possible design, Figure 18 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 18 As shown, the electronic device 1800 may include a transceiver unit 1801 and a processing unit 1802. The electronic device 1800 can be used to implement the functions of the electronic device involved in the above method embodiments.

[0433] Optionally, the processing unit 1802 is used to support the electronic device 1800 in performing operations. Figure 3 S310 and S340; and / or, for supporting electronic device 1800 to execute S350 to S380; and / or, for supporting electronic device to execute such as Figure 9 S910 to S997, and / or, for supporting electronic device 1800 to perform Figure 13 S1310 to S1370 shown, and / or, are used to support the electronic device 1800 in performing... Figure 17 S1310 to S1393 are shown.

[0434] The transceiver unit may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the electronic device 1800 are respectively for implementing the corresponding process of the gesture recognition method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, it will not be repeated here.

[0435] Optionally, Figure 18 The illustrated electronic device 1800 may also include a storage unit ( Figure 18 (not shown in the image), this storage unit stores a program or instruction. When the processing unit 1802 executes the program or instruction, it causes... Figure 18 The electronic device 1800 shown can perform the gesture recognition method described in the above method embodiments.

[0436] Figure 18The technical effects of the electronic device 1800 shown can be referred to the technical effects of the gesture recognition method described in the above method embodiments, and will not be repeated here.

[0437] In addition to being in the form of electronic device 1800, the technical solution provided in this application can also be a functional unit or chip in an electronic device, or a device used in conjunction with an electronic device.

[0438] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0439] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0440] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0441] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0442] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0443] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the aforementioned steps to implement the gesture recognition method in the above embodiments.

[0444] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the gesture recognition method described in the above embodiments.

[0445] In addition, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory stores a computer program. When the computer program is executed by one or more processors, the apparatus performs the gesture recognition method described in the above-described method embodiments.

[0446] In this application, the computer-readable storage medium, computer program product, or chip provided in the embodiments are all used to execute the corresponding methods described above. Therefore, the beneficial effects they can achieve can be referred to in the beneficial effects of the corresponding methods described above, and will not be repeated here.

[0447] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, optical discs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0448] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0449] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be electrical, mechanical or other forms.

[0450] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk, and other media capable of storing program code.

[0451] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gesture recognition method, characterized by, The method is executed by a first system service in an electronic device, and the method comprises: in response to a user operation for triggering a gesture control, acquiring gesture sensing information; acquiring a first gesture according to the gesture sensing information; reporting information describing the first gesture to a first application, so that the first application executes a first operation corresponding to the first gesture, the first application being an application subscribing to the first gesture.

2. The method of claim 1, wherein, After acquiring the first gesture according to the gesture sensing information, the method further comprises: in response to the user operation, acquiring updated gesture sensing information; acquiring motion parameters of the first gesture according to the updated gesture sensing information; reporting the motion parameters of the first gesture to the first application, so that the first application continues to execute the first operation.

3. The method of claim 2, wherein, The motion parameters of the first gesture are related to a gesture control type to which the first gesture belongs; in the case that the first gesture belongs to a sliding control type, the motion parameters of the first gesture are used to indicate a change in sliding distance, or, in the case that the first gesture belongs to a zooming control type, the motion parameters of the first gesture are used to indicate a change in zooming ratio, or, in the case that the first gesture belongs to a rotating control type, the motion parameters of the first gesture are used to indicate a change in rotating angle.

4. The method according to any one of claims 1 to 3, characterized in that, Before acquiring the gesture sensing information in response to the user operation, the method further comprises: acquiring subscription information of the first application, the subscription information being used to indicate at least one gesture supported by the first application, the at least one gesture including the first gesture.

5. The method according to any one of claims 1 to 4, characterized in that, The acquiring the first gesture according to the gesture sensing information comprises: acquiring at least one type of gesture factor according to the gesture sensing information, each type of gesture factor in the at least one type of gesture factor being used to describe a gesture parameter in a corresponding dimension; acquiring the first gesture according to the at least one type of gesture factor.

6. The method of claim 5, wherein, The acquiring the first gesture according to the at least one type of gesture factor comprises: acquiring a first gesture attribute according to the at least one type of gesture factor, the first gesture attribute being used to represent a first gesture control type to which the first gesture belongs; acquiring at least one second gesture attribute according to the at least one type of gesture factor, the second gesture attribute being used to represent an attribute parameter associated with the first gesture control type; acquiring the first gesture according to the first gesture attribute and the at least one second gesture attribute.

7. The method of claim 6, wherein, The at least one type of gesture factor includes multiple types of gesture factors having competitive priorities, the competitive priorities being used to represent an order in which the multiple types of gesture factors participate in gesture recognition; The acquiring the first gesture attribute according to the at least one type of gesture factor comprises: acquiring a first gesture factor from the multiple types of gesture factors, the first gesture factor being a gesture factor having a competitive priority ranked first in the multiple types of gesture factors; in the case that the first gesture factor meets a corresponding determination condition, determining the first gesture attribute according to a gesture control type corresponding to the first gesture factor.

8. The method of claim 6, wherein, The at least one gesture factor includes multiple gesture factors with competitive priorities, the competitive priorities being used to represent an order in which the multiple gesture factors participate in gesture recognition; The acquiring the first gesture attribute according to the at least one gesture factor comprises: acquiring a first gesture factor from the multiple gesture factors, the first gesture factor being a gesture factor with a highest competitive priority in the multiple gesture factors; in a case where the first gesture factor does not satisfy a corresponding determination condition, acquiring a second gesture factor from the multiple gesture factors, the second gesture factor being a gesture factor with a highest competitive priority in gesture factors that do not participate in gesture recognition in the multiple gesture factors; in a case where the second gesture factor satisfies a corresponding determination condition, determining the first gesture attribute according to a gesture control type corresponding to the second gesture factor.

9. The method according to claim 7 or 8, characterized in that, The acquiring the first gesture attribute according to the at least one gesture factor further comprises: acquiring a third gesture factor from the at least one gesture factor, the third gesture factor being used to verify validity of the user operation; in a case where the third gesture factor satisfies a validity condition, acquiring the first gesture attribute used to represent a gesture control type corresponding to a selected gesture factor, the selected gesture factor being the first gesture factor or the second gesture factor.

10. The method of claim 9, wherein, The third gesture factor comprises at least one of: a distance factor used to represent a moving distance of at least one gesture recognition point indicated by the gesture sensing information; a speed factor used to represent a moving speed of the at least one gesture recognition point.

11. The method according to any one of claims 7 to 10, characterized in that, The multiple gesture factors include a direction factor and a barycenter factor; a competitive priority of the direction factor is higher than a competitive priority of the barycenter factor; a gesture control type corresponding to the direction factor is a sliding control type, and a gesture control type corresponding to the barycenter factor is a rotating control type, or a gesture control type corresponding to the direction factor is a sliding control type, and a gesture control type corresponding to the barycenter factor is a zooming control type; wherein the direction factor is used to represent a moving direction of at least one gesture recognition point indicated by the gesture sensing information, and the barycenter factor is used to represent a distance between the at least one gesture recognition point and a geometric barycenter in a case where a total number of the gesture recognition points is greater than 2, the geometric barycenter being a barycenter of a geometric figure with the at least one gesture recognition point as a vertex.

12. The method according to any one of claims 6 to 11, characterized in that, The acquiring the first gesture according to the first gesture attribute and at least one second gesture attribute comprises: during a duration of the user operation, continuously acquiring n predicted gestures according to the corresponding first gesture attribute and the at least one second gesture attribute, n being a positive integer; in a case where the n predicted gestures satisfy an accuracy condition, acquiring the first gesture according to the n predicted gestures.

13. The method according to any one of claims 6 to 12, characterized in that, The at least one second gesture attribute corresponding to the first gesture control type comprises at least one of: a total number of gesture recognition points indicated by the gesture sensing information, a motion attribute; the first gesture attribute comprises at least one of: a sliding control type, a zooming control type, and a rotating control type; The motion attribute corresponding to the sliding control type comprises a sliding direction attribute, the motion attribute corresponding to the zooming control type comprises a zooming trend attribute, and the motion attribute corresponding to the rotating control type comprises a rotating direction attribute; The total number of the gesture recognition points is at least one of the following: one finger, two fingers, three fingers, four fingers, and five fingers. The sliding direction attribute is at least one of the following: up, down, left, right, left up, right up, left down, and right down. The zooming trend attribute is at least one of the following: opening and pinching. The rotating direction attribute is at least one of the following: clockwise and counterclockwise.

14. The method of claim 13, wherein, The information used to describe the first gesture comprises first information and second information. The first information is determined according to the first gesture attribute and the motion attribute. The second information is the total number of the gesture recognition points.

15. The method according to any one of claims 5 to 14, characterized in that, The at least one type of gesture factor comprises at least one of the following: A number factor used to represent the total number of at least one gesture recognition point indicated by the gesture sensing information; A direction factor used to represent the moving direction of the at least one gesture recognition point; A distance factor used to represent the moving distance of the at least one gesture recognition point; A barycenter factor used to represent the distance between at least one gesture recognition point and a geometric barycenter in the case that the total number of the gesture recognition points is greater than two, the geometric barycenter being the barycenter of a geometric figure with the at least one gesture recognition point as a vertex; A speed factor used to represent the moving speed of the at least one gesture recognition point.

16. The method of claim 15, wherein, The gesture sensing information is used to indicate the position information of at least one gesture recognition point. The at least one type of gesture factor is acquired according to the gesture sensing information, comprising: In the case that the barycenter factor is included in the at least one type of gesture factor, the position information of the geometric barycenter is acquired according to the position information of the at least one gesture recognition point; The barycenter factor corresponding to each gesture recognition point in the at least one gesture recognition point is acquired according to the position information of the at least one gesture recognition point and the position information of the geometric barycenter.

17. The method of claim 15, wherein, The gesture sensing information is used to indicate the position information of at least one gesture recognition point. The at least one type of gesture factor is acquired according to the position information of the gesture recognition point, comprising: In the case that the relative factor is included in the at least one type of gesture factor, for each relative factor, the relative factor corresponding to each gesture recognition point in the at least one gesture recognition point is acquired according to the position information of the at least one gesture recognition point and position reference information, the relative factor being at least one of the direction factor, the speed factor, and the distance factor.

18. The method according to any one of claims 1 to 17, characterized in that, The first system service is located at a system framework layer in the electronic device.

19. An electronic device, comprising: The electronic device comprises: A display screen used for interface display; A transceiver used for sending and receiving radio signals; A memory used for storing computer program instructions; A processor used for executing the computer program instructions to support the electronic device to implement the method according to any one of claims 1 to 18.

20. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and, when run on a computer, causes the method of any one of claims 1 to 18 to be performed.

21. A computer program product comprising instructions, wherein: The computer program product, when run on a computer, causes the computer to perform the method of any one of claims 1 to 18.

Images