Air gesture recognition method and electronic equipment

Abstract

The invention is suitable for the technical field of image processing, and provides an air gesture recognition method and electronic equipment, and the method comprises the steps: enabling a front-facing camera to collect a first image at a second moment after a sliding gesture is recognized at a first moment; the second moment is later than the first moment; if it is detected that the first image comprises the starting gesture at the third moment, the interval between the third moment and the first moment is smaller than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, the hand type icon corresponding to the starting gesture in the first image is not displayed; according to the embodiment of the invention, the user can be prompted not to recognize the dynamic gesture currently in the recovery process of the target sliding gesture corresponding to the continuous same-direction sliding, so that the condition of turning pages back and forth can be avoided, and the operation experience during continuous same-direction screen sliding is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a method for recognizing air gestures and an electronic device. Background Art

[0002] Currently, electronic devices such as mobile phones can use multiple frames of images collected by a front camera to recognize dynamic air gestures, and can perform corresponding operations in response to the recognized air gestures to achieve air control of the screen.

[0003] The air gesture may include a swipe gesture, and the swipe gesture can be used to swipe the screen or turn pages. However, the current method for recognizing swipe gestures has a low accuracy rate. There may be a situation where the user only wants to swipe the screen once, but the screen swipes back and forth or the pages turn back and forth, resulting in a poor operation experience when continuously swiping the screen in the same direction. Summary of the Invention

[0004] Embodiments of this application provide a method for recognizing air gestures and an electronic device, which can prompt the user that dynamic gesture recognition is not performed currently during the recovery process of the target swipe gesture corresponding to continuous swiping in the same direction, thereby avoiding the situation of turning pages back and forth and improving the operation experience when continuously swiping the screen in the same direction.

[0005] In a first aspect, an embodiment of this application provides a method for recognizing air gestures, which is applied to an electronic device including a camera. The method for recognizing air gestures includes: after recognizing a swipe gesture at a first moment, the camera captures a first image at a second moment; the second moment is later than the first moment; if it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image is not displayed.

[0006] According to the method for recognizing air gestures provided by the embodiments of this application, if after recognizing a swipe gesture at a first moment, a first image captured by a front camera at a second moment later than the first moment is obtained, and it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, it indicates that the current is in the recovery process of the target swipe gesture corresponding to continuous swiping in the same direction. In this case, by not displaying the hand icon corresponding to the starting gesture in the first image, the user can be prompted that dynamic gesture recognition is not performed currently, thereby avoiding recognizing a gesture in the opposite direction to the swipe gesture, and further avoiding the situation of turning pages back and forth, and improving the operation experience when continuously swiping the screen in the same direction.

[0007] In an alternative implementation of the first aspect, it further includes: if it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

[0008] According to the air gesture recognition method provided by the embodiments of the present application, when it is determined that the current is in the recovery process of the target sliding gesture corresponding to the continuous same-direction sliding operation, by not executing the dynamic gesture recognition process, it is possible to avoid recognizing a gesture in the opposite direction to the sliding gesture, thereby avoiding the situation of back-and-forth page turning; in addition, by not storing the information of the first image, not only can the storage space be saved, but also the use of the information of the first image for dynamic gesture recognition can be avoided, thereby reducing the misrecognition probability of air gestures.

[0009] In an alternative implementation of the first aspect, it further includes: if it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is the same as the starting gesture of the sliding gesture, then a hand icon corresponding to the starting gesture in the first image is displayed, and the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information record queue.

[0010] According to the air gesture recognition method provided by the embodiments of the present application, if it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is the same as the starting gesture of the sliding gesture, it indicates that the recovery process of the target sliding gesture corresponding to the continuous same-direction sliding operation has been completed. In this case, by displaying a hand icon corresponding to the starting gesture in the first image, the user can be prompted that the dynamic gesture can be recognized normally currently, which is convenient for the user to continue to input the target sliding gesture corresponding to the continuous same-direction sliding operation; in addition, by storing the information of the first image in the frame information record queue, the electronic device can accurately recognize the air gesture based on the information of multiple frames of images in the frame information record queue subsequently, improving the accuracy of air gesture recognition.

[0011] In an alternative implementation of the first aspect, it further includes: If it is detected at the third moment that the starting gesture is included in the first image, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swiping gesture, and the gesture in the third image captured by the camera within the second duration after the second moment is the same as the starting gesture in the first image, then display the hand - shaped icon corresponding to the starting gesture in the first image, execute the dynamic gesture recognition process, and store the information of the first image and the information of the third image in the frame information record queue; the second duration is less than the first duration.

[0012] According to the air gesture recognition method provided by the embodiments of the present application, if it is detected at the third moment that the starting gesture is included in the first image, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swiping gesture, and the gesture in the third image captured by the camera within the second duration after the second moment is the same as the starting gesture in the first image, it indicates that the user deliberately maintains the starting gesture of the reverse - direction swiping gesture, that is, it indicates that the user wants to switch to the reverse - direction swiping gesture. In this case, by displaying the hand - shaped icon corresponding to the starting gesture in the first image and executing the dynamic gesture recognition process, the operation of the user switching the swiping gesture can be quickly responded to, thereby improving the user's operation experience; in addition, by storing the information of the first image and the information of the third image in the frame information record queue, it helps the electronic device to accurately recognize the swiping gesture switched by the user based on the information of multiple frames of images in the frame information record queue.

[0013] In an alternative implementation of the first aspect, it further includes: If the interval between the third moment and the first moment is greater than the first duration, then display the hand - shaped icon corresponding to the starting gesture in the first image, execute the dynamic gesture recognition process, and store the information of the first image in the frame information record queue.

[0014] In an alternative implementation of the first aspect, it further includes: If other dynamic gestures except the swiping gesture are recognized at the first moment, then execute the dynamic gesture recognition process.

[0015] According to the air gesture recognition method provided by the embodiments of the present application, in the case of detecting that the user does not perform a continuous same - direction swiping operation, by displaying the hand - shaped icon corresponding to the starting gesture in the first image and executing the dynamic gesture recognition process, the air gesture recognition process can proceed normally.

[0016] In an alternative implementation of the first aspect, the swiping gesture includes: an upward - swiping gesture, or a downward - swiping gesture, or a left - swiping gesture, or a right - swiping gesture.

[0017] In an alternative implementation of the first aspect, the starting gesture includes: a palm with fingers facing up, or a palm with fingers facing left, or a palm with fingers facing right, or a back of the hand with fingers facing down, or a back of the hand with fingers facing left, or a back of the hand with fingers facing right.

[0018] In an alternative implementation of the first aspect, before recognizing the swipe gesture at the first moment, it further includes: recognizing a dynamic gesture based on the information of multiple frames of images in the frame information recording queue; the frame information recording queue is used to store the information of the starting gesture images and the information of the stable state images, the starting gesture images are the images including the starting gesture, and the stable state images are the non-starting gesture images with stable hand forms; after recognizing the swipe gesture at the first moment, it further includes: deleting the information of all images related to the swipe gesture in the frame information recording queue.

[0019] According to the air gesture recognition method provided by the embodiments of the present application, after each air gesture is recognized, by deleting the information of all images related to the swipe gesture in the frame information recording queue, not only can the storage space be saved, but also the normal progress of the subsequent air gesture recognition process can be ensured, and the efficiency and accuracy of air gesture recognition can be improved.

[0020] In an alternative implementation of the first aspect, before recognizing the dynamic gesture based on the information of multiple frames of images in the frame information recording queue, it further includes: acquiring multiple consecutive frames of images, the multiple consecutive frames of images are captured by a camera; for each frame of image in the multiple consecutive frames of images in turn, determining the hand feature information of the current frame image; based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand form in the current frame image is stable; in the case where the current frame image includes a starting gesture image, or the hand form in the current frame image is stable, storing the information of the current frame image in the frame information recording queue; the information of the current frame image includes the hand feature information.

[0021] According to the air gesture recognition method provided by the embodiments of the present application, by setting up the frame information recording queue and configuring the frame information recording queue to only store the information of the starting gesture images and the information of the stable state images, enabling the recognition of dynamic gestures only based on the information of multiple frames of images stored in the frame information recording queue, the accuracy of air gesture recognition can be improved.

[0022] In an alternative implementation of the first aspect, based on the hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable, including: when the frame information recording queue is empty, based on the hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture; when the frame information recording queue is not empty, based on the hand feature information of the current frame image, it is determined whether the hand shape in the current frame image is stable.

[0023] According to the air gesture recognition method provided by the embodiments of the present application, by determining whether the current frame image includes the starting gesture of any dynamic gesture when the frame information recording queue is empty, and determining whether the hand shape in the current frame image is stable when the frame information recording queue is not empty, the efficiency of air gesture recognition can be improved.

[0024] In an alternative implementation of the first aspect, the hand feature information includes the hand category and the finger orientation; determining whether the current frame image includes the starting gesture of any dynamic gesture based on the hand feature information of the current frame image includes: when the hand category corresponding to the current frame image is the palm and the finger orientation is upward, or leftward, or rightward, it is determined that the current frame image includes the starting gesture; or, when the hand category corresponding to the current frame image is the back of the hand and the finger orientation is downward, or leftward, or rightward, it is determined that the current frame image includes the starting gesture.

[0025] In an alternative implementation of the first aspect, the hand feature information includes the hand category and the finger orientation; determining whether the current frame image includes the starting gesture of any dynamic gesture based on the hand feature information of the current frame image includes: when the hand category corresponding to the current frame image is the palm and the finger orientation is not upward, not leftward, and not rightward, it is determined that the current frame image does not include the starting gesture; or, when the hand category corresponding to the current frame image is the back of the hand and the finger orientation is not downward, not leftward, and not rightward, it is determined that the current frame image does not include the starting gesture; or, when the hand category corresponding to the current frame image is not the palm and not the back of the hand, it is determined that the current frame image does not include the starting gesture.

[0026] In an alternative implementation of the first aspect, the human hand feature information includes the information of the human hand detection box, the human hand category, the coordinates of the human hand key points, and the finger orientation; determining whether the hand gesture in the current frame image is stable according to the human hand feature information of the current frame image includes: calculating the intersection over union (IoU) between the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image; calculating the standard deviation of the first-order difference between the human hand key points corresponding to the current frame image and the human hand key points corresponding to the previous frame image; determining that the hand gesture in the current frame image is stable when the human hand category corresponding to the current frame image is the same as the human hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, the IoU is greater than or equal to a first preset IoU, and the standard deviation of the first-order difference is less than or equal to a preset standard deviation.

[0027] In an alternative implementation of the first aspect, it further includes: determining that the hand gesture in the current frame image is unstable when the human hand category corresponding to the current frame image is different from the human hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the IoU is less than the first preset IoU, or the standard deviation of the first-order difference is greater than the preset standard deviation.

[0028] In an alternative implementation of the first aspect, when it is determined that the hand gesture in the current frame image is stable, it further includes: determining whether the current is in a gesture hovering state according to the human hand feature information of the current frame image and the human hand feature information of the previous frame image; performing a dynamic gesture recognition process when the current is not in a gesture hovering state.

[0029] According to the air gesture recognition method provided by the embodiments of the present application, by determining whether the current is in a gesture hovering state and performing a dynamic gesture recognition process only when the current is not in a gesture hovering state, the power consumption of the electronic device can be saved.

[0030] In an alternative implementation of the first aspect, the human hand feature information includes the information of the human hand detection box, the human hand category, the coordinates of the key points of the human hand, and the finger orientation; determining whether the current state is a gesture hovering state according to the human hand feature information of the current frame image and the human hand feature information of the previous frame image includes: calculating the intersection over union (IoU) of the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image; calculating a first ratio of the human hand detection box corresponding to the current frame image in the current frame image according to the information of the human hand detection box corresponding to the current frame image; calculating a second ratio of the human hand detection box corresponding to the previous frame image in the previous frame image according to the information of the human hand detection box corresponding to the previous frame image; calculating the difference between the first ratio and the second ratio; if the human hand category corresponding to the current frame image is different from the human hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the IoU is less than a second preset IoU, or the difference between the first ratio and the second ratio is greater than a preset difference, it is determined that the current state is not a gesture hovering state.

[0031] In an alternative implementation of the first aspect, it further includes: when the human hand category corresponding to the current frame image is the same as the human hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the IoU is greater than or equal to the second preset IoU, and the difference between the first ratio and the second ratio is less than or equal to the preset difference, it is determined that the current state is a gesture hovering state.

[0032] In a second aspect, an embodiment of the present application provides an electronic device, including: one or more processors; one or more memories; the one or more memories store one or more computer-executable programs, and the one or more computer-executable programs include instructions that, when executed by the one or more processors, cause the electronic device to execute the steps in the air gesture recognition method according to any one of the implementations of the first aspect described above.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer-executable program, and the computer-executable program, when called by an electronic device, causes the electronic device to execute the steps in the air gesture recognition method according to any one of the implementations of the first aspect described above.

[0034] In a fourth aspect, an embodiment of the present application provides a computer-executable program product, when the computer-executable program product runs on an electronic device, it causes the electronic device to execute the steps in the air gesture recognition method according to any one of the implementations of the first aspect described above.

[0035] In a fifth aspect, an embodiment of the present application provides a chip system. The chip system is applied to an electronic device and includes a processor coupled to a memory. The memory is used to store computer program instructions. When the processor calls the computer program instructions, the electronic device implements the steps in the air gesture recognition method in any implementation manner of the first aspect as described above. The chip system may be a single chip or a chip module composed of multiple chips.

[0036] It can be understood that for the beneficial effects of the second to fifth aspects above, reference can be made to the relevant descriptions in the first aspect, and details are not elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 FIG. is a schematic diagram of a scenario for controlling an electronic device through air gestures;

[0038] Figure 2 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0039] Figure 3 FIG. is a schematic software architecture diagram of an electronic device provided by an embodiment of the present application;

[0040] Figure 4 FIG. is a schematic diagram of the change process of the hand shape corresponding to each air gesture provided by an embodiment of the present application;

[0041] Figure 5 FIG. is a schematic diagram of the interaction timing between modules in the mobile phone system architecture during the implementation process of an air gesture recognition method provided by an embodiment of the present application;

[0042] Figure 6 FIG. is a schematic diagram of an image with a human hand captured by the AO camera provided by an embodiment of the present application;

[0043] Figure 7 FIG. is a schematic diagram of the setting scenario of a preset operation provided by an embodiment of the present application;

[0044] Figure 8 FIG. is a specific implementation flowchart of S52 in an air gesture recognition method provided by an embodiment of the present application;

[0045] Figure 9 FIG. is a specific implementation flowchart of S53 in an air gesture recognition method provided by an embodiment of the present application;

[0046] Figure 10 FIG. is a schematic diagram of a finger orientation determination scenario provided by an embodiment of the present application;

[0047] Figure 11 FIG. is a specific implementation flowchart of S532 in an air gesture recognition method provided by an embodiment of the present application;

[0048] Figure 12 This is the specific implementation flowchart of S5322 in an air gesture recognition method provided by an embodiment of the present application;

[0049] Figure 13 This is the specific implementation flowchart of step b3 in an air gesture recognition method provided by an embodiment of the present application;

[0050] Figure 14 This is the specific implementation flowchart of the continuous same - direction sliding detection process in an air gesture recognition method provided by an embodiment of the present application;

[0051] Figure 15 This is a schematic diagram of the user interface involved in an air gesture recognition method provided by an embodiment of the present application;

[0052] Figure 16 This is the specific implementation flowchart of step c3 in an air gesture recognition method provided by an embodiment of the present application;

[0053] Figure 17 This is the specific implementation flowchart of S535 in an air gesture recognition method provided by an embodiment of the present application;

[0054] Figure 18 This is a schematic flowchart of an air gesture recognition method provided by another embodiment of the present application. Detailed implementation manners

[0055] It should be noted that the terms used in the implementation manner part of the embodiments of the present application are only used to explain the specific embodiments of the present application, rather than being intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B; "and / or" herein is only a description of the association relationship of related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more than two, and "at least one", "one or more" mean one, two or more than two.

[0056] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0057] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc., which appear in different places in this specification, do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0058] Currently, electronic devices such as mobile phones can support non-contact air gesture interaction technology. The air gesture interaction technology can achieve non-contact human-computer interaction in scenarios where users are inconvenient to touch the electronic device, improving the convenience of operating the electronic device.

[0059] Specifically, the electronic device can support the camera always-on (AO) function. When the camera AO function is turned on, the front camera of the electronic device is in a constantly-on state and can collect images in real time. The electronic device can identify the air gestures input by the user by analyzing multiple frames of images collected by the AO camera (i.e., the front camera in the constantly-on state), and can perform corresponding operations in response to the identified air gestures to achieve air control of the electronic device.

[0060] Exemplarily, please refer to Figure 1 , assuming that the palm with fingers upward shown in (a) of Figure 1 changes to the grasping gesture of a clenched fist shown in (b) of Figure 1 corresponds to a screenshot operation, then the electronic device can perform a screenshot operation after recognizing the grasping gesture to obtain a screenshot picture, thereby realizing air screenshot. Assuming that the palm with fingers upward shown in (c) of Figure 1 changes to the downward-sliding gesture of the back of the hand with fingers downward shown in (d) of Figure 1 corresponds to a downward screen sliding operation or a downward page turning operation, then the electronic device can perform a downward screen sliding operation or a downward page turning operation after recognizing the downward-sliding gesture, thereby realizing air downward screen sliding or air downward page turning. Assuming that the back of the hand with fingers downward shown in (e) of Figure 1 changes to the upward-sliding gesture of the palm with fingers upward shown in (f) of Figure 1 corresponds to an upward screen sliding operation or an upward page turning operation, then the electronic device can perform an upward screen sliding operation or an upward page turning operation after recognizing the upward-sliding gesture, thereby realizing air upward screen sliding or air upward page turning.

[0061] According to Figure 1the downward swipe gesture shown in (c) and (d) in Figure 1 and the upward swipe gesture shown in (e) and (f) in

[0062] it can be seen that the processes of hand form changes involved in the two swipe gestures in opposite directions are two completely opposite processes. Therefore, when the user wants to continuously swipe the screen in the same direction or continuously turn pages in the same direction, such as continuously turning pages downward, after the user executes a downward swipe gesture, the electronic device will also use each frame of image collected during the process of restoring from the end gesture of this downward swipe gesture (i.e., the back of the hand with fingers pointing downward) to the start gesture of the next downward swipe gesture (i.e., the palm with fingers pointing upward) for air gesture recognition. In this way, the electronic device will recognize an upward swipe gesture after recognizing the downward swipe gesture, resulting in a situation of back-and-forth page turning, which reduces the poor operation experience when the user continuously swipes the screen in the same direction or continuously turns pages in the same direction.

[0063] In view of this, the embodiments of the present application provide an air gesture recognition method and an electronic device. If after recognizing a swipe gesture at a first moment, a first image collected by a front camera at a second moment later than the first moment is obtained, and at a third moment, it is detected that the first image includes a start gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the start gesture in the first image is different from the start gesture of the swipe gesture, it indicates that the current is in the recovery process of the target swipe gesture corresponding to the continuous same-direction swipe operation. In this case, by not displaying the hand shape icon corresponding to the start gesture in the first image, the user can be prompted that dynamic gesture recognition is not performed currently, thereby avoiding recognizing a gesture in the opposite direction to the swipe gesture, and further avoiding the situation of back-and-forth page turning, and improving the operation experience when continuously swiping the screen in the same direction.

[0064] The air gesture recognition method provided by the embodiments of the present application can be applied to an electronic device including a front camera. The electronic device may include a mobile phone, a tablet computer, a wearable device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not limit the specific type of the electronic device.

[0065] Exemplarily, please refer to Figure 2 , which is a schematic structural diagram of an electronic device provided by the embodiments of the present application.

[0066] As Figure 2As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0067] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0068] Exemplarily, the processor 110 may be used to execute the air gesture recognition method in the embodiments of the present application.

[0069] The controller may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0070] A memory for storing instructions and data may also be provided in the processor 110. For example, a first storage area and a second storage area may be provided in the memory. Among them, the first storage area may be used to store the frame information record queue involved in the air gesture recognition process; the second storage area may be used to store the recognition time and type of each recognized air gesture, etc.

[0071] The camera 193 may be used to capture static images or videos. The electronic device may include one or N cameras 193, where N is a positive integer greater than 1. At least one of the N cameras 193 is a front camera.

[0072] The display screen 194 is used to display images, videos, hand - shaped icons involved in the air gesture recognition process, etc. The display screen 194 may include a display panel. The electronic device may implement the display function through the GPU, the display screen 194, and the application processor, etc.

[0073] It can be understood that the above is an exemplary description of the structure of the electronic device. It should be understood that in other embodiments, the electronic device may include more or fewer components than shown in the figure, or may combine certain components, or split certain components, or may have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0074] The software system of the electronic device may adopt a layered architecture, an event - driven architecture, a micro - kernel architecture, a microservices architecture, or a cloud architecture. In this embodiment of the application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software architecture of the electronic device.

[0075] Please refer to Figure 3 , which is a schematic diagram of the software architecture of an electronic device provided by an embodiment of this application.

[0076] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system can be divided into four layers, from top to bottom, namely the application layer (application), the application framework layer (application framework), the system runtime library layer, and the kernel layer (kernel).

[0077] The application layer may include a series of application packages. For example, it may include application packages such as settings, camera, and intelligent perception. For the convenience of description, the application packages may be abbreviated as applications hereinafter.

[0078] The intelligent perception application can be used to support the air gesture service. The air gesture service can include a display service related to air gestures, such as displaying a hand icon corresponding to the starting gesture of the air gesture; or, the air gesture service can include a response service for responding to air gestures, such as a service for performing an operation corresponding to the air gesture.

[0079] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer can include some predefined functions.

[0080] Exemplarily, the application framework layer can include a window manager, a message manager, a sensor service, etc.

[0081] The window manager can be used to manage window programs. For example, the window manager can be used to obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0082] The message manager can be used to implement the message passing between different components. Specifically, the message manager can decouple different components through the message publishing and subscribing mechanism, so that a component can directly communicate with other components through message publishing without calling another component.

[0083] Exemplarily, the intelligent perception application can subscribe to messages related to the air gesture service through the message manager.

[0084] The sensor service can be used to manage and provide sensor data. For example, it can be used to manage the data of the acceleration sensor, magnetic sensor, and gyroscope sensor in the hardware layer, and provide the data of each sensor to other modules.

[0085] The system runtime library layer can include an air gesture algorithm library, etc.

[0086] The air gesture algorithm library can be used to implement the recognition of air gestures based on a series of consecutive frames of images captured by the AO camera in the hardware layer. Exemplarily, the air gesture algorithm library can include a human hand feature acquisition module and an air gesture recognition module.

[0087] Among them, the human hand feature acquisition module can include a human hand target detection unit, a human hand classification unit, and a human hand key point detection unit. The air gesture recognition module can include an input unit, a human hand stable state detection unit, a continuous same-direction sliding detection unit, a gesture hovering state detection unit, a dynamic gesture recognition unit, and a timeout detection unit. It should be noted that the specific functions of the human hand feature acquisition module and the air gesture recognition module will be introduced in subsequent embodiments and will not be elaborated here for the time being.

[0088] The kernel layer is the layer between the hardware and the software. The kernel layer may include a camera driver, a sensor driver, etc.

[0089] It should be noted that Figure 3 Only the modules related to the embodiments of the present application are shown. In other embodiments, each layer may further include any other possible modules, and each module may further include one or more sub-modules, which are not limited in the present application.

[0090] The following will exemplarily illustrate the interaction timing between the modules in the mobile phone system architecture during the implementation process of the air gesture recognition method provided by the embodiments of the present application with reference to the accompanying drawings.

[0091] For ease of understanding, the air gestures involved in the embodiments of the present application will be introduced first.

[0092] In the embodiments of the present application, the air gesture may be a dynamic gesture. Exemplarily, the air gesture may include types such as a grasping gesture, a flipping gesture, a sliding gesture, and a pressing gesture. Among them, the sliding gesture may include types such as an upward sliding gesture, a downward sliding gesture, a leftward sliding gesture, and a rightward sliding gesture. It can be understood that each air gesture may correspond to a dynamic hand shape change process. The hand shape change processes corresponding to different air gestures may be different.

[0093] Please refer to Figure 4 , which is a schematic diagram of the hand shape change process corresponding to each air gesture provided by the embodiments of the present application.

[0094] As Figure 4 shown, the hand shape change process corresponding to the grasping gesture may include: changing from a palm with fingertips upward to a fist.

[0095] The hand shape change process corresponding to the flipping gesture may include: changing from a palm with fingertips upward to a back of the hand with fingertips upward.

[0096] The hand shape change process corresponding to the upward sliding gesture may include: changing from a back of the hand with fingertips downward to a palm with fingertips upward.

[0097] The hand shape change process corresponding to the downward sliding gesture may include: changing from a palm with fingertips upward to a back of the hand with fingertips downward.

[0098] The hand shape change process corresponding to the leftward sliding gesture may include: changing from a palm with fingertips to the right to a back of the hand with fingertips to the left; or changing from a back of the hand with fingertips to the right to a palm with fingertips to the left.

[0099] The hand shape change process corresponding to the rightward sliding gesture may include: changing from a palm with fingertips to the left to a back of the hand with fingertips to the right; or changing from a back of the hand with fingertips to the left to a palm with fingertips to the right.

[0100] The process of hand shape change corresponding to the pressing gesture may include: changing from a palm with fingertips facing up and a relatively large distance from the screen to a palm with fingertips facing up and a relatively small distance from the screen.

[0101] In the embodiments of the present application, to facilitate determining the type of air gesture, multiple hand shapes involved in the air gesture can be divided into a starting gesture, an intermediate gesture, and an ending gesture. Among them, the starting gesture can refer to the starting action when the user performs the air gesture, the ending gesture can refer to the ending action when the user finishes performing the air gesture, and the intermediate gesture can include various hand shapes experienced during the process of changing from the starting gesture to the ending gesture. It can be understood that at least one of the starting gesture and the ending gesture of different types of air gestures is different. Therefore, when performing air gesture recognition, the type of air gesture can be determined at least based on the starting gesture and the ending gesture.

[0102] Please continue to refer to Figure 4 , the starting gesture of the grasping gesture can be a palm with fingertips facing up, the ending gesture can be a fist, and the intermediate gesture can include various hand shapes experienced during the process of changing from a palm with fingertips facing up to a fist.

[0103] The starting gesture of the flipping gesture can be a palm with fingertips facing up, the ending gesture can be the back of the hand with fingertips facing up, and the intermediate gesture can include various hand shapes experienced during the process of changing from a palm with fingertips facing up to the back of the hand with fingertips facing up.

[0104] The starting gesture of the upward sliding gesture can be the back of the hand with fingertips facing down, the ending gesture can be a palm with fingertips facing up, and the intermediate gesture can include various hand shapes experienced during the process of changing from the back of the hand with fingertips facing down to a palm with fingertips facing up.

[0105] The starting gesture of the downward sliding gesture can be a palm with fingertips facing up, the ending gesture can be the back of the hand with fingertips facing down, and the intermediate gesture can include various hand shapes experienced during the process of changing from a palm with fingertips facing up to the back of the hand with fingertips facing down.

[0106] The starting gesture of the left sliding gesture can be a palm with fingertips facing right, the ending gesture can be the back of the hand with fingertips facing left, and the intermediate gesture can include various hand shapes experienced during the process of changing from a palm with fingertips facing right to the back of the hand with fingertips facing left. Alternatively, the starting gesture of the left sliding gesture can be the back of the hand with fingertips facing right, the ending gesture can be a palm with fingertips facing left, and the intermediate gesture can include various hand shapes experienced during the process of changing from the back of the hand with fingertips facing right to a palm with fingertips facing left.

[0107] The start of the right - swipe gesture can be a back of the hand with fingertips pointing left, the end gesture can be a palm with fingertips pointing right, and the intermediate gestures can include all the hand shapes experienced during the process of changing from a back of the hand with fingertips pointing left to a palm with fingertips pointing right. Or, the start gesture of the right - swipe gesture can be a palm with fingertips pointing left, the end gesture can be a back of the hand with fingertips pointing right, and the intermediate gestures can include all the hand shapes experienced during the process of changing from a palm with fingertips pointing left to a back of the hand with fingertips pointing right.

[0108] The start gesture of the pressing gesture can be a palm with fingertips pointing up, the end gesture can also be a palm with fingertips pointing up, and the proportion of the start gesture in the image is less than the proportion of the end gesture in the image. The intermediate gestures of the pressing gesture can include all the hand shapes experienced during the process of changing from a palm with fingertips pointing up and having a smaller proportion in the image to a palm with fingertips pointing up and having a larger proportion in the image.

[0109] Please refer to Figure 5 , which is a schematic diagram of the interaction timing between modules in the mobile phone system architecture during the implementation process of a method for recognizing air gestures provided by an embodiment of this application. In some embodiments, when the mobile phone performs air gesture recognition, information interaction can occur between the AO camera, the human hand feature acquisition module, the air gesture recognition module, the message manager, and the intelligent perception application in the mobile phone system architecture. The specific interaction process can include S51 - S54, which are described in detail as follows:

[0110] S51, The AO camera captures multiple consecutive frames of images and sequentially sends each frame of image to the human hand feature acquisition module according to the acquisition timing. Among them, the acquisition timing can be used to represent the order of the AO camera capturing each frame of image.

[0111] In an optional implementation manner, the air gesture recognition operation can be implemented in the screen - on state. Based on this, the AO camera can sequentially send each frame of image captured in the screen - on state to the human hand feature acquisition module according to the acquisition timing.

[0112] S52, The human hand feature acquisition module determines the human hand feature information of the current frame of image for each frame of the multiple consecutive frames of images in turn, and sends the human hand feature information of the current frame of image to the air gesture recognition module.

[0113] In this embodiment, the human hand feature information of the current frame of image determined by the human hand feature acquisition module can include information such as the human hand detection box corresponding to the current frame of image, the human hand category, and the coordinates of the human hand key points.

[0114] Among them, the information of the human hand detection box can be used to represent the position and range of the human hand in the corresponding image. Exemplarily, the human hand detection box can be the smallest rectangular box that can enclose the human hand. Based on this, the information of the human hand detection box can be represented by the coordinates of the center point of the human hand detection box in the first preset coordinate system and the side length of the human hand detection box; or can be represented by the coordinates of the two diagonal vertices of the human hand detection box in the first preset coordinate system, etc., and the embodiments of the present application do not limit this.

[0115] The first preset coordinate system can be a plane rectangular coordinate system established based on the image. For example, it can be a plane rectangular coordinate system established with the upper left vertex of the image as the coordinate origin and the two sides intersecting at the upper left vertex as the x-axis and y-axis respectively.

[0116] Exemplarily, please refer to Figure 6 , which is a schematic diagram of an image with a human hand captured by the AO camera provided in the embodiment of the present application. As Figure 6 shown, when the human hand feature acquisition module determines the human hand feature information of the current frame image 61, it can establish a first preset coordinate system with the upper left vertex O of the image as the coordinate origin and the straight lines where the first side OL and the second side OH are located as the x-axis and y-axis respectively. Based on this, assuming that the two diagonal vertices of the human hand detection box 610 of the current frame image determined by the human hand feature acquisition module are vertex R and vertex S respectively, the human hand feature acquisition module can use the coordinates (x R , y R ) of vertex R and the coordinates (x S , y S ) of vertex S to determine the information of the human hand detection box corresponding to the current frame image.

[0117] The human hand category can be used to represent the type of the hand shape in the figure.

[0118] Exemplarily, the human hand category can include the palm, the back of the hand, or the fist, etc.

[0119] The coordinates of the human hand key points can be used to represent the positions of each human hand key point in the image.

[0120] Exemplarily, please continue to refer to Figure 6 , the human hand can include 21 key points, which are the key points corresponding to label 0 to label 20 respectively. Based on this, the coordinates of the human hand key points corresponding to the current frame image can be a coordinate array composed of the coordinates of the 21 key points of the human hand in the first preset coordinate system.

[0121] In an alternative implementation, when there is a human hand in the current frame image, the human hand feature acquisition module can determine the human hand feature information of the current frame image. When there is no human hand in the current frame image, the human hand feature acquisition module can discard the current frame image, which can improve the recognition efficiency of air gestures.

[0122] In this embodiment, the human hand feature acquisition module can facilitate the air gesture recognition module to recognize whether there is an air gesture and determine the type of the air gesture based on the human hand feature information of multiple frame images by determining the human hand feature information of the images collected by the AO camera and sending the human hand feature information of each frame image to the air gesture recognition module.

[0123] S53, the air gesture recognition module determines, based on the human hand feature information of the current frame image, whether the current frame image includes the starting gesture of any dynamic gesture or whether the hand shape in the current frame image is stable; when the current frame image includes the starting gesture image or the hand shape in the current frame image is stable, the air gesture recognition module stores the information of the current frame image in the frame information record queue; and recognizes the air gesture based on the information of multiple frame images in the frame information record queue.

[0124] Among them, the information of the current frame image may include the human hand feature information of the current frame image.

[0125] It can be understood that since the user's hand usually maintains a static state briefly when starting to perform an air gesture or after finishing performing an air gesture, the starting gesture and the ending gesture of the air gesture are usually gestures with a stable hand shape. The stable hand shape can refer to states such as a clear hand, no ghosting of the hand, and no deformation of the hand.

[0126] In a specific implementation, for any frame image collected by the AO camera, when the air gesture recognition module determines whether the current frame image includes the starting gesture of any dynamic gesture, it can determine whether the hand shape in the current frame image is the same as the hand shape corresponding to the starting gesture of any air gesture according to the human hand feature information of the current frame image. Optionally, when the hand shape in the current frame image is the same as the hand shape corresponding to the starting gesture of any air gesture, the air gesture recognition module can determine that the current frame image includes the starting gesture, that is, determine that the current frame image is a starting gesture image; optionally, when the hand shape in the current frame image is not the same as the hand shapes corresponding to the starting gestures of all air gestures, the air gesture recognition module can determine that the current frame image does not include the starting gesture, that is, determine that the current frame image is a non-starting gesture image.

[0127] In a specific implementation, for any non-start gesture image of a frame, when the air gesture recognition module determines whether the hand shape in the current frame image is stable, it can determine whether the hand shape in the current frame image is stable according to the hand feature information of the current frame image and the hand feature information of the previous frame image. The specific determination method will be described in detail in the subsequent embodiments and will not be elaborated here for the time being.

[0128] The frame information recording queue can be used to store the information of the start gesture image or the stable state image recognized by the air gesture recognition module. The information of the images stored in the frame information recording queue can include the hand feature information of the images. In this embodiment, the stable state image can refer to a non-start gesture image in which the hand shape is in a stable state.

[0129] In this embodiment, different air gestures can be configured with different recognition strategies. The recognition strategy can be used to represent the change characteristics of the hand shape of the air gesture.

[0130] Based on this, in a specific implementation, when the air gesture recognition module recognizes an air gesture based on the information of multiple frames of images in the frame information recording queue and determines the type of the air gesture, it can, based on the recognition strategies of each air gesture, determine whether there are multiple consecutive frames of images in the frame information recording queue that conform to the change characteristics of the hand shape of any air gesture. Optionally, when there are multiple consecutive frames of images in the frame information recording queue that conform to the change characteristics of the hand shape of the target air gesture, the air gesture recognition module can determine that there is an air gesture currently, and can determine the type of the target air gesture as the type of the currently recognized air gesture. For example, assuming that there are multiple consecutive frames of images in the frame information recording queue that conform to the change characteristics of the hand shape of the upward swipe gesture, the air gesture recognition module can determine that there is an upward swipe gesture currently. Optionally, when there are no multiple consecutive frames of images in the frame information recording queue that conform to the change characteristics of the hand shape of any air gesture, the air gesture recognition module can determine that there is no air gesture currently.

[0131] In this embodiment, every time the air gesture recognition module recognizes an air gesture, it can delete the information of all images related to the air gesture from the frame information recording queue. In this way, not only can the storage space of the mobile phone be saved, but it is also convenient for the air gesture recognition module to recognize the start gesture image or the stable state.

[0132] In addition, every time the air gesture recognition module recognizes an air gesture, it can also send the type of the air gesture to the message manager, so that the intelligent perception application can obtain the type of the air gesture from the message manager and execute the preset operation corresponding to the air gesture.

[0133] S54, the intelligent perception application obtains the type of the air gesture from the message manager and performs a preset operation corresponding to the type of the air gesture.

[0134] Exemplarily, each air gesture can correspond to a preset operation. The preset operations corresponding to the respective air gestures can be the system default or can be customized by the user.

[0135] Exemplarily, the grasping gesture can correspond to the air screenshot operation. Based on this, when the type of the air gesture is the grasping gesture, the intelligent perception application can perform the screenshot operation.

[0136] The flipping gesture can correspond to the air page turning operation. Based on this, when the type of the air gesture is the grasping gesture, the intelligent perception application can perform the page turning operation.

[0137] The upward sliding gesture can correspond to the upward screen sliding operation or the upward page turning operation. Based on this, when the type of the air gesture is the upward sliding gesture, the intelligent perception application can perform the upward screen sliding operation or the upward page turning operation.

[0138] The downward sliding gesture can correspond to the downward screen sliding operation or the downward page turning operation. Based on this, when the type of the air gesture is the downward sliding gesture, the intelligent perception application can perform the downward screen sliding operation or the downward page turning operation.

[0139] The leftward sliding gesture can correspond to the leftward screen sliding operation or the leftward page turning operation. Based on this, when the type of the air gesture is the leftward sliding gesture, the intelligent perception application can perform the leftward screen sliding operation or the leftward page turning operation.

[0140] The rightward sliding gesture can correspond to the rightward screen sliding operation or the rightward page turning operation. Based on this, when the type of the air gesture is the rightward sliding gesture, the intelligent perception application can perform the rightward screen sliding operation or the rightward page turning operation.

[0141] Exemplarily, please refer to Figure 7 , which is a schematic diagram of the setting scenario of the preset operation provided by the embodiment of the present application.

[0142] When the user wants to turn on or off the preset operation, or wants to set the air gesture corresponding to the preset operation, the user can input a first operation for the settings application icon on the mobile phone desktop shown in (a) of Figure 7 . The first operation can be, for example, a click operation. The mobile phone can respond to the first operation and display the settings interface shown in (b) of Figure 7 .

[0143] As shown in Figure 7As shown in (b) in [reference], the setting interface may include an assistive function setting item. The user may input a second operation for the assistive function setting item. The second operation may be, for example, a click operation. The mobile phone may, in response to the second operation, display an assistive function setting interface as shown in Figure 7 in (c).

[0144] As Figure 7 shown in (c), the assistive function setting interface may include a smart perception setting item. The user may input a third operation for the smart perception setting item. The third operation may be, for example, a click operation. The mobile phone may, in response to the third operation, display a smart perception setting interface as shown in Figure 7 in (d).

[0145] As Figure 7 shown in (d), the smart perception setting interface may include multiple preset operation setting items, such as a screen sliding setting item, an air screenshot setting item, and an air page turning setting item, etc. The user may enter the setting interface of the corresponding preset operation by clicking any one of the preset operation setting items. The user may turn on or off the corresponding preset operation in the setting interface of the preset operation, or set the air gesture corresponding to the preset operation.

[0146] Exemplarily, the user may enter Figure 7 the setting interface of the air screenshot operation as shown in (d) by clicking the air screenshot setting item, and may turn on or off the air screenshot operation in the setting interface of the air screenshot operation, or set the air gesture corresponding to the air screenshot operation, etc. Figure 7

[0147] As can be seen from the above, in this embodiment, by collecting multiple consecutive frames of images, the hand feature information of each frame of image is determined in sequence, the starting gesture image or stable state image of the air gesture is recognized according to the hand feature information of each frame of image, the hand feature information of the starting gesture image or the hand feature information of the stable state image is stored in the frame information record queue, and the air gesture is recognized only according to the hand feature information of the starting gesture image and the hand feature information of the stable state image stored in the frame information record queue, and the type of the air gesture is determined. Since the hand shape in the starting gesture image and the hand shape in the stable state image are both in a stable state, the influence of the image with the hand shape in an unstable state on the recognition accuracy of the air gesture can be avoided, and the misrecognition probability of the air gesture recognition can be reduced.

[0148] Figure 8 In a specific implementation manner, the hand feature acquisition module may include a hand target detection unit, a hand classification unit, and a hand key point detection unit. Based on this, S52 in the above embodiment may specifically include S521 to S523 as shown in Figure 8 shown below:

[0149] S521. For each frame of image in sequence according to the acquisition timing, the human hand target detection unit performs human hand target detection on the current frame of image, obtains the information of the human hand detection box corresponding to the current frame of image, and sends the information of the human hand detection box corresponding to the current frame of image to the human hand classification unit, the human hand key point detection unit, and the air gesture recognition module.

[0150] It can be understood that since it is uncertain whether there is a human hand in each frame of image collected by the AO camera, during the air gesture recognition process, after receiving each frame of image from the AO camera, the human hand feature acquisition module can first preliminarily identify the images with human hands through the human hand target detection unit and determine the information of the human hand detection box corresponding to the images with human hands. It should be noted that the specific content of the information about the human hand detection box can refer to the relevant description in the foregoing embodiments and will not be elaborated here.

[0151] In this embodiment, the purpose of the human hand target detection unit outputting the information of the human hand detection box is to enable the human hand classification unit or the human hand key point detection unit, etc. to locate the human hand in the corresponding image based on the information of the human hand detection box, facilitating subsequent operations on the image by the human hand classification unit, the human hand key point detection unit, or the air gesture recognition module, etc.

[0152] Optionally, the human hand target detection unit can perform human hand target detection on the current frame of image based on the object detection (OD) algorithm and obtain the information of the human hand detection box corresponding to the current frame of image.

[0153] S522. For each frame of image in sequence according to the acquisition timing, the human hand classification unit classifies the human hand in the current frame of image based on the information of the human hand detection box corresponding to the current frame of image, obtains the human hand category corresponding to the current frame of image, and sends the human hand category corresponding to the current frame of image to the air gesture recognition module.

[0154] Exemplarily, a pre-trained human hand classification model can be configured in the human hand classification unit. The human hand classification model can be used to classify the human hand to determine the human hand category. Based on this, for each frame of image, the human hand classification unit can determine the human hand category corresponding to the current frame of image through the pre-trained human hand classification model.

[0155] In an optional implementation manner, in order to reduce the computational complexity of air gesture recognition and improve the efficiency of air gesture recognition, for each frame of image, the human hand classification unit can crop out a human hand local image including only the human hand from the current frame of image based on the information of the human hand detection box corresponding to the current frame of image, and input the human hand local image corresponding to the current frame of image into the pre-trained human hand classification model to obtain the human hand category corresponding to the current frame of image.

[0156] Exemplarily, the hand classification model may be a neural network module trained based on a deep learning algorithm. The embodiments of the present application do not make any limitations on the specific type of the hand classification model, the training method, etc.

[0157] S523. Sequentially for each frame of image according to the acquisition time sequence, based on the information of the hand detection box corresponding to the current frame of image, the hand key point detection unit performs hand key point detection on the current frame of image, obtains the coordinates of the hand key points corresponding to the current frame of image, and sends the coordinates of the hand key points corresponding to the current frame of image to the air gesture recognition module.

[0158] Exemplarily, a pre-trained hand key point detection model may be configured in the hand key point detection unit. The hand key point detection model can be used to detect the hand key points in an image and output the coordinates of the hand key points. Based on this, for each frame of image, the hand key point detection unit can obtain the coordinates of the hand key points corresponding to the current frame of image through the pre-trained hand key point detection model.

[0159] In an alternative implementation, in order to reduce the computational complexity of air gesture recognition and improve the efficiency of air gesture recognition, for each frame of image, the hand key point detection unit can, based on the information of the hand detection box corresponding to the current frame of image, crop out a hand local image including only the hand from the current frame of image, and input the hand local image corresponding to the current frame of image into the hand key point detection model to obtain the coordinates of the hand key points corresponding to the current frame of image.

[0160] In another specific implementation, the air gesture recognition module may include an input unit, a hand stable state detection unit, a continuous same-direction sliding detection unit, a gesture hovering state detection unit, a dynamic gesture recognition unit, and a timeout detection unit.

[0161] Based on this, S53 in the above embodiments may specifically include S531 to S536 as shown in Figure 9 and is described in detail as follows:

[0162] S531. Sequentially for each frame of image according to the acquisition time sequence, the input unit determines the finger orientation corresponding to the current frame of image according to the coordinates of the hand key points corresponding to the current frame of image, and sends the information of the hand detection box corresponding to the current frame of image, the hand category, the coordinates of the hand key points, and the gesture orientation to the hand stable state detection unit.

[0163] In a specific implementation, the input unit may adopt the following steps a1 to a3 to determine the finger orientation corresponding to the current frame of image:

[0164] Step a1, the input unit determines the first coordinate of the midpoint of the line connecting the first target key point and the second target key point according to the coordinates of the first target key point and the coordinates of the second target key point among the coordinates of the hand key points corresponding to the current frame image.

[0165] Among them, the first target key point can be the hand key point corresponding to the fingertip of the middle finger, and the second target key point can be the hand key point corresponding to the fingertip of the ring finger. Exemplarily, please refer to Figure 10 , the first target key point can be the hand key point corresponding to label 12, and the second target key point can be the hand key point corresponding to label 16.

[0166] Exemplarily, the input unit can use the average value of the abscissa of the first target key point and the abscissa of the second target key point as the first abscissa of the midpoint of the line connection, and use the average value of the ordinate of the first target key point and the ordinate of the second target key point as the first ordinate of the midpoint of the line connection. For example, assume that the coordinates of the first target key point are (x 12 , y 12 ), and the coordinates of the second target key point are (y 12 , y 16 ), then the input unit can use (x 12 + x 16 ) / 2 as the first abscissa of the midpoint of the line connection, and use (y 12 + y 16 ) / 2 as the first ordinate of the midpoint of the line connection. That is, the first coordinate of the midpoint of the line connection can be [(x 12 + x 16 ) / 2, (y 12 + y 16 ) / 2].

[0167] It can be understood that since the coordinates of the hand key points corresponding to each frame of image refer to the coordinates of the corresponding hand key points in the first preset coordinate system, the above-mentioned first coordinate of the midpoint of the line connection also refers to the coordinate of the midpoint of the line connection in the first preset coordinate.

[0168] Step a2, the input unit performs coordinate conversion on the first coordinate of the midpoint of the line connection to obtain the second coordinate of the midpoint of the line connection in the second preset coordinate system, and calculates the angle between the direction vector corresponding to the midpoint of the line connection and the positive direction of the x-axis of the second preset coordinate system based on the second coordinate of the midpoint of the line connection.

[0169] Among them, the second preset coordinate system can be a plane rectangular coordinate system established with the third target key point among the hand key points as the coordinate origin, and with two rays intersecting at the third target key point and respectively parallel to two mutually perpendicular sides of the image as the x-axis and the y-axis. The third target key point can be the hand key point corresponding to the human hand wrist. For example, the second preset coordinate system can be as Figure 10In the coordinate system shown in (a) thereof, the third target key point may be the key point corresponding to label 0.

[0170] It can be understood that since the finger orientation can usually be determined by the orientation of the connection center point relative to the wrist, therefore, by converting the first coordinate of the connection center point into the second coordinate, the input unit can conveniently determine the finger orientation.

[0171] Since the second preset coordinate system is equivalent to translating the coordinate origin of the first preset coordinate system from the position of the upper left vertex of the image to the position of the third target key point, therefore, in a specific implementation manner, the input unit may determine the second coordinate of the connection center point based on the coordinate of the third target key point (i.e., the coordinate in the first preset coordinate system) and the first coordinate of the connection center point (i.e., the coordinate in the first preset coordinate system).

[0172] Specifically, the input unit may determine the difference between the first coordinate of the connection center point and the coordinate of the third target key point as the second coordinate of the connection center point, that is, the input unit may determine the difference between the first abscissa of the connection center point and the abscissa of the third target key point as the second abscissa of the connection center point, and determine the difference between the first ordinate of the connection center point and the ordinate of the third target key point as the second ordinate of the connection center point. Exemplarily, assuming that the first coordinate of the connection center point is (5, -1) and the coordinate of the third target key point is (3, -6), then the input unit may determine 5 - 3 = 2 as the second abscissa of the connection center point, and determine -1 - (-6) = 5 as the second ordinate of the connection center point, that is, the second coordinate of the connection center point is (2, 5).

[0173] After the input unit determines the second coordinate of the connection center point, it may use the vector pointing from the coordinate origin of the second preset coordinate system to the connection midpoint as the direction vector corresponding to the connection midpoint. The direction indicated by the direction vector can be used to represent the finger orientation.

[0174] In a specific embodiment, assuming that the second coordinate of the connection center point is (x z1 , y z1 ), then the input unit may calculate the angle between the direction vector and the positive x-axis direction of the second preset coordinate system by using the following angle calculation formula:

[0175] θ = arctan2(y z1 , x z1 ) × (180 / π);

[0176] where θ may be the angle between the direction vector and the positive x-axis direction, and arctan2 may be the arctangent function.

[0177] Step a3, the input unit determines the finger orientation according to the angle between the direction vector corresponding to the midpoint of the connection line and the positive x-axis direction of the second preset coordinate system.

[0178] In a specific implementation, multiple corresponding relationships between preset angle ranges and finger orientations can be pre-stored in the mobile phone. Combining Figure 10 with (b) in [reference], the corresponding relationships between preset angle ranges and finger orientations can be as shown in Table 1.

[0179] Table 1

[0180] Preset Angle Range (unit: degree) Finger Orientation [-135,-45) Upward [-135,-180]∪[135,180] Leftward [45,135) Downward [-45,45) Rightward

[0181] Based on this, the input unit can first determine the target angle range where the angle between the direction vector corresponding to the midpoint of the connection line and the positive x-axis direction is located from multiple preset angle ranges, and then determine the target finger orientation corresponding to the target angle range according to the corresponding relationship between the preset angle range and the finger orientation, and determine the target finger orientation as the finger orientation corresponding to the corresponding image.

[0182] Exemplarily, if the angle between the direction vector corresponding to the midpoint of the connection line and the positive x-axis direction is -85 degrees, then according to the corresponding relationship between the preset angle range and the finger orientation shown in Table 1, the target angle range where -85 degrees is located can be determined as [-135, -45). Based on this, the input unit can determine the target finger orientation (upward) corresponding to the target angle range [-135, -45) as the finger orientation corresponding to the corresponding image.

[0183] S532, the human hand stable state detection unit determines whether any starting gesture of a dynamic gesture is included in the current frame image, or whether the hand shape in the current frame image is stable; in the case where the starting gesture is included in the current frame image, store the human hand feature information and detection time of the current frame image in the frame information record queue, and enter the continuous same-direction sliding detection process; in the case where the hand shape in the current frame image is stable, store the human hand feature information and detection time of the current frame image in the frame information record queue, and enter the gesture hovering state detection process; in the case where the starting gesture image is not included in the current frame image and the hand shape in the current frame image is unstable, discard the human hand feature information of the current frame image.

[0184] Discarding the human hand feature information of the current frame image may mean not storing the human hand feature information of the current frame image.

[0185] Among them, the human hand feature information of the current frame image stored by the human hand stable state detection unit in the frame information record queue may include: the information of the human hand detection box corresponding to the current frame image, the human hand category, the coordinates of the human hand key points, and the finger orientation.

[0186] Optionally, in the case that the current frame image includes the starting gesture, the detection time of the current frame image may be the time when the human hand stability state detection unit detects that the current frame image includes the starting gesture.

[0187] Optionally, when the hand shape of the current frame image is stable, the detection time of the current frame image may be a time when the hand shape of the current frame image is stable as determined by the human hand stability state detection unit.

[0188] The continuous same-direction sliding detection process may be a process executed by the continuous same-direction sliding detection unit in the subsequent S533 .

[0189] The gesture hovering state detection process may be a process executed by the gesture hovering state detection unit in the subsequent S534 .

[0190] In a specific implementation, S532 may specifically include: Figure 11 S5321 to S5325 shown are described in detail as follows:

[0191] S5321, the hand stability state detection unit determines whether the frame information recording queue is empty.

[0192] Optionally, when the frame information recording queue is empty, the hand stability state detection unit can execute S5322.

[0193] Optionally, when the frame information recording queue is not empty, the hand stability state detection unit may execute S5323.

[0194] S5322, the hand stability state detection unit determines whether the current frame image includes any starting gesture of an air gesture based on the hand feature information of the current frame image.

[0195] It can be understood that, since the air gesture recognition module deletes the hand feature information and detection information of all images related to the air gesture from the frame information recording queue after each air gesture is recognized, when the frame information recording queue is empty, it means that the current frame image may be the starting gesture of the next air gesture. Based on this, when the frame information recording queue is empty, the hand stable state detection unit can determine whether the current frame image includes any starting gesture of the air gesture based on the hand feature information of the current frame image.

[0196] Combination Figure 6 It can be seen that the starting gestures of the air gesture can include the following types: palm with fingertips facing up, back of hand with fingertips facing down, palm with fingertips facing left, back of hand with fingertips facing left, palm with fingertips facing right, and back of hand with fingertips facing right.

[0197] Among them, the palm with the fingertips facing upward can be the starting gesture for pressing gestures, grasping gestures, flipping gestures, and swiping-down gestures; the palm with the fingertips facing right can be the starting gesture for swiping-left gestures; the palm with the fingertips facing left can be the starting gesture for swiping-right gestures; the back of the hand with the fingertips facing downward can be the starting gesture for swiping-up gestures; the back of the hand with the fingertips facing right can be the starting gesture for swiping-left gestures; the back of the hand with the fingertips facing left can be the starting gesture for swiping-right gestures.

[0198] It can be seen that the starting gesture is related to the human hand type and the finger orientation. Exemplarily, the relationship between the starting gesture and the human hand type and the finger orientation can be as shown in Table 2.

[0199] Table 2

[0200]

[0201] Based on this, in a specific implementation, the human hand stable state detection unit can determine whether the current frame image includes the starting gesture of any air gesture according to the human hand type and the finger orientation corresponding to the current frame image.

[0202] Optionally, the human hand stable state detection unit can determine that the current frame image includes the starting gesture of any air gesture in any of the following cases:

[0203] (1) The human hand type corresponding to the current frame image is a palm, and the finger orientation is upward, left, or right.

[0204] (2) The human hand type corresponding to the current frame image is the back of the hand, and the finger orientation is downward, left, or right.

[0205] Optionally, the human hand stable state detection unit can determine that the previous frame image does not include the starting gesture of any air gesture in any of the following cases, that is, determine that the current frame image is a non-starting gesture image:

[0206] (1) The human hand type corresponding to the current frame image is neither a palm nor the back of the hand.

[0207] (2) The human hand type corresponding to the current frame image is a palm, and the finger orientation is not upward, left, or right.

[0208] (3) The human hand type corresponding to the current frame image is the back of the hand, and the finger orientation is not downward, left, or right.

[0209] In a specific implementation, S5322 can include S5322.1 to S5322.6 as shown in Figure 12 and are described in detail as follows:

[0210] S5322.1, determine whether the human hand type corresponding to the current frame image is a palm.

[0211] Optionally, when the hand category corresponding to the current frame image is a palm, S5322.2 can be executed.

[0212] Optionally, when the hand category corresponding to the current frame image is not a palm, S5322.3 can be executed.

[0213] In S5322.2, determine whether the finger orientation corresponding to the current frame image is upward, or leftward, or rightward.

[0214] Optionally, when the finger orientation corresponding to the current frame image is upward, or leftward, or rightward, S5322.5 can be executed.

[0215] Optionally, when the finger orientation corresponding to the current frame image is not upward, not leftward, and not rightward, S5322.6 can be executed.

[0216] In S5322.3, determine whether the hand category corresponding to the current frame image is the back of the hand.

[0217] Optionally, when the hand category corresponding to the current frame image is the back of the hand, S5322.4 can be executed.

[0218] Optionally, when the hand category corresponding to the current frame image is not the back of the hand, S5322.6 can be executed.

[0219] In S5322.4, determine whether the finger orientation corresponding to the current frame image is downward, or leftward, or rightward.

[0220] Optionally, when the finger orientation corresponding to the current frame image is downward, or leftward, or rightward, S5322.5 can be executed.

[0221] Optionally, when the finger orientation corresponding to the current frame image is not downward, not leftward, and not rightward, S5322.6 can be executed.

[0222] In S5322.5, determine that the current frame image includes a starting gesture.

[0223] In S5322.6, determine that the current frame image does not include a starting gesture.

[0224] It should be noted that Figure 12Only one optional starting gesture image judgment process is shown. In other embodiments, when the human hand stable state detection unit recognizes the starting gesture image, it can first determine whether the human hand category corresponding to the current frame image is the back of the hand, and then determine whether the human hand category corresponding to the current frame image is the palm; or, it can first determine the finger orientation corresponding to the current frame image, and then determine the human hand category corresponding to the current frame image. All of these are within the protection scope of this application.

[0225] Optionally, when the current frame image is the starting gesture image of any air gesture, the human hand stable state detection unit can execute S5324 and enter the continuous same-direction sliding detection process in S533.

[0226] Optionally, when the current frame image is not the starting gesture image of any air gesture, the human hand stable state detection unit can execute S5325.

[0227] S5323. The human hand stable state detection unit determines whether the hand shape in the current frame image is stable based on the human hand feature information of the current frame image.

[0228] In a specific implementation manner, the human hand stable state detection unit can use the following steps b1 to b3 to determine whether the hand shape in the current frame image is stable, which is described in detail as follows:

[0229] Step b1. According to the information of the human hand detection box corresponding to the current frame image and the information of the human hand detection box corresponding to the previous frame image, calculate the intersection over union (IoU) of the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image.

[0230] Specifically, the human hand stable state detection unit can calculate the intersection area and union area of the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image based on the information of the human hand detection box corresponding to the current frame image and the information of the human hand detection box corresponding to the previous frame image, and determine the ratio of the intersection area to the union area as the intersection over union (IoU) of the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image.

[0231] It should be noted that since the calculation methods of the intersection area and union area of two rectangular boxes with known position information are prior arts, the specific calculation methods of the intersection area and union area of the human hand detection box corresponding to the current frame image and the human hand detection box corresponding to the previous frame image are not described in detail here.

[0232] Step b2. According to the coordinates of the human hand key points corresponding to the current frame image and the coordinates of the human hand key points corresponding to the previous frame image, calculate the standard deviation of the first-order difference between the human hand key points corresponding to the current frame image and the human hand key points corresponding to the previous frame image.

[0233] It should be noted that since the coordinates of the hand key points corresponding to the current frame image and the coordinates of the hand key points corresponding to the previous frame image are both coordinate arrays composed of the coordinates of 21 hand key points, and the calculation method of the standard deviation of the first-order difference between the two arrays is a prior art, the specific calculation method of the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image will not be elaborated here.

[0234] Step b3: Determine whether the hand form in the current frame image is stable according to the hand category corresponding to the current frame image, the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image, the finger orientation corresponding to the previous frame image, the intersection over union of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image, and the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image.

[0235] Optionally, when the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, the intersection over union of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the first preset intersection over union, and the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is less than or equal to the preset standard deviation, the hand stable state detection unit may determine that the hand form in the current frame image is stable.

[0236] Optionally, in any of the following cases, the hand stable state detection unit may determine that the hand form in the current frame image is unstable, that is, determine that the current frame image is an unstable state image:

[0237] (1) The hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image.

[0238] (2) The finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image.

[0239] (3) The intersection over union of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the first preset intersection over union.

[0240] (4) The standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is greater than the preset standard deviation.

[0241] Among them, both the first preset intersection over union and the preset standard deviation can be set according to actual requirements.

[0242] In specific implementation, step b3 may include as Figure 13S5323.1 to S5323.6 shown are described in detail as follows:

[0243] S5323.1 determines whether the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image.

[0244] Optionally, when the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, S5323.2 can be executed.

[0245] Optionally, when the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, S5323.6 can be executed.

[0246] S5323.2 determines whether the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image.

[0247] Optionally, when the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, S5323.3 can be executed.

[0248] Optionally, when the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, S5323.6 can be executed.

[0249] S5323.3 determines whether the intersection over union (IoU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to a first preset IoU.

[0250] Optionally, when the IoU of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the first preset IoU, S5323.4 can be executed.

[0251] Optionally, when the IoU of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the first preset IoU, S5323.6 can be executed.

[0252] S5323.4 determines whether the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is less than or equal to a preset standard deviation.

[0253] Optionally, when the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is less than or equal to the preset standard deviation, S5323.5 can be executed.

[0254] Optionally, when the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is greater than the preset standard deviation, S5323.6 can be executed.

[0255] S5323.5, Determine that the hand form in the current frame image is stable.

[0256] S5323.6, Determine that the hand form in the current frame image is unstable.

[0257] Optionally, when the hand form in the current frame image is stable, the hand stable state detection unit may execute S5324 and enter the gesture hover state detection process.

[0258] Optionally, when the hand form in the current frame image is unstable, that is, the current frame image is an unstable state image, the hand stable state detection unit may execute S5325.

[0259] S5324, The hand stable state detection unit stores the hand feature information corresponding to the current frame image and the detection time in the frame information record queue in an associated manner.

[0260] In addition, when the current frame image is a starting gesture image, the hand stable state detection unit may also store the type of the starting gesture corresponding to the current frame image in the frame information record queue in an associated manner.

[0261] S5325, The hand stable state detection unit discards the hand feature information of the current frame image.

[0262] S533, The continuous same-direction sliding detection unit obtains the recognition time and type of the previous air gesture that has been recognized, and determines whether the current meets the continuous same-direction sliding short-term shielding condition according to the recognition time of the previous air gesture, the type of the previous air gesture, the detection time of the current frame image, and the type of the starting gesture corresponding to the current frame image; when the current meets the continuous same-direction sliding short-term shielding condition, the dynamic gesture recognition process is not executed, and the corresponding hand shape icon is not displayed; when the current does not meet the continuous same-direction sliding short-term shielding condition, the dynamic gesture recognition process is executed, and the hand feature information of the current frame image is sent to the message manager.

[0263] Optionally, in any of the following cases, the continuous same-direction sliding detection unit may determine that the current does not meet the continuous same-direction sliding short-term shielding condition:

[0264] (1) The type of the previous air gesture is not a sliding gesture.

[0265] (2) The type of the previous air gesture is a sliding gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is greater than the first duration.

[0266] (3) The type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture.

[0267] (4) The type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, and the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration.

[0268] Among them, the first duration can be used to represent the recovery time limit of the starting gesture of the target swipe gesture in the scenario of continuously swiping the screen in the same direction in the air. Exemplarily, the first duration can be 2 seconds.

[0269] The second duration can be less than the first duration. Exemplarily, the second duration can be 800 milliseconds.

[0270] It can be understood that for the above situation (1), when the type of the previous air gesture is not a swipe gesture, it indicates that the user is definitely not performing a continuous swiping operation in the same direction, so the dynamic gesture recognition process needs to be executed normally.

[0271] For the above situation (2), in the scenario of continuously swiping the screen in the same direction in the air, the process of the user returning to the starting gesture of the target swipe gesture again after executing a target swipe gesture is relatively rapid, and the duration usually will not be greater than the first duration. Therefore, if the starting gesture image is detected after the first duration after the previous swipe gesture is executed, it indicates that the user does not want to perform a continuous swiping operation in the same direction, so the dynamic gesture recognition process needs to be executed normally.

[0272] For the above situation (3), this situation indicates that the user has quickly returned to the starting gesture of the target swipe gesture after executing a target swipe gesture, that is, it indicates that the user is currently performing a continuous air swipe operation and has returned to the starting gesture of the target swipe gesture. Therefore, the dynamic gesture recognition process needs to be executed normally so that the dynamic gesture recognition unit can perform the next round of dynamic gesture recognition.

[0273] For the above-mentioned situation (4), it indicates that after the user executes a target sliding gesture and within the starting gesture recovery time limit of the target sliding gesture, the user deliberately maintains another starting gesture different from the starting gesture of the target sliding gesture for at least a second duration, which means that the user may want to switch to another different type of air gesture. For example, the user wants to switch from a downward sliding gesture to an upward sliding gesture. Therefore, this situation needs to normally execute the dynamic gesture recognition process so that the dynamic gesture recognition unit can recognize the new air gesture switched by the user.

[0274] Optionally, when the type of the previous air gesture is a sliding gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, and the duration of the stable state corresponding to the current frame image is less than the second duration, the continuous same-direction sliding detection unit can determine that the current satisfies the continuous same-direction sliding short-time shielding condition.

[0275] It can be understood that when the continuous same-direction sliding short-time shielding condition is satisfied, it means that the current is in the process of recovering the starting gesture of the target sliding gesture. To prevent the dynamic gesture recognition unit from recognizing a sliding gesture opposite to the target sliding gesture, the corresponding hand icon can be not displayed, and the dynamic gesture recognition process can be not executed, so that the dynamic gesture recognition unit does not use the frame images in the process of recovering the starting gesture of the target sliding gesture for dynamic gesture recognition. In this way, the situation of back-and-forth page turning can be avoided, and the operation experience when the user slides the screen continuously in the same direction in the air can be improved.

[0276] In a specific implementation manner, the continuous same-direction sliding detection unit can use, for example, Figure 14 as shown in S5331 to S5336 to determine whether the current satisfies the continuous same-direction sliding short-time shielding condition, which is described in detail as follows:

[0277] S5331, determine whether the previous air gesture is a sliding gesture.

[0278] Optionally, when the previous air gesture is a sliding gesture, S5332 can be executed.

[0279] Optionally, when the previous air gesture is not a sliding gesture, S5335 can be executed.

[0280] S5332, determine whether the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration.

[0281] Optionally, when the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, S5333 can be executed.

[0282] Optionally, when the interval between the detection time of the current frame image and the recognition time of the previous air gesture is greater than the first duration, S5335 can be executed.

[0283] S5333, determine whether the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture.

[0284] Optionally, when the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture, S5335 can be executed.

[0285] Optionally, when the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, S5334 can be executed.

[0286] S5334, determine whether the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration.

[0287] In a specific embodiment, it can be determined whether the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration by determining whether the gesture in the image captured by the AO camera within the second duration after the acquisition moment of the current frame image is the same as the starting gesture in the current frame image. In the case of being the same, it can be determined that the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration; in the case of being different, it can be determined that the duration of the stable state corresponding to the current frame image is less than the second duration.

[0288] Optionally, when the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration, S5335 can be executed.

[0289] Optionally, when the duration of the stable state corresponding to the current frame image is less than the second duration, S5336 can be executed.

[0290] S5335, display the hand shape icon corresponding to the starting gesture in the current frame image and enter the dynamic gesture recognition process.

[0291] S5336, do not display the hand shape icon corresponding to the starting gesture in the current frame image and do not execute the dynamic gesture recognition process.

[0292] It can be understood that when the condition of short-term shielding for continuous same-direction sliding is not currently met, the purpose of the continuous same-direction sliding detection unit to send the hand feature information of the current frame image to the message manager is to enable the intelligent perception application to obtain the hand feature information of the current frame image from the message manager, and then display a hand icon that matches the starting gesture corresponding to the current frame image. The hand icon can be used to represent the type of the starting gesture corresponding to the current frame image.

[0293] In an alternative implementation, since the type of the starting gesture can be determined by the hand category and finger orientation corresponding to the image, the hand feature information of the current frame image sent by the continuous same-direction sliding detection unit to the message manager may include the hand category and finger orientation corresponding to the current frame image.

[0294] Exemplarily, as Figure 15 shown, assuming that the starting gesture corresponding to the current frame image is a palm with fingers facing up and the condition of short-term shielding for continuous same-direction sliding is not currently met, the hand feature information of the current frame image sent by the continuous same-direction sliding detection unit to the message manager may include: Hand category: Palm, Finger orientation: Upward. At this time, the intelligent perception application can display Figure 15 the palm icon with fingers facing up shown in 151 on the display screen.

[0295] S534. The gesture hovering state detection unit determines whether the current frame image is in the gesture hovering state based on the hand feature information of the current frame image; if the current is in the gesture hovering state, enter the timeout detection process; if the current is not in the gesture hovering state, enter the dynamic gesture recognition process.

[0296] Among them, the gesture hovering state may refer to a state where the hand shape remains unchanged for a third duration threshold. The third duration threshold can be set according to actual needs. For example, the third duration threshold can be 900 milliseconds.

[0297] The dynamic gesture recognition process may refer to the process executed by the dynamic gesture recognition unit in S535.

[0298] The timeout detection process may refer to the process executed by the timeout detection unit in S536.

[0299] In a specific implementation, the gesture hovering state detection unit may use the following steps c1 to c3 to determine whether the current is in the gesture hovering state, which is described in detail as follows:

[0300] Step c1, according to the information of the hand detection box corresponding to the current frame image and the information of the hand detection box corresponding to the previous frame image, calculate the intersection over union of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image.

[0301] It should be noted that the specific calculation method of the intersection over union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image can refer to the relevant description in step b1 of the foregoing embodiments, and will not be elaborated here.

[0302] Step c2, according to the information of the hand detection box corresponding to the current frame image, calculate the first ratio of the hand detection box corresponding to the current frame image in the current image, according to the information of the hand detection box corresponding to the previous frame image, calculate the second ratio of the hand detection box corresponding to the previous frame image in the previous frame image, and calculate the difference between the first ratio and the second ratio.

[0303] Specifically, the gesture hovering state detection unit can calculate the area of the hand detection box corresponding to the current frame image according to the information of the hand detection box corresponding to the current frame image, and determine the ratio of the area of the hand detection box corresponding to the current frame image to the area of the current frame image as the first ratio.

[0304] Similarly, the gesture hovering state detection unit can calculate the area of the hand detection box corresponding to the previous frame image according to the information of the hand detection box corresponding to the previous frame image, and determine the ratio of the area of the hand detection box corresponding to the previous frame image to the area of the previous frame image as the second ratio.

[0305] Step c3, according to the hand category corresponding to the current frame image, the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image, the finger orientation corresponding to the previous frame image, the intersection over union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image, and the difference between the first ratio and the second ratio, determine whether the current is in the gesture hovering state.

[0306] Optionally, when the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection over union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the second preset intersection over union ratio, and the difference between the first ratio and the second ratio is less than or equal to the preset difference, the gesture hovering state detection unit can determine that the current is in the gesture hovering state.

[0307] It can be understood that when the current is in the gesture hovering state, it means that the user deliberately maintains the starting gesture of a certain air gesture and does not want to immediately execute the subsequent gesture of the air gesture. Therefore, in this case, the dynamic gesture recognition process can be executed, and the hand shape icon corresponding to the corresponding start is displayed, so as to quickly respond to the air gesture after the user inputs the subsequent gesture of the air gesture.

[0308] Optionally, in any of the following cases, the gesture hovering state detection unit may determine that the current state is not a gesture hovering state, that is, the current state is a non-gesture hovering state:

[0309] (1) The hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image.

[0310] (2) The finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image.

[0311] (3) The intersection over union (IoU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than a second preset IoU.

[0312] (4) The difference between the first ratio and the second ratio is greater than a preset difference.

[0313] Among them, the second preset IoU may be greater than the first preset IoU.

[0314] The preset difference can be set according to actual needs and is not limited herein.

[0315] In specific implementation, step c3 may include S5341 to S5346 as shown in Figure 16 and are detailed as follows:

[0316] S5341, determine whether the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image.

[0317] Optionally, in the case where the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, S5342 may be executed.

[0318] Optionally, in the case where the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, S5346 may be executed and the dynamic gesture recognition process in S535 may be entered.

[0319] S5342, determine whether the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image.

[0320] Optionally, in the case where the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, S5343 may be executed.

[0321] Optionally, in the case where the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, S5346 may be executed and the dynamic gesture recognition process in S535 may be entered.

[0322] S5343, determine whether the intersection over union (IoU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the second preset IoU.

[0323] Optionally, when the intersection over union (IoU) between the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to a second preset IoU, S5344 can be executed.

[0324] Optionally, when the intersection over union (IoU) between the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the second preset IoU, S5346 can be executed, and the dynamic gesture recognition process in S535 can be entered.

[0325] S5344, determine whether the difference between the first ratio and the second ratio is less than or equal to a preset difference.

[0326] Optionally, when the difference between the first ratio and the second ratio is less than or equal to the preset difference, S5345 can be executed, and the timeout detection process in S536 can be entered.

[0327] Optionally, when the difference between the first ratio and the second ratio is greater than the preset difference, S5346 can be executed, and the dynamic gesture recognition process in S535 can be entered.

[0328] S5345, determine that the current is in the gesture hovering state.

[0329] S5346, determine that the current is in the non-gesture hovering state.

[0330] S535, the dynamic gesture recognition unit determines whether there is an air gesture currently based on the hand feature information of multiple frames of images in the frame information record queue; when there is an air gesture currently, send the type of the air gesture to the message manager, and delete the hand feature information of all images related to the current air gesture from the frame information record queue; when there is no air gesture currently, enter the timeout detection process.

[0331] In a specific implementation, the dynamic gesture recognition unit can determine whether there are consecutive multiple frames of images in the frame information record queue that conform to the hand shape change characteristics of any air gesture based on the recognition strategies of each air gesture. Among them, the preset strategy of each air gesture can be configured according to the hand shape change characteristics of the air gesture.

[0332] Optionally, when there are consecutive multiple frames of images in the frame information record queue that conform to the hand shape change characteristics of the target air gesture, the dynamic gesture recognition unit can determine that there is an air gesture currently, and determine the type of the target air gesture as the type of the current air gesture. For example, assuming that there are consecutive multiple frames of images in the frame information record queue that conform to the hand shape change characteristics of the upward swipe gesture, the air gesture recognition module can determine that there is an upward swipe gesture currently.

[0333] Optionally, when there are no consecutive multiple frames of images in the frame information recording queue that conform to the hand shape change characteristics of any air gesture, the air gesture recognition module may determine that there is no air gesture currently.

[0334] In this embodiment, after the dynamic gesture recognition unit determines the type of the air gesture, by deleting the hand feature information of all images related to the current air gesture from the frame information recording queue, not only can the storage space of the mobile phone be saved, but also it is beneficial for the recognition of the next round of air gestures.

[0335] In addition, after the dynamic gesture recognition unit determines the type of the air gesture, it may update the recognition time and type of the most recent air gesture in the second storage area to the recognition time and type of the current air gesture, which is convenient for the detection of consecutive same-direction sliding gestures in the next round of air gesture recognition process.

[0336] In a specific implementation manner, the dynamic gesture recognition unit may sequentially determine whether there is a corresponding air gesture currently according to the preset priorities of each air gesture, so as to improve the efficiency of air gesture recognition.

[0337] Among them, the preset priorities of each air gesture may be determined according to the execution frequencies of each air gesture. Exemplarily, the more frequently an air gesture is used, the higher its preset priority may be. In a specific application, the execution frequencies of each air gesture may be obtained by counting the usage situations of air gestures of each user.

[0338] Assume that the preset priorities of each air gesture are sorted from high to low as: grasping gesture, sliding gesture, flipping gesture, and pressing gesture. Then S535 may include S5351 to S5358 as Figure 17 shown below, which are described in detail as follows:

[0339] S5351, based on the recognition strategy of the grasping gesture, determine whether there are consecutive multiple frames of images in the frame information recording queue that conform to the hand shape change characteristics of the grasping gesture.

[0340] Optionally, when there are consecutive multiple frames of images in the frame information recording queue that conform to the hand shape change characteristics of the grasping gesture, S5355 may be executed.

[0341] Optionally, when there are no consecutive multiple frames of images in the frame information recording queue that conform to the hand shape change characteristics of the grasping gesture, S5352 may be executed.

[0342] S5352, based on the recognition strategy of the sliding gesture, determine whether there are consecutive multiple frames of images in the frame information recording queue that conform to the hand shape change characteristics of the sliding gesture.

[0343] Optionally, when there are multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a sliding gesture, S5356 can be executed.

[0344] Optionally, when there are no multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a sliding gesture, S5353 can be executed.

[0345] S5353: Based on the recognition strategy for a flipping gesture, determine whether there are multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a flipping gesture.

[0346] Optionally, when there are multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a flipping gesture, S5357 can be executed.

[0347] Optionally, when there are no multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a flipping gesture, S5354 can be executed.

[0348] S5354: Based on the recognition strategy for a pressing gesture, determine whether there are multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a pressing gesture.

[0349] Optionally, when there are multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a pressing gesture, S5358 can be executed.

[0350] Optionally, when there are no multiple consecutive frames of images in the frame information recording queue that conform to the characteristics of hand shape changes for a pressing gesture, S5359 can be executed.

[0351] S5355: Determine that there is a current air gesture, and the type of the current air gesture is a grasping gesture.

[0352] S5356: Determine that there is a current air gesture, and the type of the current air gesture is a sliding gesture.

[0353] S5357: Determine that there is a current air gesture, and the type of the current air gesture is a flipping gesture.

[0354] S5358: Determine that there is a current air gesture, and the type of the current air gesture is a pressing gesture.

[0355] S5359: Determine that there is no current air gesture.

[0356] S536: The timeout detection unit determines whether the current timeout condition is met; if the current timeout condition is met, exit the air gesture recognition process.

[0357] Among them, the air gesture recognition process may include all processes executed in the air gesture recognition module.

[0358] Optionally, in any of the following cases, the timeout detection unit may determine that the current timeout condition is met:

[0359] (1) The starting gesture image is not detected by the human hand stability state detection unit for N consecutive frames.

[0360] (2) The stable state image is not detected by the human hand stability state detection unit for L consecutive frames.

[0361] (3) The gesture hovering state is detected by the gesture hovering state detection unit for three consecutive time periods.

[0362] Among them, the values of N and L can be set according to the actual situation. For example, both N and L can be 10.

[0363] In the case of meeting the timeout condition, the power consumption of the mobile phone can be saved by exiting the gesture recognition process.

[0364] Please refer to Figure 18 , which is a schematic flowchart of an air gesture recognition method provided by another embodiment of the present application. As Figure 18 shown, the air gesture recognition method may include S181 to S182, which are described in detail as follows:

[0365] S181, after recognizing a swipe gesture at the first moment, the front camera captures a first image at the second moment; the second moment is later than the first moment.

[0366] Among them, the swipe gesture may include: an upward swipe gesture, a downward swipe gesture, a left swipe gesture, or a right swipe gesture.

[0367] S182, if it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first time period, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image is not displayed.

[0368] Among them, the starting gesture may include: a palm with fingers facing up, a palm with fingers facing left, a palm with fingers facing right, a back of the hand with fingers facing down, a back of the hand with fingers facing left, or a back of the hand with fingers facing right.

[0369] It should be noted that the specific content of S182 can refer to the relevant description in S533, and will not be elaborated here.

[0370] In another embodiment of the present application, after S181, it may further include:

[0371] If it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

[0372] It should be noted that the specific content of this step can be referred to the relevant description in S533, and will not be elaborated here.

[0373] In another embodiment of the present application, after S181, it may further include:

[0374] If it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is the same as the starting gesture of the sliding gesture, then a hand - shaped icon corresponding to the starting gesture in the first image is displayed, and the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information record queue.

[0375] It should be noted that the specific content of this step can be referred to the relevant description in S533, and will not be elaborated here.

[0376] In another embodiment of the present application, after S181, it may further include:

[0377] If it is detected at the third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, and the gesture in the third image captured by the front - facing camera within the second duration after the second moment is the same as the starting gesture in the first image, then a hand - shaped icon corresponding to the starting gesture in the first image is displayed, and the dynamic gesture recognition process is executed, and the information of the first image and the information of the third image are stored in the frame information record queue; the second duration is less than the first duration.

[0378] It should be noted that the specific content of this step can be referred to the relevant description in S533, and will not be elaborated here.

[0379] In another embodiment of the present application, after S181, it may further include:

[0380] If the interval between the third moment and the first moment is greater than the first duration, then a hand - shaped icon corresponding to the starting gesture in the first image is displayed, and the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information record queue.

[0381] It should be noted that the specific content of this step can be referred to the relevant description in S533, and will not be elaborated here.

[0382] In yet another embodiment of the present application, the air gesture recognition method may further include:

[0383] If other dynamic gestures except the sliding gesture are recognized at the first moment, execute the dynamic gesture recognition process.

[0384] It should be noted that the specific content of this step can refer to the relevant description in S533, and will not be elaborated here.

[0385] In yet another embodiment of the present application, before S181, it may further include:

[0386] Recognize dynamic gestures according to the information of multiple frames of images in the frame information recording queue; the frame information recording queue is used to store the information of the starting gesture image and the stable state image, the starting gesture image is an image including the starting gesture, and the stable state image is a non-starting gesture image with a stable hand form.

[0387] Based on this, after S181, it may further include:

[0388] Delete the information of all images related to the sliding gesture in the frame information recording queue.

[0389] It should be noted that the content of this step can refer to the relevant description in S535, and will not be elaborated here.

[0390] In yet another embodiment of the present application, before recognizing dynamic gestures according to the information of multiple frames of images in the frame information recording queue, it may further include:

[0391] Obtain a continuous multi-frame image, and the continuous multi-frame image is collected by the front camera;

[0392] For each frame image in the continuous multi-frame image in sequence, determine the hand feature information of the current frame image;

[0393] According to the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand form in the current frame image is stable;

[0394] When the current frame image includes the starting gesture image, or when the hand form in the current frame image is stable, store the information of the current frame image in the frame information recording queue; the information of the current frame image includes the hand feature information.

[0395] It should be noted that the content of this step can refer to the relevant description in S51 - S53, and will not be elaborated here.

[0396] In yet another embodiment of the present application, according to the human hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand form in the current frame image is stable, including:

[0397] When the frame information recording queue is empty, according to the human hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture;

[0398] When the frame information recording queue is not empty, according to the human hand feature information of the current frame image, it is determined whether the hand form in the current frame image is stable.

[0399] It should be noted that the content of this step can refer to the relevant description in S532 and will not be elaborated here.

[0400] In yet another embodiment of the present application, the human hand feature information includes the human hand category and the finger orientation; according to the human hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture, including:

[0401] When the human hand category corresponding to the current frame image is the palm and the finger orientation is upward, or leftward, or rightward, it is determined that the current frame image includes the starting gesture; or,

[0402] When the human hand category corresponding to the current frame image is the back of the hand and the finger orientation is downward, or leftward, or rightward, it is determined that the current frame image includes the starting gesture.

[0403] In yet another embodiment of the present application, the human hand feature information includes the human hand category and the finger orientation; according to the human hand feature information of the current frame image, it is determined whether the current frame image includes the starting gesture of any dynamic gesture, including:

[0404] When the human hand category corresponding to the current frame image is the palm and the finger orientation is not upward, not leftward, and not rightward, it is determined that the current frame image does not include the starting gesture; or,

[0405] When the human hand category corresponding to the current frame image is the back of the hand and the finger orientation is not downward, not leftward, and not rightward, it is determined that the current frame image does not include the starting gesture; or,

[0406] When the human hand category corresponding to the current frame image is not the palm and not the back of the hand, it is determined that the current frame image does not include the starting gesture.

[0407] In another embodiment of the present application, the human hand feature information includes the information of the human hand detection frame, the human hand category, the coordinates of the human hand key points, and the finger orientation; judging whether the hand shape in the current frame image is stable according to the human hand feature information of the current frame image, including:

[0408] Calculating the intersection over union of the human hand detection frame corresponding to the current frame image and the human hand detection frame corresponding to the previous frame image;

[0409] Calculating the standard deviation of the first-order difference between the human hand key points corresponding to the current frame image and the human hand key points corresponding to the previous frame image;

[0410] When the human hand category corresponding to the current frame image is the same as the human hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection over union is greater than or equal to the first preset intersection over union, and the standard deviation of the first-order difference is less than or equal to the preset standard deviation, it is determined that the hand shape in the current frame image is stable.

[0411] In another embodiment of the present application, the air gesture recognition method further includes:

[0412] When the human hand category corresponding to the current frame image is different from the human hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the intersection over union is less than the first preset intersection over union, or the standard deviation of the first-order difference is greater than the preset standard deviation, it is determined that the hand shape in the current frame image is unstable.

[0413] In another embodiment of the present application, when it is determined that the hand shape in the current frame image is stable, it further includes:

[0414] Judging whether the current is in a gesture hovering state according to the human hand feature information of the current frame image and the human hand feature information of the previous frame image;

[0415] When not in the gesture hovering state currently, execute the dynamic gesture recognition process.

[0416] In another embodiment of the present application, the human hand feature information includes the information of the human hand detection frame, the human hand category, the coordinates of the human hand key points, and the finger orientation; judging whether the current is in a gesture hovering state according to the human hand feature information of the current frame image and the human hand feature information of the previous frame image, including:

[0417] Calculating the intersection over union of the human hand detection frame corresponding to the current frame image and the human hand detection frame corresponding to the previous frame image;

[0418] Calculating the first ratio of the human hand detection frame corresponding to the current frame image in the current frame image according to the information of the human hand detection frame corresponding to the current frame image;

[0419] According to the information of the human hand detection box corresponding to the previous frame image, calculate the second proportion of the human hand detection box corresponding to the previous frame image in the previous frame image;

[0420] Calculate the difference between the first proportion and the second proportion;

[0421] If the human hand category corresponding to the current frame image is different from the human hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the intersection over union is less than the second preset intersection over union, or the difference between the first proportion and the second proportion is greater than the preset difference, it is determined that the current is not in the gesture hovering state.

[0422] In another embodiment of the present application, it further includes:

[0423] If the human hand category corresponding to the current frame image is the same as the human hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection over union is greater than or equal to the second preset intersection over union, and the difference between the first proportion and the second proportion is less than or equal to the preset difference, it is determined that the current is in the gesture hovering state.

[0424] It should be noted that the content of this step can refer to the relevant description in S534, and will not be elaborated here.

[0425] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer-executable program. When the computer-executable program is called by a computer, the computer executes one or more steps in any of the above method embodiments.

[0426] Based on the same technical concept, an embodiment of the present application also provides a chip system, including a processor, the processor is coupled to a memory, and the processor executes the computer-executable program stored in the memory to implement one or more steps in any of the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0427] Based on the same technical concept, an embodiment of the present application also provides a computer-executable program product. When the computer-executable program product runs on an electronic device, the electronic device is enabled to execute one or more steps in any of the above method embodiments.

[0428] In the above embodiments, the descriptions of the various embodiments each have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0429] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0430] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by computer programs instructing relevant hardware. The programs can be stored in a computer-readable storage medium. When the programs are executed, they can include the processes of the above method embodiments. The aforementioned storage media include: various media such as ROM or random access memory RAM, magnetic disks, or optical discs that can store program codes.

[0431] As described above, the above are only the specific implementation manners of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for air gesture recognition, characterized in that: Applied to an electronic device including a front camera, the method includes: After recognizing a sliding gesture at a first moment, the front camera captures a first image at a second moment; the second moment is later than the first moment; If it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, then the hand-shaped icon corresponding to the starting gesture in the first image is not displayed.

2. The gesture recognition method according to claim 1, wherein It further includes: If it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

3. The air gesture recognition method according to claim 1 or 2, characterized in that It further includes: If it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is the same as the starting gesture of the sliding gesture, then the hand-shaped icon corresponding to the starting gesture in the first image is displayed, the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information record queue.

4. The air gesture recognition method according to claim 1 or 2, characterized in that It further includes: If it is detected at a third moment that the first image includes a starting gesture, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, and the gesture in the third image captured by the front camera within a second duration after the second moment is the same as the starting gesture in the first image, then the hand-shaped icon corresponding to the starting gesture in the first image is displayed, the dynamic gesture recognition process is executed, and the information of the first image and the information of the third image are stored in the frame information record queue; The second duration is less than the first duration.

5. The air gesture recognition method according to claim 1 or 2, characterized in that It further includes: If the interval between the third moment and the first moment is greater than the first duration, then the hand-shaped icon corresponding to the starting gesture in the first image is displayed, the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information record queue.

6. The air gesture recognition method according to claim 1 or 2, characterized in that It further includes: If other dynamic gestures except the sliding gesture are recognized at the first moment, then the dynamic gesture recognition process is executed.

7. The air gesture recognition method according to any one of claims 1-6, characterized in that The sliding gesture includes: an upward sliding gesture, or a downward sliding gesture, or a leftward sliding gesture, or a rightward sliding gesture.

8. The air gesture recognition method according to any one of claims 1-6, characterized in that The starting gesture includes: a palm with fingers facing up, or a palm with fingers facing left, or a palm with fingers facing right, or a back of the hand with fingers facing down, or a back of the hand with fingers facing left, or a back of the hand with fingers facing right.

9. The air gesture recognition method according to any one of claims 1-8, characterized in that Before recognizing a sliding gesture at the first moment, it further includes: Recognizing dynamic gestures according to the information of multiple frames of images in the frame information record queue; the frame information record queue is used to store the information of starting gesture images and stable state images, the starting gesture image is an image including a starting gesture, and the stable state image is a non-starting gesture image with a stable hand shape; After recognizing a swipe gesture at the first moment, it further includes: Deleting the information of all images related to the swipe gesture in the frame information recording queue.

10. The air gesture recognition method according to claim 9, wherein Before recognizing a dynamic gesture based on the information of multiple frames of images in the frame information recording queue, it further includes: Obtaining a continuous plurality of frames of images, which are captured by the front camera; Sequentially for each frame of the continuous plurality of frames of images, determining the hand feature information of the current frame image; Based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable; When the current frame image includes a starting gesture image, or the hand shape in the current frame image is stable, storing the information of the current frame image in the frame information recording queue; the information of the current frame image includes the hand feature information.

11. The air gesture recognition method according to claim 10, wherein Based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable, includes: When the frame information recording queue is empty, based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture; When the frame information recording queue is not empty, based on the hand feature information of the current frame image, determining whether the hand shape in the current frame image is stable.

12. The air gesture recognition method according to claim 11, wherein The hand feature information includes the hand category and the finger orientation; Based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture, includes: When the hand category corresponding to the current frame image is a palm and the finger orientation is upward, or leftward, or rightward, determining that the current frame image includes a starting gesture; or, When the hand category corresponding to the current frame image is a back of the hand and the finger orientation is downward, or leftward, or rightward, determining that the current frame image includes a starting gesture.

13. The air gesture recognition method according to claim 11, characterized in that, The hand feature information includes the hand category and the finger orientation; Based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture, includes: When the hand category corresponding to the current frame image is a palm and the finger orientation is not upward, not leftward, and not rightward, determining that the current frame image does not include a starting gesture; or, When the hand category corresponding to the current frame image is a back of the hand and the finger orientation is not downward, not leftward, and not rightward, determining that the current frame image does not include a starting gesture; or, When the hand category corresponding to the current frame image is neither a palm nor a back of the hand, determining that the current frame image does not include a starting gesture.

14. The air gesture recognition method according to claim 11, wherein The hand feature information includes the information of the hand detection box, the hand category, the coordinates of the hand key points, and the finger orientation; Based on the hand feature information of the current frame image, determining whether the hand shape in the current frame image is stable, includes: Calculate the intersection over union (IoU) between the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image; Calculate the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image; When the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, the IoU is greater than or equal to a first preset IoU, and the standard deviation of the first-order difference is less than or equal to a preset standard deviation, determine that the hand gesture in the current frame image is stable.

15. The air gesture recognition method according to claim 14, wherein Further includes: When the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the IoU is less than the first preset IoU, or the standard deviation of the first-order difference is greater than the preset standard deviation, determine that the hand gesture in the current frame image is unstable.

16. The air gesture recognition method according to claim 11, wherein When it is determined that the hand gesture in the current frame image is stable, further includes: Judge whether it is currently in a gesture hovering state according to the hand feature information of the current frame image and the hand feature information of the previous frame image; When not currently in a gesture hovering state, execute the dynamic gesture recognition process.

17. The air gesture recognition method according to claim 16, characterized in that, The hand feature information includes information of the hand detection box, hand category, coordinates of hand key points, and finger orientation; Judging whether it is currently in a gesture hovering state according to the hand feature information of the current frame image and the hand feature information of the previous frame image includes: Calculate the intersection over union (IoU) between the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image; According to the information of the hand detection box corresponding to the current frame image, calculate the first ratio of the hand detection box corresponding to the current frame image in the current frame image; According to the information of the hand detection box corresponding to the previous frame image, calculate the second ratio of the hand detection box corresponding to the previous frame image in the previous frame image; Calculate the difference between the first ratio and the second ratio; When the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the IoU is less than a second preset IoU, or the difference between the first ratio and the second ratio is greater than a preset difference, determine that it is not currently in a gesture hovering state.

18. The air gesture recognition method according to claim 17, wherein Further includes: When the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, the IoU is greater than or equal to the second preset IoU, and the difference between the first ratio and the second ratio is less than or equal to the preset difference, determine that it is currently in a gesture hovering state.

19. An electronic device, characterized in that, Includes: One or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 18.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when run on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 18.

21. A chip system, characterized in that The chip system is applied to an electronic device, the chip system includes one or more processors, and the one or more processors are used to call computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 18.

Images