Air gesture recognition method and electronic device

Abstract

The application is suitable for the technical field of image processing, and provides a method for recognizing a gesture in the air and an electronic device, which comprises the following steps: after recognizing a sliding gesture at a first time, a front camera collects a first image at a second time; the second time is later than the first time; if it is detected that the first image includes a starting gesture at a third time, the interval between the third time and the first time is less than or equal to a first length, and the starting gesture in the first image is different from the starting gesture of the sliding gesture, a hand icon corresponding to the starting gesture in the first image is not displayed; in the recovery process of a target sliding gesture corresponding to continuous sliding in the same direction, the user is prompted that the current dynamic gesture is not recognized, so that the situation of back-and-forth page turning can be avoided, and the operation experience of continuously sliding the screen in the same direction is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and electronic device for air gesture recognition. Background Technology

[0002] Currently, mobile phones and other electronic devices can use multi-frame images captured by the front-facing camera to recognize dynamic air gestures, and can respond to the recognized air gestures to perform corresponding operations, thus realizing air control of the screen.

[0003] Air gestures can include swipe gestures, which can be used to swipe the screen or turn pages. However, current swipe gesture recognition methods have low accuracy, and users may intend to swipe the screen only once, but end up swiping back and forth or turning pages back and forth, resulting in a poor user experience when swiping the screen continuously in the same direction. Summary of the Invention

[0004] This application provides a method and electronic device for air gesture recognition, which can prompt the user not to perform dynamic gesture recognition during the recovery process of the target swipe gesture corresponding to continuous swiping in the same direction, thereby avoiding the situation of flipping back and forth and improving the operating experience when swiping the screen continuously in the same direction.

[0005] In a first aspect, embodiments of this application provide a method for air gesture recognition, applied to an electronic device including a camera. The air gesture recognition method includes: after recognizing a swipe gesture at a first moment, the camera captures a first image at a second moment; the second moment is later than the first moment; if a starting gesture is detected in the first image at a third moment, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image is not displayed.

[0006] According to the air gesture recognition method provided in the embodiments of this application, if after recognizing a swipe gesture at the first moment, a first image is acquired by the front camera at a second moment later than the first moment, and the first image is detected to include a starting gesture at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, it indicates that the current process is in the recovery process of the target swipe gesture corresponding to the continuous same-direction swipe operation. In this case, by not displaying the hand icon corresponding to the starting gesture in the first image, the user can be prompted not to recognize dynamic gestures at present, thereby avoiding the recognition of gestures in the opposite direction of the swipe gesture, thus avoiding the situation of flipping back and forth, and improving the operation experience when continuously swiping the screen in the same direction.

[0007] In one optional implementation of the first aspect, the method further includes: if a starting gesture is detected in the first image at a third time, and the interval between the third time and the first time is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

[0008] According to the air gesture recognition method provided in the embodiments of this application, when it is determined that the current process is in the recovery process of the target swipe gesture corresponding to the continuous swiping operation, the dynamic gesture recognition process is not executed, which can avoid recognizing the gesture in the opposite direction of the swipe gesture, thereby avoiding the situation of flipping back and forth; in addition, by not storing the information of the first image, not only can storage space be saved, but also the use of the information of the first image for dynamic gesture recognition can be avoided, thereby reducing the probability of misrecognition of air gestures.

[0009] In one optional implementation of the first aspect, the method further includes: if a starting gesture is detected in the first image at a third time, and the interval between the third time and the first time is less than or equal to a first duration, and the starting gesture in the first image is the same as the starting gesture of the swipe gesture, then a hand icon corresponding to the starting gesture in the first image is displayed, a dynamic gesture recognition process is executed, and the information of the first image is stored in a frame information recording queue.

[0010] According to the air gesture recognition method provided in the embodiments of this application, if the first image is detected to include a starting gesture at a third time, and the interval between the third time and the first time is less than or equal to a first duration, and the starting gesture in the first image is the same as the starting gesture of the swipe gesture, it indicates that the recovery process of the target swipe gesture corresponding to the continuous unidirectional swipe operation has been completed. In this case, by displaying the hand icon corresponding to the starting gesture in the first image, the user can be prompted that dynamic gesture recognition can be performed normally, making it convenient for the user to continue inputting the target swipe gesture corresponding to the continuous unidirectional swipe operation. In addition, by storing the information of the first image in the frame information recording queue, the electronic device can accurately recognize the air gesture based on the information of multiple frames in the frame information recording queue, thereby improving the accuracy of air gesture recognition.

[0011] In one optional implementation of the first aspect, the method further includes: if a starting gesture is detected in the first image at a third time, and the interval between the third time and the first time is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of a swipe gesture, and the gesture in the third image captured by the camera within a second duration after the second time is the same as the starting gesture in the first image, then a hand icon corresponding to the starting gesture in the first image is displayed, a dynamic gesture recognition process is executed, and the information of the first image and the information of the third image are stored in a frame information recording queue; the second duration is less than the first duration.

[0012] According to the air gesture recognition method provided in this application embodiment, if the first image is detected to include a starting gesture at a third moment, and the interval between the third moment and the first moment is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of a swipe gesture, and the gesture in the third image captured by the camera within a second duration after the second moment is the same as the starting gesture in the first image, it indicates that the user is deliberately maintaining the starting gesture of the reverse swipe gesture, that is, the user wants to switch to the reverse swipe gesture. In this case, by displaying the hand icon corresponding to the starting gesture in the first image and executing the dynamic gesture recognition process, the user's operation of switching swipe gestures can be responded to quickly, thereby improving the user's operating experience. In addition, by storing the information of the first image and the information of the third image in the frame information recording queue, the electronic device can accurately identify the swipe gesture switched by the user based on the information of multiple frames in the frame information recording queue.

[0013] In one optional implementation of the first aspect, it further includes: if the interval between the third moment and the first moment is greater than the first duration, then displaying a hand icon corresponding to the starting gesture in the first image, executing a dynamic gesture recognition process, and storing the information of the first image in a frame information recording queue.

[0014] In one alternative implementation of the first aspect, the method further includes: if a dynamic gesture other than a swipe gesture is identified at the first moment, then a dynamic gesture recognition process is executed.

[0015] According to the air gesture recognition method provided in the embodiments of this application, when it is detected that the user does not perform a continuous swiping operation in the same direction, the air gesture recognition process can be carried out normally by displaying a hand icon corresponding to the starting gesture in the first image and executing a dynamic gesture recognition process.

[0016] In one alternative implementation of the first aspect, the swipe gesture includes: an up swipe gesture, or a down swipe gesture, or a left swipe gesture, or a right swipe gesture.

[0017] In one alternative implementation of the first aspect, the starting gesture includes: a palm with fingers pointing upwards, or a palm with fingers pointing to the left, or a palm with fingers pointing to the right, or a back of the hand with fingers pointing downwards, or a back of the hand with fingers pointing to the left, or a back of the hand with fingers pointing to the right.

[0018] In one optional implementation of the first aspect, before recognizing the swipe gesture at the first moment, the method further includes: recognizing the dynamic gesture based on information from multiple frames of images in the frame information recording queue; the frame information recording queue is used to store information about the initial gesture image and information about the stable state image, wherein the initial gesture image is an image including the initial gesture, and the stable state image is a non-initial gesture image with a stable hand shape; after recognizing the swipe gesture at the first moment, the method further includes: deleting information from all images related to the swipe gesture in the frame information recording queue.

[0019] According to the air gesture recognition method provided in the embodiments of this application, after each air gesture is recognized, the information of all images related to the swipe gesture in the frame information recording queue is deleted. This not only saves storage space, but also ensures the normal operation of the subsequent air gesture recognition process, thereby improving the efficiency and accuracy of air gesture recognition.

[0020] In one optional implementation of the first aspect, before recognizing the dynamic gesture based on the information of multiple frames of images in the frame information recording queue, the method further includes: acquiring multiple consecutive frames of images, which are captured by a camera; determining the hand feature information of the current frame image for each frame of the multiple consecutive frames in turn; determining, based on the hand feature information of the current frame image, whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable; if the current frame image includes the starting gesture image, or the hand shape in the current frame image is stable, storing the information of the current frame image in the frame information recording queue; the information of the current frame image includes the hand feature information.

[0021] According to the air gesture recognition method provided in the embodiments of this application, by setting a frame information recording queue and configuring the frame information recording queue to store only the information of the initial gesture image and the information of the stable state image, the recognition of dynamic gestures can be performed only based on the information of the multiple frames stored in the frame information recording queue, thereby improving the accuracy of air gesture recognition.

[0022] In one optional implementation of the first aspect, determining whether the current frame image contains the starting gesture of any dynamic gesture or whether the hand shape in the current frame image is stable, based on the hand feature information of the current frame image, includes: when the frame information recording queue is empty, determining whether the current frame image contains the starting gesture of any dynamic gesture based on the hand feature information of the current frame image; and when the frame information recording queue is not empty, determining whether the hand shape in the current frame image is stable based on the hand feature information of the current frame image.

[0023] According to the air gesture recognition method provided in the embodiments of this application, the efficiency of air gesture recognition can be improved by determining whether the current frame image includes the starting gesture of any dynamic gesture when the frame information recording queue is empty, and determining whether the hand shape in the current frame image is stable when the frame information recording queue is not empty.

[0024] In one optional implementation of the first aspect, the hand feature information includes hand category and finger orientation; based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture includes: if the hand category corresponding to the current frame image is palm and the finger orientation is upward, leftward, or rightward, determining that the current frame image includes the starting gesture; or, if the hand category corresponding to the current frame image is back of hand and the finger orientation is downward, leftward, or rightward, determining that the current frame image includes the starting gesture.

[0025] In one optional implementation of the first aspect, the hand feature information includes hand category and finger orientation; based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture includes: if the hand category corresponding to the current frame image is palm and the finger orientation is neither upward, nor left, nor right, then the current frame image does not include the starting gesture; or, if the hand category corresponding to the current frame image is back of hand and the finger orientation is neither downward, nor left, nor right, then the current frame image does not include the starting gesture; or, if the hand category corresponding to the current frame image is neither palm nor back of hand, then the current frame image does not include the starting gesture.

[0026] In one optional implementation of the first aspect, the hand feature information includes information about the hand detection box, the hand category, the coordinates of the hand key points, and the finger orientation. Based on the hand feature information of the current frame image, determining whether the hand shape in the current frame image is stable includes: calculating the intersection-union ratio (IU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image; calculating the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image; and determining that the hand shape in the current frame image is stable if the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the IU is greater than or equal to a first preset IU, and the standard deviation of the first-order difference is less than or equal to a preset standard deviation.

[0027] In one optional implementation of the first aspect, the method further includes: determining that the hand shape in the current frame image is unstable when the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the intersection-union ratio is less than a first preset intersection-union ratio, or the standard deviation of the first difference is greater than a preset standard deviation.

[0028] In one optional implementation of the first aspect, if the hand shape in the current frame image is determined to be stable, the method further includes: determining whether the hand is currently in a gesture hovering state based on the hand feature information of the current frame image and the hand feature information of the previous frame image; and if the hand is not currently in a gesture hovering state, executing a dynamic gesture recognition process.

[0029] According to the air gesture recognition method provided in the embodiments of this application, by determining whether the current state is a gesture hovering state, the dynamic gesture recognition process is executed only when the current state is not a gesture hovering state, which can save the power consumption of electronic devices.

[0030] In one optional implementation of the first aspect, the hand feature information includes information about the hand detection box, the hand category, the coordinates of the hand key points, and the finger orientation. Determining whether the current state is a gesture hovering state based on the hand feature information of the current frame image and the previous frame image includes: calculating the intersection-union ratio (IU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image; calculating a first proportion of the hand detection box corresponding to the current frame image in the current frame image based on the information of the hand detection box corresponding to the current frame image; calculating a second proportion of the hand detection box corresponding to the previous frame image in the previous frame image based on the information of the hand detection box corresponding to the previous frame image; calculating the difference between the first proportion and the second proportion; if the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the IU is less than a second preset IU, or the difference between the first proportion and the second proportion is greater than a preset difference, then it is determined that the current state is not a gesture hovering state.

[0031] In one optional implementation of the first aspect, it further includes: determining that the current state is a gesture hovering state when the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, the intersection-union ratio is greater than or equal to a second preset intersection-union ratio, and the difference between the first ratio and the second ratio is less than or equal to a preset difference.

[0032] Secondly, embodiments of this application provide an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more computer-executable programs, the one or more computer-executable programs including instructions, which, when executed by the one or more processors, cause the electronic device to perform each step of the air gesture recognition method as described in any implementation of the first aspect above.

[0033] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer-executable program, which, when invoked by an electronic device, causes the electronic device to perform the steps of the air gesture recognition method as described in any implementation of the first aspect above.

[0034] Fourthly, embodiments of this application provide a computer-executable program product that, when run on an electronic device, causes the electronic device to perform each step of the air gesture recognition method of any implementation of the first aspect described above.

[0035] Fifthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes a processor coupled to a memory, which stores computer program instructions. When the processor invokes the computer program instructions, the electronic device performs the steps of the air gesture recognition method as described in any of the implementations of the first aspect above. The chip system can be a single chip or a chip module composed of multiple chips.

[0036] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating a scenario where electronic devices can be controlled via air gestures.

[0038] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0040] Figure 4 A schematic diagram illustrating the hand shape change process corresponding to various air gestures provided in the embodiments of this application;

[0041] Figure 5 A schematic diagram illustrating the interaction timing between various modules in the mobile phone system architecture during the implementation of an air gesture recognition method provided in this application embodiment;

[0042] Figure 6 This is a schematic diagram of an image containing a human hand captured by an AO camera provided in an embodiment of this application;

[0043] Figure 7 A schematic diagram illustrating the setting scenario of the preset operation provided in the embodiments of this application;

[0044] Figure 8 This is a flowchart illustrating the specific implementation of step S52 in a gesture recognition method provided in this application embodiment.

[0045] Figure 9 This is a flowchart illustrating the specific implementation of step S53 in a gesture recognition method provided in this application embodiment.

[0046] Figure 10 This is a schematic diagram illustrating a scenario for determining finger orientation, provided in an embodiment of this application.

[0047] Figure 11 This is a flowchart illustrating the specific implementation of step S532 in a gesture recognition method provided in this application embodiment.

[0048] Figure 12 This is a flowchart illustrating the specific implementation of step S5322 in a gesture recognition method provided in this application embodiment.

[0049] Figure 13 This is a flowchart illustrating the specific implementation of step b3 in a gesture recognition method provided in an embodiment of this application.

[0050] Figure 14 This is a flowchart illustrating the specific implementation of the continuous same-direction swiping detection process in a gesture recognition method provided in this application embodiment;

[0051] Figure 15 A schematic diagram of a user interface related to an air gesture recognition method provided in an embodiment of this application;

[0052] Figure 16 This is a flowchart illustrating the specific implementation of step c3 in a gesture recognition method provided in an embodiment of this application.

[0053] Figure 17 This is a flowchart illustrating the specific implementation of step S535 in a gesture recognition method provided in this application embodiment.

[0054] Figure 18 This is a schematic flowchart illustrating a gesture recognition method for air gestures, provided as another embodiment of this application. Detailed Implementation

[0055] It should be noted that the terminology used in the implementation section of the embodiments of this application is only used to explain the specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related items, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more.

[0056] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0057] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0058] Currently, mobile phones and other electronic devices support contactless gesture interaction technology. This technology enables contactless human-computer interaction in scenarios where it is inconvenient for users to touch the electronic device, improving the ease of operation.

[0059] Specifically, electronic devices can support always-on (AO) camera functionality. When AO is enabled, the front-facing camera of the electronic device is always on and can capture images in real time. The electronic device can analyze multiple frames of images captured by the AO camera (i.e., the always-on front-facing camera) to recognize user-inputted air gestures and can respond to the recognized air gestures by performing corresponding operations, thus enabling air control of the electronic device.

[0060] For example, please refer to Figure 1 Assuming that by Figure 1 The palm with fingers pointing upwards, shown in (a) in the image, becomes... Figure 1 The clenched fist gesture shown in (b) corresponds to a screenshot operation. Therefore, the electronic device can perform a screenshot operation after recognizing the clenched fist gesture, obtaining a screenshot image, thus achieving air screenshotting. Assuming that... Figure 1 The palm with fingers pointing upwards, as shown in (c), becomes... Figure 1 As shown in (d), a downward-pointing hand gesture corresponds to a swipe-down screen operation or a page-down operation. Therefore, the electronic device can execute the swipe-down screen operation or page-down operation after recognizing the swipe-down gesture, thus achieving a non-invisible swipe-down screen operation or page-down operation. Assuming that... Figure 1 The back of the hand with fingers pointing downwards, as shown in (e), becomes... Figure 1 The upward swipe gesture of the palm with the fingers pointing upward, as shown in (f), corresponds to the swipe-up screen operation or the page-turning operation. The electronic device can then perform the swipe-up screen operation or the page-turning operation after recognizing the upward swipe gesture, thereby realizing the screen swipe-up or page-turning operation without air contact.

[0061] according to Figure 1The swipe gesture shown in (c) and (d) and Figure 1 As shown in (e) and (f), the swipe gestures in opposite directions involve two completely opposite hand shape changes. Therefore, when a user wants to continuously swipe the screen or flip pages in the same direction, such as continuously flipping down pages, the electronic device will also use the frames captured during the process of the user completing a swipe gesture, from the end gesture of that swipe gesture (i.e., the back of the hand with fingers pointing down) to the beginning gesture of the next swipe gesture (i.e., the palm with fingers pointing up) as the recognition of air gestures. This will cause the electronic device to recognize the swipe gesture after recognizing the swipe gesture, resulting in back-and-forth page flipping, which reduces the user's operating experience when continuously swiping the screen or flipping pages in the same direction.

[0062] In view of this, this application provides a method and electronic device for air gesture recognition. If, after recognizing a swipe gesture at the first moment, a first image is acquired by the front-facing camera at a second moment later than the first moment, and the first image is detected to include a starting gesture at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, it indicates that the current process is in the recovery process of the target swipe gesture corresponding to the continuous unidirectional swipe operation. In this case, by not displaying the hand icon corresponding to the starting gesture in the first image, the user can be prompted not to recognize dynamic gestures at present, thereby avoiding the recognition of gestures in the opposite direction of the swipe gesture, thus avoiding the situation of flipping back and forth, and improving the operating experience when continuously swiping the screen in the same direction.

[0063] The following provides a detailed description of the air gesture recognition method provided in the embodiments of this application.

[0064] The air gesture recognition method provided in this application embodiment can be applied to electronic devices including front-facing cameras. Electronic devices may include mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs), etc. This application embodiment does not limit the specific type of electronic device.

[0065] For example, please refer to Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0066] like Figure 2As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0067] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0068] For example, processor 110 can be used to execute the air gesture recognition method in the embodiments of this application.

[0069] A controller can be the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0070] The processor 110 may also include a memory for storing instructions and data. For example, the memory may include a first storage area and a second storage area. The first storage area may be used to store the frame information recording queue involved in the air gesture recognition process; the second storage area may be used to store the recognition time and type of each recognized air gesture.

[0071] Camera 193 can be used to capture still images or videos. An electronic device may include one or N cameras 193, where N is a positive integer greater than 1. At least one of the N cameras 193 is a front-facing camera.

[0072] Display screen 194 is used to display images, videos, hand icons involved in air gesture recognition, etc. Display screen 194 may include a display panel. Electronic devices can implement display functions through GPU, display screen 194, and application processor, etc.

[0073] It is understood that the above is an exemplary description of the structure of an electronic device. It should be understood that in other embodiments, the electronic device may include more or fewer components than illustrated, or may combine or separate certain components, or may have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0074] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software architecture of the electronic device.

[0075] Please see Figure 3 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application.

[0076] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the system runtime library layer, and the kernel layer.

[0077] The application layer can include a series of application packages, such as settings, camera, and smart sensing application packages. For ease of explanation, these application packages will be referred to as applications from now on.

[0078] Intelligent sensing applications can be used to support gesture control services. Gesture control services can include display services related to gesture control, such as displaying a hand icon corresponding to the initial gesture of the gesture; or, gesture control services can include response services for responding to gesture control, such as performing operations corresponding to the gesture control.

[0079] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0080] For example, the application framework layer may include a window manager, a message manager, a sensor service, etc.

[0081] A window manager can be used to manage windowed applications. For example, a window manager can be used to get the screen size, determine if there is a status bar, lock the screen, and capture the screen.

[0082] A message manager can be used to pass messages between different components. Specifically, a message manager can decouple different components through a message publish and subscribe mechanism, allowing one component to communicate directly with another component without calling the other, by publishing messages.

[0083] For example, smart sensing applications can subscribe to messages related to air gesture services through a message manager.

[0084] Sensor services can be used to manage and provide sensor data, such as data from accelerometers, magnetometers, and gyroscopes in the hardware layer, and to provide data from each sensor to other modules.

[0085] The system runtime library layer may include air gesture algorithm libraries, etc.

[0086] The air gesture algorithm library can be used to recognize air gestures based on multiple consecutive frames of images captured by an AO camera at the hardware layer. For example, the air gesture algorithm library may include a hand feature acquisition module and an air gesture recognition module.

[0087] The hand feature acquisition module may include a hand target detection unit, a hand classification unit, and a hand key point detection unit. The air gesture recognition module may include an input unit, a hand stability detection unit, a continuous unidirectional swipe detection unit, a gesture hovering state detection unit, a dynamic gesture recognition unit, and a timeout detection unit. It should be noted that the specific functions of the hand feature acquisition module and the air gesture recognition module will be described in subsequent embodiments and will not be detailed here.

[0088] The kernel layer is the layer between hardware and software. The kernel layer can contain camera drivers, sensor drivers, etc.

[0089] It should be noted that, Figure 3 Only the modules relevant to the embodiments of this application are shown. In other embodiments, each layer may include any other possible modules, and each module may include one or more sub-modules. This application does not limit these modules.

[0090] The following description, in conjunction with the accompanying drawings, illustrates the interaction timing between various modules in the mobile phone system architecture during the implementation of the air gesture recognition method provided in this application embodiment.

[0091] To facilitate understanding, the air gestures involved in the embodiments of this application will be introduced first.

[0092] In this embodiment, the air gesture can be a dynamic gesture. For example, air gestures can include grasping gestures, flipping gestures, swiping gestures, and pressing gestures. Swiping gestures can include upward swipes, downward swipes, leftward swipes, and rightward swipes. It is understood that each air gesture can correspond to a dynamic hand shape change process. Different air gestures may correspond to different hand shape change processes.

[0093] Please see Figure 4 This is a schematic diagram illustrating the hand shape change process corresponding to various air gestures provided in the embodiments of this application.

[0094] like Figure 4 As shown, the hand shape change process corresponding to the grasping gesture can include: from a palm with fingertips pointing upwards to a fist.

[0095] The hand shape change process corresponding to the flip gesture can include: changing from a palm with fingertips pointing upwards to the back of the hand with fingertips pointing upwards.

[0096] The hand shape change process corresponding to the swipe gesture can include: from the back of the hand with fingertips pointing down to the palm with fingertips pointing up.

[0097] The hand shape change process corresponding to the downward swiping gesture can include: from the palm with fingertips pointing upwards to the back of the hand with fingertips pointing downwards.

[0098] The hand shape change process corresponding to the left swipe gesture can include: from a palm with fingertips pointing to the right to a back of the hand with fingertips pointing to the left; or from a back of the hand with fingertips pointing to the right to a palm with fingertips pointing to the left.

[0099] The hand shape change process corresponding to the right swipe gesture can include: from a palm with fingertips pointing to the left to a back of the hand with fingertips pointing to the right; or from a back of the hand with fingertips pointing to the left to a palm with fingertips pointing to the right.

[0100] The hand shape change process corresponding to the pressing gesture can include: from a palm with fingertips pointing upwards and far from the screen to a palm with fingertips pointing upwards and closer to the screen.

[0101] In this embodiment of the application, to facilitate the determination of the type of air gesture, the multiple hand shapes involved in the air gesture can be divided into a starting gesture, an intermediate gesture, and a ending gesture. The starting gesture can refer to the initial action when the user performs the air gesture, the ending gesture can refer to the final action after the user completes the air gesture, and the intermediate gestures can include the various hand shapes experienced during the transition from the starting gesture to the ending gesture. It is understood that different types of air gestures will have at least one different starting and ending gesture; therefore, when performing air gesture recognition, the type of air gesture can be determined at least based on the starting and ending gestures.

[0102] Please continue reading. Figure 4 The starting hand gesture for grasping can be a palm with fingertips pointing upwards, and the ending hand gesture can be a fist. The intermediate hand gestures can include the various hand shapes experienced during the process of changing from a palm with fingertips pointing upwards to a fist.

[0103] The starting gesture for a flip gesture can be a palm with fingertips pointing upwards, and the ending gesture can be the back of the hand with fingertips pointing upwards. The intermediate gestures can include the various hand shapes experienced during the process of changing from a palm with fingertips pointing upwards to the back of the hand with fingertips pointing upwards.

[0104] The starting gesture for an upward swipe gesture can be the back of the hand with the fingertips pointing downwards, and the ending gesture can be the palm with the fingertips pointing upwards. The intermediate gestures can include the various hand shapes experienced during the process of changing from the back of the hand with the fingertips pointing downwards to the palm with the fingertips pointing upwards.

[0105] The starting gesture for the downward swipe gesture can be a palm with fingertips pointing upwards, and the ending gesture can be the back of the hand with fingertips pointing downwards. The intermediate gestures can include the various hand positions experienced during the process of changing from a palm with fingertips pointing upwards to the back of the hand with fingertips pointing downwards.

[0106] The starting gesture for a left swipe gesture can be a palm with fingertips pointing to the right, and the ending gesture can be the back of the hand with fingertips pointing to the left. The intermediate gestures can include the various hand positions experienced during the transition from a palm with fingertips pointing to the right to the back of the hand with fingertips pointing to the left. Alternatively, the starting gesture for a left swipe gesture can be the back of the hand with fingertips pointing to the right, and the ending gesture can be a palm with fingertips pointing to the left. The intermediate gestures can include the various hand positions experienced during the transition from the back of the hand with fingertips pointing to the right to the palm with fingertips pointing to the left.

[0107] The right swipe gesture can start with the back of the hand with fingertips pointing left and end with the palm with fingertips pointing right. The intermediate gestures can include all the hand positions involved in the transition from the back of the hand with fingertips pointing left to the palm with fingertips pointing right. Alternatively, the right swipe gesture can start with the palm with fingertips pointing left and end with the back of the hand with fingertips pointing right. The intermediate gestures can include all the hand positions involved in the transition from the palm with fingertips pointing left to the back of the hand with fingertips pointing right.

[0108] The starting gesture of the pressing gesture can be a palm with fingertips pointing upwards, and the ending gesture can also be a palm with fingertips pointing upwards. The starting gesture occupies less of the image than the ending gesture. The intermediate gestures of the pressing gesture can include the various hand shapes experienced during the process of changing from a palm with fingertips pointing upwards and occupying a smaller proportion of the image to a palm with fingertips pointing upwards and occupying a larger proportion of the image.

[0109] Please see Figure 5 This diagram illustrates the interaction timing between various modules in a mobile phone system architecture during the implementation of a gesture recognition method provided in this application. In some embodiments, when the mobile phone performs gesture recognition, the AO camera, hand feature acquisition module, gesture recognition module, message manager, and smart sensing application in the mobile phone system architecture can interact with each other. The specific interaction process may include S51 to S54, as detailed below:

[0110] S51, the AO camera acquires multiple consecutive frames of images and sends each frame to the hand feature acquisition module in sequence according to the acquisition time sequence. The acquisition time sequence indicates the order in which the AO camera acquires each frame.

[0111] In one optional implementation, air gesture recognition can be performed while the screen is on. Based on this, the AO camera can sequentially send the images captured while the screen is on to the hand feature acquisition module according to the acquisition sequence.

[0112] S52, the hand feature acquisition module sequentially determines the hand feature information of the current frame image for each frame in a series of consecutive images, and sends the hand feature information of the current frame image to the air gesture recognition module.

[0113] In this embodiment, the hand feature information of the current frame image determined by the hand feature acquisition module may include the information of the hand detection box corresponding to the current frame image, the hand category, and the coordinates of key points of the hand.

[0114] The information of the hand detection box can be used to represent the position and range of the hand in the corresponding image. For example, the hand detection box can be the smallest rectangle that can select the hand. Based on this, the information of the hand detection box can be represented by the coordinates of the center point of the hand detection box in a first preset coordinate system and the side length of the hand detection box; or it can be represented by the coordinates of the two diagonal vertices of the hand detection box in the first preset coordinate system, etc., and this application embodiment does not limit it in this way.

[0115] The first preset coordinate system can be a Cartesian coordinate system based on the image. For example, it can be a Cartesian coordinate system with the top left vertex of the image as the origin and the two sides intersecting at the top left vertex as the x-axis and y-axis, respectively.

[0116] For example, please refer to Figure 6 This is a schematic diagram of an image containing a human hand captured by an AO camera provided in an embodiment of this application. Figure 6 As shown, when determining the hand feature information of the current frame image 61, the hand feature acquisition module can establish a first preset coordinate system with the top-left vertex O of the image as the origin, and the straight line containing the first edge OL and the straight line containing the second edge OH as the x-axis and y-axis, respectively. Based on this, assuming that the two diagonal vertices of the hand detection box 610 of the current frame image determined by the hand feature acquisition module are vertex R and vertex S, the hand feature acquisition module can determine the coordinates (x-y-y) of vertex R. R ,y R ) and the coordinates (x) of vertex S S ,y S The information of the hand detection box corresponding to the current frame image is determined.

[0117] The "Hand" category can be used to represent the type of hand shape in a drawing.

[0118] For example, the human hand category can include the palm, the back of the hand, or the fist, etc.

[0119] The coordinates of key points on a person's hand can be used to represent the position of each key point on a person's hand in an image.

[0120] For example, please continue reading Figure 6 A hand can contain 21 key points, which are labeled 0 to 20. Based on this, the coordinates of the key points of the hand in the current frame image can be a coordinate array composed of the coordinates of the 21 key points of the hand in the first preset coordinate system.

[0121] In one optional implementation, the hand feature acquisition module can determine the hand feature information of the current frame image if a hand is present. If no hand is present in the current frame image, the hand feature acquisition module can discard the current frame image, thus improving the recognition efficiency of air gestures.

[0122] In this embodiment, the hand feature acquisition module determines the hand feature information of the images captured by the AO camera and sends the hand feature information of each frame of the image to the air gesture recognition module. This allows the air gesture recognition module to easily identify whether an air gesture exists and determine the type of air gesture based on the hand feature information of multiple frames of images.

[0123] S53, the air gesture recognition module determines whether the current frame image contains the starting gesture of any dynamic gesture or whether the hand shape in the current frame image is stable based on the hand feature information of the current frame image; if the current frame image contains the starting gesture image or the hand shape in the current frame image is stable, the information of the current frame image is stored in the frame information recording queue; the air gesture is recognized based on the information of multiple frames in the frame information recording queue.

[0124] The information in the current frame image may include the hand feature information of the current frame image.

[0125] Understandably, since the user's hand usually remains still briefly when initiating or completing a gesture, the starting and ending gestures of a gesture are typically those with a stable hand shape. A stable hand shape can refer to a clear hand, without ghosting, and without distortion.

[0126] In one specific implementation, for any frame image captured by the AO camera, when determining whether the current frame image includes the starting gesture of any dynamic gesture, the air gesture recognition module can determine whether the hand shape in the current frame image is the same as the hand shape corresponding to the starting gesture of any air gesture based on the hand feature information of the current frame image. Optionally, if the hand shape in the current frame image is the same as the hand shape corresponding to the starting gesture of any air gesture, the air gesture recognition module can determine that the current frame image includes the starting gesture, that is, determine that the current frame image is the starting gesture image; alternatively, if the hand shape in the current frame image is different from the hand shapes corresponding to the starting gestures of all air gestures, the air gesture recognition module can determine that the current frame image does not include the starting gesture, that is, determine that the current frame image is a non-starting gesture image.

[0127] In one specific implementation, for any non-initial gesture image frame, when determining whether the hand shape in the current frame is stable, the air gesture recognition module can determine whether the hand shape in the current frame is stable based on the hand feature information of the current frame and the hand feature information of the previous frame. The specific determination method will be described in detail in subsequent embodiments and will not be elaborated here.

[0128] The frame information recording queue can be used to store information about the initial gesture image or the stable state image recognized by the air gesture recognition module. The image information stored in the frame information recording queue may include the hand feature information of the image. In this embodiment, the stable state image can refer to a non-initial gesture image where the hand shape is in a stable state.

[0129] In this embodiment, different air gestures can be configured with different recognition strategies. The recognition strategy can be used to represent the characteristics of hand shape changes in air gestures.

[0130] Based on this, in a specific implementation, when the air gesture recognition module identifies air gestures based on information from multiple frames in the frame information recording queue and determines the type of the air gesture, it can determine whether there are consecutive multiple frames in the frame information recording queue that match the hand shape change characteristics of any air gesture, based on the recognition strategy for each air gesture. Optionally, if there are consecutive multiple frames in the frame information recording queue that match the hand shape change characteristics of the target air gesture, the air gesture recognition module can determine that an air gesture currently exists and can determine the type of the target air gesture as the type of the currently recognized air gesture. For example, assuming there are consecutive multiple frames in the frame information recording queue that match the hand shape change characteristics of an up swipe gesture, the air gesture recognition module can determine that an up swipe gesture currently exists. Optionally, if there are no consecutive multiple frames in the frame information recording queue that match the hand shape change characteristics of any air gesture, the air gesture recognition module can determine that no air gesture currently exists.

[0131] In this embodiment, each time the air gesture recognition module recognizes an air gesture, it can delete all image information related to that air gesture from the frame information recording queue. This not only saves the phone's storage space, but also makes it easier for the air gesture recognition module to recognize the initial gesture image or the stable state.

[0132] In addition, each time the air gesture recognition module recognizes an air gesture, it can send the type of the air gesture to the message manager, so that the smart sensing application can obtain the type of the air gesture from the message manager and execute the preset operation corresponding to the air gesture.

[0133] S54, the smart sensing application obtains the type of air gesture from the message manager and executes the preset operation corresponding to the type of air gesture.

[0134] For example, each air gesture can correspond to a preset operation. The preset operation for each air gesture can be a system default or can be customized by the user.

[0135] For example, a hand gesture can correspond to an air screenshot operation. Based on this, when the air gesture type is a hand gesture, the smart sensing application can perform a screenshot operation.

[0136] A flip gesture corresponds to an air-to-pages operation. Based on this, when the air gesture is a grasping gesture, the smart sensing application can perform the page-turning operation.

[0137] A swipe-up gesture can correspond to swiping up on the screen or turning up a page. Therefore, when the air gesture type is a swipe-up gesture, the smart sensing application can perform a swipe-up screen operation or a page-turning operation.

[0138] A swipe down gesture can correspond to swiping down on the screen or turning down a page. Therefore, when the air gesture is a swipe down gesture, the smart sensing application can execute either a swipe down on the screen or a turn down a page.

[0139] A left swipe gesture can correspond to swiping the screen left or turning a page to the left. Therefore, when the air gesture is a left swipe, the smart sensing application can perform either a swipe-the-screen operation or a page-turning operation.

[0140] A right swipe gesture can correspond to swiping the screen right or turning a page to the right. Therefore, when the air gesture type is a right swipe, the smart sensing application can perform a right swipe screen operation or a right page turn.

[0141] For example, please refer to Figure 7 This is a schematic diagram of the preset operation setting scenario provided in the embodiments of this application.

[0142] When a user wants to enable or disable preset actions, or wants to configure the corresponding air gestures for preset actions, the user can... Figure 7 In (a) of the image, the Settings app icon on the phone's home screen is used to input a first action, such as a tap. The phone can respond to this first action by displaying... Figure 7 The settings interface shown in (b) is shown in the image.

[0143] like Figure 7As shown in (b), the settings interface may include accessibility settings. Users can input a second action for these accessibility settings, such as a click. The phone can respond to this second action by displaying... Figure 7 The accessibility settings interface shown in (c) is shown in the image.

[0144] like Figure 7 As shown in (c), the accessibility settings interface may include Smart Sensing settings. Users can input a third action for these Smart Sensing settings, such as a click. The phone can respond to this third action by displaying... Figure 7 The smart sensing settings interface is shown in (d) above.

[0145] like Figure 7 As shown in (d), the smart sensing settings interface can include multiple preset operation settings, such as screen swipe settings, air screenshot settings, and air page turning settings. Users can enter the settings interface for the corresponding preset operation by clicking on any preset operation setting. Users can enable or disable the corresponding preset operation in the preset operation settings interface, or configure the air gestures corresponding to the preset operation.

[0146] For example, a user can click Figure 7 Enter the air screenshot settings item shown in (d) in the middle. Figure 7 The settings interface for the air screenshot operation is shown in (e). You can enable or disable the air screenshot operation or set the corresponding air gestures in the settings interface.

[0147] As can be seen from the above, this embodiment acquires multiple consecutive frames of images, sequentially determines the hand feature information of each frame, identifies the starting gesture image or stable state image of the air gesture based on the hand feature information of each frame, and stores the hand feature information of the starting gesture image or stable state image in the frame information recording queue. The air gesture is identified only based on the hand feature information of the starting gesture image and the stable state image stored in the frame information recording queue, and the type of air gesture is determined. Since the hand shape in the starting gesture image and the hand shape in the stable state image are both in a stable state, the influence of images with unstable hand shapes on the accuracy of air gesture recognition can be avoided, and the probability of misidentification of air gesture recognition can be reduced.

[0148] In a specific implementation, the hand feature acquisition module may include a hand target detection unit, a hand classification unit, and a hand key point detection unit. Based on this, S52 in the above embodiment may specifically include, for example... Figure 8 S521 to S523 are described in detail below:

[0149] S521, according to the acquisition time sequence, for each frame of the image, the hand target detection unit performs hand target detection on the current frame image, obtains the information of the hand detection box corresponding to the current frame image, and sends the information of the hand detection box corresponding to the current frame image to the hand classification unit, the hand key point detection unit and the air gesture recognition module.

[0150] It is understandable that, since the presence of a human hand in each frame of the image captured by the AO camera is uncertain, during the air gesture recognition process, after receiving each frame of the image from the AO camera, the hand feature acquisition module can first use the hand target detection unit to preliminarily identify images containing a human hand and determine the information of the hand detection box corresponding to the image containing the human hand. It should be noted that the specific content of the hand detection box information can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0151] In this embodiment, the purpose of the hand target detection unit outputting the hand detection box information is to enable the hand classification unit or the hand key point detection unit to locate the hand in the corresponding image based on the hand detection box information, so as to facilitate the hand classification unit, the hand key point detection unit or the air gesture recognition module to perform subsequent operations on the image.

[0152] Optionally, the hand target detection unit can perform hand target detection on the current frame image based on the object detection (OD) algorithm to obtain the information of the hand detection box corresponding to the current frame image.

[0153] S522, according to the acquisition sequence, for each frame of image, the hand classification unit classifies the hand in the current frame image based on the information of the hand detection box corresponding to the current frame image, obtains the hand category corresponding to the current frame image, and sends the hand category corresponding to the current frame image to the air gesture recognition module.

[0154] For example, the hand classification unit can be configured with a pre-trained hand classification model. This model can be used to classify hands to determine their category. Therefore, for each frame of an image, the hand classification unit can use the pre-trained model to determine the hand category corresponding to that frame.

[0155] In one optional implementation, in order to reduce the computational load of air gesture recognition and improve its efficiency, for each frame of image, the hand classification unit can crop out a partial image of the hand that only includes the hand from the current frame image based on the information of the hand detection box corresponding to the current frame image, and input the partial image of the hand corresponding to the current frame image into the pre-trained hand classification model to obtain the hand category corresponding to the current frame image.

[0156] For example, the hand classification model can be a neural network module trained based on a deep learning algorithm. This application does not limit the specific type of hand classification model or the training method.

[0157] S523, according to the acquisition sequence, for each frame of the image, the hand key point detection unit performs hand key point detection on the current frame image based on the information of the hand detection box corresponding to the current frame image, obtains the coordinates of the hand key points corresponding to the current frame image, and sends the coordinates of the hand key points corresponding to the current frame image to the air gesture recognition module.

[0158] For example, the hand keypoint detection unit can be configured with a pre-trained hand keypoint detection model. This model can detect hand keypoints in an image and output their coordinates. Therefore, for each frame, the hand keypoint detection unit can obtain the coordinates of the hand keypoints corresponding to that frame using the pre-trained model.

[0159] In one optional implementation, in order to reduce the computational load of air gesture recognition and improve its efficiency, for each frame of image, the hand key point detection unit can crop out a partial image of the hand that only includes the hand from the current frame image based on the information of the hand detection box corresponding to the current frame image, and input the partial image of the hand corresponding to the current frame image into the hand key point detection model to obtain the coordinates of the hand key points corresponding to the current frame image.

[0160] In another specific implementation, the air gesture recognition module may include an input unit, a hand stability detection unit, a continuous unidirectional swipe detection unit, a gesture hovering state detection unit, a dynamic gesture recognition unit, and a timeout detection unit.

[0161] Based on this, S53 in the above embodiments may specifically include, for example: Figure 9 S531 to S536 are described in detail below:

[0162] S531, according to the acquisition sequence, for each frame of image, the input unit determines the finger orientation of the current frame image based on the coordinates of the key points of the hand corresponding to the current frame image, and sends the information of the hand detection box corresponding to the current frame image, the hand category, the coordinates of the key points of the hand, and the gesture orientation to the hand stability detection unit.

[0163] In one specific implementation, the input unit can determine the finger orientation corresponding to the current frame image using the following steps a1 to a3:

[0164] Step a1: The input unit determines the first coordinate of the midpoint of the line connecting the first target key point and the second target key point based on the coordinates of the first target key point and the second target key point in the coordinates of the hand key points corresponding to the current frame image.

[0165] The first target key point can be the hand key point corresponding to the tip of the middle finger, and the second target key point can be the hand key point corresponding to the tip of the ring finger. For an example, please refer to [link to example]. Figure 10 The first target key point can be the manpower key point corresponding to number 12, and the second target key point can be the manpower key point corresponding to number 16.

[0166] For example, the input unit can use the average of the x-coordinates of the first target keypoint and the x-coordinates of the second target keypoint as the first x-coordinate of the midpoint of the connecting line, and the average of the y-coordinates of the first target keypoint and the y-coordinates of the second target keypoint as the first y-coordinate of the midpoint of the connecting line. For example, suppose the coordinates of the first target keypoint are (x... 12 ,y 12 The coordinates of the second target key point are (y 12 ,y 16 Then the input unit can (x) 12 +x 16 ) / 2 is used as the first x-coordinate of the midpoint of the line connecting the two points. 12 +y 16 The first ordinate of the midpoint of the line connecting (x, y) / 2, i.e., the first coordinate of the midpoint of the line, can be [(x, y) / 2. 12 +x 16 ) / 2,(y 12 +y 16 ) / 2].

[0167] It is understandable that, since the coordinates of the key points of the human hand corresponding to each frame of the image refer to the coordinates of the corresponding key points of the human hand in the first preset coordinate system, the first coordinate of the midpoint of the above-mentioned connection also refers to the coordinate of the center point of the connection in the first preset coordinate system.

[0168] Step a2: The input unit performs coordinate transformation on the first coordinate of the midpoint of the line to obtain the second coordinate of the midpoint of the line in the second preset coordinate system, and calculates the angle between the direction vector corresponding to the midpoint of the line and the positive x-axis direction of the second preset coordinate system based on the second coordinate of the center point of the line.

[0169] The second preset coordinate system can be a Cartesian coordinate system with the third target key point among the key points of the human hand as the origin, and two rays intersecting the third target key point and parallel to two mutually perpendicular sides of the image as the x-axis and y-axis, respectively. The third target key point can be the key point of the human hand corresponding to the wrist. For example, the second preset coordinate system can be as follows: Figure 10In the coordinate system shown in (a), the third target key point can be the key point corresponding to label 0.

[0170] Understandably, since finger orientation can usually be determined by the position of the center point of the line relative to the wrist, the input unit can easily determine finger orientation by converting the first coordinate of the center point of the line into a second coordinate.

[0171] Since the second preset coordinate system is equivalent to shifting the origin of the first preset coordinate system from the position of the top left vertex of the image to the position of the third target key point, in a specific implementation, the input unit can determine the second coordinate of the connecting line center point based on the coordinates of the third target key point (i.e., the coordinates in the first preset coordinate system) and the first coordinate of the connecting line center point (i.e., the coordinates in the first preset coordinate system).

[0172] Specifically, the input unit can determine the second coordinate of the connection center point by the difference between the first coordinate of the connection center point and the coordinate of the third target key point. That is, the input unit can determine the second abscissa of the connection center point by the difference between the first x-coordinate of the connection center point and the x-coordinate of the third target key point, and determine the second ordinate of the connection center point by the difference between the first ordinate of the connection center point and the ordinate of the third target key point. For example, assuming the first coordinate of the connection center point is (5, -1) and the coordinate of the third target key point is (3, -6), the input unit can determine 5 - 3 = 2 as the second abscissa of the connection center point and -1 - (-6) = 5 as the second ordinate of the connection center point, that is, the second coordinate of the connection center point is (2, 5).

[0173] After the input unit determines the second coordinates of the center point of the connecting line, the vector pointing from the origin of the second preset coordinate system to the midpoint of the connecting line can be used as the direction vector corresponding to the midpoint of the connecting line. The direction pointed to by the direction vector can be used to represent the direction of the finger.

[0174] In a specific embodiment, assume the second coordinate of the center point of the connecting line is (x z1 ,y z1 If the input unit can calculate the angle between the direction vector and the positive x-axis of the second preset coordinate system using the following angle calculation formula:

[0175] θ=arctan2(y z1 ,x z1 )×(180 / π);

[0176] Where θ can be the angle between the direction vector and the positive x-axis, and arctan2 can be the arctangent function.

[0177] Step a3: The input unit determines the finger orientation based on the angle between the direction vector corresponding to the midpoint of the connecting line and the positive x-axis direction of the second preset coordinate system.

[0178] In one specific implementation, the phone can pre-store multiple preset angle ranges and their corresponding finger orientations. Combined with... Figure 10 In (b), the correspondence between the preset angle range and the finger orientation can be shown in Table 1.

[0179] Table 1

[0180] Preset angle range (unit: degrees) finger towards [-135,-45) Up [-135,-180]∪[135,180] To the left [45,135) Down [-45,45) To the right

[0181] Based on this, the input unit can first determine the target angle range where the angle between the direction vector corresponding to the midpoint of the connecting line and the positive x-axis is located from multiple preset angle ranges, and then determine the target finger orientation corresponding to the target angle range according to the correspondence between the preset angle range and the finger orientation, and determine the target finger orientation as the finger orientation corresponding to the corresponding image.

[0182] For example, if the angle between the direction vector corresponding to the midpoint of the line and the positive x-axis is -85 degrees, then according to the correspondence between the preset angle range and the finger orientation shown in Table 1, the target angle range of -85 degrees can be determined to be [-135, -45). Based on this, the input unit can determine the target finger orientation (upward) corresponding to the target angle range [-135, -45) as the finger orientation corresponding to the corresponding image.

[0183] S532, the hand stability detection unit determines whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable; if the current frame image includes the starting gesture, the hand feature information and detection time of the current frame image are stored in the frame information recording queue, and the continuous same-direction sliding detection process is entered; if the hand shape in the current frame image is stable, the hand feature information and detection time of the current frame image are stored in the frame information recording queue, and the gesture hovering state detection process is entered; if the current frame image does not include the starting gesture image, and the hand shape in the current frame image is unstable, the hand feature information of the current frame image is discarded.

[0184] Discarding the hand feature information of the current frame image can mean not storing the hand feature information of the current frame image.

[0185] The hand feature information of the current frame image stored in the frame information recording queue by the hand stability detection unit may include: information of the hand detection box corresponding to the current frame image, hand category, coordinates of key points of the hand, and finger orientation.

[0186] Optionally, if the current frame image includes the initial gesture, the detection time of the current frame image can be the time when the hand stabilization detection unit detects that the current frame image includes the initial gesture.

[0187] Optionally, if the hand shape in the current frame image is stable, the detection time of the current frame image can be used by the hand stability detection unit to determine the time when the hand shape in the current frame image is stable.

[0188] The continuous same-direction sliding detection process can be the process executed by the continuous same-direction sliding detection unit in S533.

[0189] The gesture hover state detection process can be the process executed by the gesture hover state detection unit in S534.

[0190] In a specific implementation, S532 may specifically include, for example: Figure 11 The details of S5321 to S5325 shown are as follows:

[0191] S5321, the hand stability detection unit determines whether the frame information recording queue is empty.

[0192] Optionally, if the frame information recording queue is empty, the hand stability detection unit can execute S5322.

[0193] Optionally, if the frame information recording queue is not empty, the hand stability detection unit can execute S5323.

[0194] S5322, the hand stability detection unit determines whether the current frame image contains the starting gesture of any air gesture based on the hand feature information of the current frame image.

[0195] It is understandable that, since the air gesture recognition module deletes all hand feature information and detection information of all images related to the air gesture from the frame information recording queue after recognizing each air gesture, when the frame information recording queue is empty, it means that the current frame image may be the starting gesture of the next air gesture. Based on this, when the frame information recording queue is empty, the hand stability detection unit can determine whether the current frame image includes the starting gesture of any air gesture based on the hand feature information of the current frame image.

[0196] Combination Figure 6 It can be seen that the starting gestures for air gestures can include the following types: palm with fingertips pointing upwards, back of hand with fingertips pointing downwards, palm with fingertips pointing to the left, back of hand with fingertips pointing to the left, palm with fingertips pointing to the right, and back of hand with fingertips pointing to the right.

[0197] Among them, a palm with fingertips pointing upwards can be the starting gesture for pressing, grasping, flipping, and sliding gestures; a palm with fingertips pointing to the right can be the starting gesture for a left swipe gesture; a palm with fingertips pointing to the left can be the starting gesture for a right swipe gesture; the back of the hand with fingertips pointing downwards can be the starting gesture for an upward swipe gesture; the back of the hand with fingertips pointing to the right can be the starting gesture for a left swipe gesture; and the back of the hand with fingertips pointing to the left can be the starting gesture for a right swipe gesture.

[0198] As can be seen, the initial gesture is related to the hand type and finger orientation. For example, the relationship between the initial gesture and the hand type and finger orientation can be shown in Table 2.

[0199] Table 2

[0200]

[0201] Based on this, in a specific implementation, the hand stability detection unit can determine whether the current frame image includes the starting gesture of any air gesture based on the hand type and finger orientation corresponding to the current frame image.

[0202] Optionally, the hand stabilization detection unit can determine the starting gesture of any air gesture in the current frame image under any of the following conditions:

[0203] (1) The hand category corresponding to the current frame image is palm, and the fingers are facing up, left or right.

[0204] (2) The hand category corresponding to the current frame image is the back of the hand, and the fingers are facing down, left or right.

[0205] Optionally, the hand stability detection unit can determine that the starting gesture, which does not include any air gesture in the previous frame image, is a non-starting gesture image under any of the following conditions:

[0206] (1) The hand category corresponding to the current frame image is neither palm nor back of hand.

[0207] (2) The hand category corresponding to the current frame image is palm, and the fingers are not facing up, left, or right.

[0208] (3) The hand category corresponding to the current frame image is the back of the hand, and the fingers are not facing down, left, or right.

[0209] In a specific implementation, S5322 may include, for example: Figure 12 S5322.1 to S5322.6 are described in detail below:

[0210] S5322.1 Determine whether the hand category corresponding to the current frame image is palm.

[0211] Optionally, if the hand category corresponding to the current frame image is palm, S5322.2 can be executed.

[0212] Optionally, if the hand category corresponding to the current frame image is not palm, S5322.3 can be executed.

[0213] S5322.2, determine whether the finger in the current frame image is facing upward, left, or right.

[0214] Optionally, if the finger in the current frame image is pointing upwards, to the left, or to the right, S5322.5 can be executed.

[0215] Optionally, if the finger orientation corresponding to the current frame image is neither upward, nor left, nor right, S5322.6 can be executed.

[0216] S5322.3, determine whether the hand category corresponding to the current frame image is the back of the hand.

[0217] Optionally, if the hand category corresponding to the current frame image is the back of the hand, S5322.4 can be executed.

[0218] Optionally, if the hand category corresponding to the current frame image is not the back of the hand, S5322.6 can be executed.

[0219] S5322.4, determine whether the finger in the current frame image is pointing downwards, to the left, or to the right.

[0220] Optionally, if the finger in the current frame image is pointing downwards, to the left, or to the right, S5322.5 can be executed.

[0221] Optionally, if the finger orientation corresponding to the current frame image is neither downward, nor left, nor right, S5322.6 can be executed.

[0222] S5322.5, determine that the current frame image includes the starting gesture.

[0223] S5322.6, Determine that the current frame image does not include the starting gesture.

[0224] It should be noted that, Figure 12Only one optional starting gesture image determination process is shown. In other embodiments, when the hand stability detection unit recognizes the starting gesture image, it can first determine whether the hand category corresponding to the current frame image is the back of the hand, and then determine whether the hand category corresponding to the current frame image is the palm; or, it can first determine the finger orientation corresponding to the current frame image, and then determine the hand category corresponding to the current frame image. All of these are within the protection scope of this application.

[0225] Optionally, if the current frame image is the starting gesture image of any air gesture, the hand stability detection unit can execute S5324 and enter the continuous same-direction sliding detection process in S533.

[0226] Optionally, if the current frame image is not the starting gesture image of any air gesture, the hand stability detection unit may execute S5325.

[0227] S5323, the hand stability detection unit determines whether the hand shape in the current frame image is stable based on the hand feature information of the current frame image.

[0228] In a specific implementation, the hand stability detection unit can use the following steps b1 to b3 to determine whether the hand shape in the current frame image is stable, as detailed below:

[0229] Step b1: Based on the information of the hand detection box corresponding to the current frame image and the information of the hand detection box corresponding to the previous frame image, calculate the intersection-union ratio (IUGR) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image.

[0230] Specifically, the hand stability detection unit can calculate the intersection area and union area of ​​the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image based on the information of the hand detection box corresponding to the current frame image and the information of the hand detection box corresponding to the previous frame image, and determine the ratio of the intersection area to the union area as the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image.

[0231] It should be noted that since the calculation method for the intersection and union areas of two rectangles with known location information is existing technology, the specific calculation method for the intersection and union areas of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image will not be described in detail here.

[0232] Step b2: Based on the coordinates of the hand key points in the current frame image and the coordinates of the hand key points in the previous frame image, calculate the standard deviation of the first-order difference between the hand key points in the current frame image and the hand key points in the previous frame image.

[0233] It should be noted that since the coordinates of the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image are both coordinate arrays composed of the coordinates of 21 hand key points, and the calculation method of the standard deviation of the first difference between the two arrays is an existing technology, the specific calculation method of the standard deviation of the first difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image will not be described in detail here.

[0234] Step b3: Based on the hand category corresponding to the current frame image, the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image, the finger orientation corresponding to the previous frame image, the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image, and the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image, determine whether the hand shape in the current frame image is stable.

[0235] Optionally, if the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection-union ratio (IU) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to a first preset IU, and the standard deviation of the first-order difference between the hand keypoints corresponding to the current frame image and the hand keypoints corresponding to the previous frame image is less than or equal to a preset standard deviation, the hand stability detection unit can determine that the hand shape in the current frame image is stable.

[0236] Optionally, the hand stability detection unit can determine that the hand shape in the current frame image is unstable, i.e., determine that the current frame image is an unstable image, under any of the following conditions:

[0237] (1) The hand category in the current frame image is different from the hand category in the previous frame image.

[0238] (2) The finger orientation in the current frame image is different from that in the previous frame image.

[0239] (3) The intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the first preset intersection-union ratio.

[0240] (4) The standard deviation of the first-order difference between the key points of the hand in the current frame image and the key points of the hand in the previous frame image is greater than the preset standard deviation.

[0241] The first preset intersection-union ratio and the preset standard deviation can both be set according to actual needs.

[0242] In a specific implementation, step b3 may include, for example: Figure 13S5323.1 to S5323.6 are described in detail below:

[0243] S5323.1 Determine whether the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image.

[0244] Optionally, if the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, S5323.2 can be executed.

[0245] Optionally, if the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, S5323.6 can be executed.

[0246] S5323.2, determine whether the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image.

[0247] Optionally, if the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, S5323.3 can be executed.

[0248] Optionally, if the finger orientation in the current frame image is different from the finger orientation in the previous frame image, S5323.6 can be executed.

[0249] S5323.3, determine whether the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the first preset intersection-union ratio.

[0250] Optionally, if the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the first preset intersection-union ratio, S5323.4 can be executed.

[0251] Optionally, if the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the first preset intersection-union ratio, S5323.6 can be executed.

[0252] S5323.4, determine whether the standard deviation of the first-order difference between the key points of the hand in the current frame image and the key points of the hand in the previous frame image is less than or equal to the preset standard deviation.

[0253] Optionally, if the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is less than or equal to the preset standard deviation, S5323.5 can be executed.

[0254] Optionally, if the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image is greater than the preset standard deviation, S5323.6 can be executed.

[0255] S5323.5, Determine that the hand shape in the current frame image is stable.

[0256] S5323.6, Determine that the hand shape in the current frame image is unstable.

[0257] Optionally, if the hand shape is stable in the current frame image, the hand stability detection unit can execute S5324 and enter the gesture hovering state detection process.

[0258] Optionally, if the hand shape in the current frame image is unstable, i.e., the current frame image is an unstable state image, the hand stability detection unit can execute S5325.

[0259] S5324, the hand stability detection unit stores the hand feature information corresponding to the current frame image and the detection time in the frame information recording queue.

[0260] Furthermore, when the current frame image is the starting gesture image, the hand stability detection unit can also associate and store the type of the starting gesture corresponding to the current frame image in the frame information recording queue.

[0261] S5325, the hand stability detection unit discards the hand feature information of the current frame image.

[0262] S533, the continuous same-direction swipe detection unit obtains the recognition time and type of the previous recognized air gesture. Based on the recognition time and type of the previous air gesture, the detection time of the current frame image, and the type of the starting gesture corresponding to the current frame image, it determines whether the current condition for short-term blocking of continuous same-direction swipes is met. If the condition for short-term blocking of continuous same-direction swipes is met, the dynamic gesture recognition process is not executed, and the corresponding hand icon is not displayed. If the condition for short-term blocking of continuous same-direction swipes is not met, the dynamic gesture recognition process is executed, and the hand feature information of the current frame image is sent to the message manager.

[0263] Optionally, the continuous same-direction sliding detection unit can determine that the short-time shielding condition for continuous same-direction sliding is not currently met in any of the following cases:

[0264] (1) The type of the previous air gesture is not a swipe gesture.

[0265] (2) The type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is greater than the first duration.

[0266] (3) The type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture.

[0267] (4) The type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, and the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration.

[0268] The first duration can be used to represent the recovery time of the initial gesture of the target swipe gesture in a scenario of continuous, same-direction swipes on the screen without air contact. For example, the first duration can be 2 seconds.

[0269] The second duration can be shorter than the first duration. For example, the second duration can be 800 milliseconds.

[0270] It is understandable that, in the case of the above situation (1), if the type of the previous air gesture is not a swipe gesture, it means that the user is definitely not performing a continuous swipe operation in the same direction, so the dynamic gesture recognition process needs to be executed normally.

[0271] In response to the above situation (2), in the scenario of continuous unidirectional swipe on the screen, the process of the user returning to the starting gesture of the target swipe gesture after performing a target swipe gesture is relatively quick and the duration is usually no longer than the first duration. Therefore, if the starting gesture image is detected only after the first duration after the previous swipe gesture is performed, it means that the user does not want to perform continuous unidirectional swipe operation, so the dynamic gesture recognition process needs to be executed normally.

[0272] Regarding the above situation (3), this situation indicates that after the user performs a target swipe gesture, he quickly returns to the starting gesture of the target swipe gesture. This means that the user is currently performing a continuous air swipe operation and has returned to the starting gesture of the target swipe gesture. Therefore, the dynamic gesture recognition process needs to be executed normally so that the dynamic gesture recognition unit can perform the next round of dynamic gesture recognition.

[0273] Regarding the above situation (4), this situation indicates that after the user completes a target swipe gesture, within the recovery time limit of the starting gesture of the target swipe gesture, the user deliberately maintains another starting gesture that is different from the starting gesture of the target swipe gesture for at least a second duration. This indicates that the user may want to switch to another different type of air gesture. For example, the user wants to switch from a swipe down gesture to an swipe up gesture. Therefore, in this situation, the dynamic gesture recognition process needs to be executed normally so that the dynamic gesture recognition unit can recognize the new air gesture switched by the user.

[0274] Optionally, if the type of the previous air gesture is a swipe gesture, and the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, and the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, and the duration of the stable state corresponding to the current frame image is less than the second duration, the continuous same-direction swipe detection unit can determine that the current continuous same-direction swipe short-term shielding condition is met.

[0275] Understandably, if the condition of short-term shielding during continuous unidirectional swipes is met, it indicates that the current state is in the process of restoring the initial gesture of the target swipe gesture. In order to prevent the dynamic gesture recognition unit from recognizing a swipe gesture that is opposite to the target swipe gesture, the corresponding hand icon may not be displayed and the dynamic gesture recognition process may not be executed. This allows the dynamic gesture recognition unit to avoid using the frames of images in the initial gesture restoration process of the target swipe gesture for dynamic gesture recognition. This can avoid the situation of flipping back and forth and improve the user's operating experience when continuously swiping the screen in the same direction without air contact.

[0276] In a specific implementation, the continuous same-direction sliding detection unit can be adopted as follows: Figure 14 The steps S5331 to S5336 are shown to determine whether the continuous unidirectional sliding short-time shielding condition is met, as detailed below:

[0277] S5331, determine whether the previous air gesture was a swipe gesture.

[0278] Optionally, if the previous air gesture was a swipe gesture, S5332 can be executed.

[0279] Optionally, if the previous air gesture was not a swipe gesture, S5335 can be executed.

[0280] S5332, determine whether the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration.

[0281] Optionally, if the interval between the detection time of the current frame image and the recognition time of the previous air gesture is less than or equal to the first duration, S5333 can be executed.

[0282] Optionally, if the interval between the detection time of the current frame image and the recognition time of the previous air gesture is greater than the first duration, S5335 can be executed.

[0283] S5333, determine whether the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture.

[0284] Optionally, if the type of the starting gesture corresponding to the current frame image is the same as the type of the starting gesture of the previous air gesture, S5335 can be executed.

[0285] Optionally, if the type of the starting gesture corresponding to the current frame image is different from the type of the starting gesture of the previous air gesture, S5334 can be executed.

[0286] S5334, determine whether the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration.

[0287] In one specific embodiment, the duration of the stable state corresponding to the current frame image can be determined by comparing the gesture in an image captured by the AO camera within a second time interval after the acquisition time of the current frame image with the initial gesture in the current frame image. If they are the same, it can be determined that the duration of the stable state corresponding to the current frame image is greater than or equal to the second time interval; if they are different, it can be determined that the duration of the stable state corresponding to the current frame image is less than the second time interval.

[0288] Optionally, if the duration of the stable state corresponding to the current frame image is greater than or equal to the second duration, S5335 can be executed.

[0289] Optionally, if the duration of the stable state corresponding to the current frame image is less than the second duration, S5336 can be executed.

[0290] S5335 displays the hand icon corresponding to the starting gesture in the current frame image and enters the dynamic gesture recognition process.

[0291] S5336: Do not display the hand icon corresponding to the starting gesture in the current frame image, and do not execute the dynamic gesture recognition process.

[0292] Understandably, when the short-term shielding condition for continuous unidirectional sliding is not met, the purpose of the continuous unidirectional sliding detection unit sending the hand feature information of the current frame image to the message manager is to enable the intelligent sensing application to obtain the hand feature information of the current frame image from the message manager, and then display a hand icon that matches the starting gesture corresponding to the current frame image. The hand icon can be used to indicate the type of starting gesture corresponding to the current frame image.

[0293] In one alternative implementation, since the type of the initial gesture can be determined by the hand category and finger orientation corresponding to the image, the hand feature information of the current frame image sent by the continuous same-direction sliding detection unit to the message manager can include the hand category and finger orientation corresponding to the current frame image.

[0294] For example, such as Figure 15 As shown, assuming the initial gesture corresponding to the current frame image is a palm with fingers pointing upwards, and the short-term masking condition for continuous same-direction sliding is not currently met, the hand feature information of the current frame image sent by the continuous same-direction sliding detection unit to the message manager may include: hand type: palm, finger orientation: upwards. At this time, the intelligent sensing application can display on the screen... Figure 15 The palm icon shown in number 151 has fingers pointing upwards.

[0295] S534, the gesture hovering state detection unit determines whether the current frame image is in a gesture hovering state based on the hand feature information of the current frame image; if it is in a gesture hovering state, it enters the timeout detection process; if it is not in a gesture hovering state, it enters the dynamic gesture recognition process.

[0296] The gesture hovering state refers to the state in which the hand shape remains unchanged for a third duration threshold. The third duration threshold can be set according to actual needs; for example, the third duration threshold can be 900 milliseconds.

[0297] The dynamic gesture recognition process can refer to the process executed by the dynamic gesture recognition unit in S535.

[0298] The timeout detection process can refer to the process executed by the timeout detection unit in S536.

[0299] In a specific implementation, the gesture hovering state detection unit can use the following steps c1 to c3 to determine whether the current state is a gesture hovering state, as detailed below:

[0300] Step c1: Based on the information of the hand detection box corresponding to the current frame image and the information of the hand detection box corresponding to the previous frame image, calculate the intersection-union ratio (IUGR) of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image.

[0301] It should be noted that the specific calculation method of the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image can be referred to the relevant description of step b1 in the aforementioned embodiment, and will not be repeated here.

[0302] Step c2: Based on the information of the hand detection box corresponding to the current frame image, calculate the first proportion of the hand detection box corresponding to the current frame image in the current image; based on the information of the hand detection box corresponding to the previous frame image, calculate the second proportion of the hand detection box corresponding to the previous frame image in the previous frame image; and calculate the difference between the first proportion and the second proportion.

[0303] Specifically, the gesture hovering state detection unit can calculate the area of ​​the hand detection box corresponding to the current frame image based on the information of the hand detection box corresponding to the current frame image, and determine the ratio of the area of ​​the hand detection box corresponding to the current frame image to the area of ​​the current frame image as the first proportion.

[0304] Similarly, the gesture hovering state detection unit can calculate the area of ​​the hand detection box corresponding to the previous frame image based on the information of the hand detection box corresponding to the previous frame image, and determine the ratio of the area of ​​the hand detection box corresponding to the previous frame image to the area of ​​the previous frame image as the second proportion.

[0305] Step c3: Based on the hand category corresponding to the current frame image, the hand category corresponding to the previous frame image, the finger orientation corresponding to the current frame image, the finger orientation corresponding to the previous frame image, the intersection-over-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image, and the difference between the first ratio and the second ratio, determine whether the current state is a gesture hovering state.

[0306] Optionally, if the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the second preset intersection-union ratio, and the difference between the first ratio and the second ratio is less than or equal to the preset difference, the gesture hovering state detection unit can determine that it is currently in a gesture hovering state.

[0307] It is understandable that when the user is in a gesture-hovering state, it means that the user is deliberately maintaining the initial gesture of a certain air gesture and does not want to immediately execute the subsequent gesture of that air gesture. Therefore, in this case, a dynamic gesture recognition process can be executed and a hand icon corresponding to the corresponding initial gesture can be displayed so that the air gesture can be quickly responded to after the user inputs the subsequent gesture of that air gesture.

[0308] Optionally, the gesture hover state detection unit can determine that it is not currently in a gesture hover state, i.e., it is currently in a non-gesture hover state, under any of the following conditions:

[0309] (1) The hand category in the current frame image is different from the hand category in the previous frame image.

[0310] (2) The finger orientation in the current frame image is different from that in the previous frame image.

[0311] (3) The intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the second preset intersection-union ratio.

[0312] (4) The difference between the first percentage and the second percentage is greater than the preset difference.

[0313] The second preset crossover-union ratio can be greater than the first preset crossover-union ratio.

[0314] The preset difference can be set according to actual needs, and there are no restrictions on it here.

[0315] In a specific implementation, step c3 may include, for example: Figure 16 The details of S5341 to S5346 shown are as follows:

[0316] S5341, determine whether the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image.

[0317] Optionally, if the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, S5342 can be executed.

[0318] Optionally, if the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, S5346 can be executed and the dynamic gesture recognition process in S535 can be entered.

[0319] S5342, determine whether the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image.

[0320] Optionally, if the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, S5343 can be executed.

[0321] Optionally, if the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, S5346 can be executed and the dynamic gesture recognition process in S535 can be entered.

[0322] S5343, determine whether the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the second preset intersection-union ratio.

[0323] Optionally, if the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is greater than or equal to the second preset intersection-union ratio, S5344 can be executed.

[0324] Optionally, if the intersection-union ratio of the hand detection box corresponding to the current frame image and the hand detection box corresponding to the previous frame image is less than the second preset intersection-union ratio, S5346 can be executed and the dynamic gesture recognition process in S535 can be entered.

[0325] S5344, determine whether the difference between the first percentage and the second percentage is less than or equal to a preset difference.

[0326] Optionally, if the difference between the first percentage and the second percentage is less than or equal to a preset difference, S5345 can be executed and the timeout detection process in S536 can be entered.

[0327] Optionally, if the difference between the first proportion and the second proportion is greater than a preset difference, S5346 can be executed and the dynamic gesture recognition process in S535 can be entered.

[0328] S5345, confirm that the current state is a gesture hover state.

[0329] S5346, confirm that the current state is not a gesture hover state.

[0330] S535, the dynamic gesture recognition unit determines whether there is an air gesture based on the hand feature information of multiple frames of images in the frame information recording queue; if there is an air gesture, it sends the type of air gesture to the message manager and deletes the hand feature information of all images related to the current air gesture from the frame information recording queue; if there is no air gesture, it enters the timeout detection process.

[0331] In one specific implementation, the dynamic gesture recognition unit can determine whether there are multiple consecutive frames in the frame information recording queue that match the hand shape change characteristics of any air gesture, based on the recognition strategy for each air gesture. The preset strategy for each air gesture can be configured according to the hand shape change characteristics of that air gesture.

[0332] Optionally, if the frame information recording queue contains multiple consecutive frames of images that match the hand shape change characteristics of the target air gesture, the dynamic gesture recognition unit can determine that an air gesture currently exists and identify the type of the target air gesture as the type of the current air gesture. For example, assuming the frame information recording queue contains multiple consecutive frames of images that match the hand shape change characteristics of an up swipe gesture, the air gesture recognition module can determine that an up swipe gesture currently exists.

[0333] Optionally, if there are no consecutive frames of images in the frame information recording queue that match the hand shape change characteristics of any air gesture, the air gesture recognition module can determine that there is currently no air gesture.

[0334] In this embodiment, after the dynamic gesture recognition unit determines the type of air gesture, it can save mobile phone storage space and facilitate the recognition of the next round of air gestures by deleting the hand feature information of all images related to the current air gesture from the frame information recording queue.

[0335] In addition, after the dynamic gesture recognition unit determines the type of air gesture, it can update the recognition time and type of the most recent air gesture in the second storage area to the recognition time and type of the current air gesture, which facilitates the detection of continuous unidirectional swipe gestures in the next round of air gesture recognition.

[0336] In one specific implementation, the dynamic gesture recognition unit can sequentially determine whether a corresponding air gesture exists based on the preset priority of each air gesture, thereby improving the efficiency of air gesture recognition.

[0337] The preset priority of each air gesture can be determined based on the frequency of its execution. For example, the more frequently an air gesture is used, the higher its preset priority can be. In practical applications, the execution frequency of each air gesture can be obtained by statistically analyzing the air gesture usage of each user.

[0338] Assuming the preset priorities of the various air gestures are ordered from highest to lowest as follows: grasping gesture, swiping gesture, flipping gesture, and pressing gesture, then S535 may include, for example: Figure 17 The details of S5351 to S5358 shown are as follows:

[0339] S5351, based on the recognition strategy of grasping hand gestures, determines whether there are consecutive multi-frame images in the frame information recording queue that match the characteristics of hand shape changes in grasping hand gestures.

[0340] Optionally, if there are multiple consecutive frames of images in the frame information recording queue that match the characteristics of hand shape changes in grasping gestures, S5355 can be executed.

[0341] Optionally, if there are no consecutive multi-frame images in the frame information recording queue that match the characteristics of hand shape changes in grasping gestures, S5352 can be executed.

[0342] S5352, based on the recognition strategy of swipe gesture, determines whether there are consecutive multi-frame images in the frame information recording queue that match the characteristics of hand shape changes in swipe gesture.

[0343] Optionally, if there are multiple consecutive frames of images in the frame information recording queue that match the hand shape change characteristics of the swipe gesture, S5356 can be executed.

[0344] Optionally, if there are no consecutive multi-frame images in the frame information recording queue that match the hand shape change characteristics of the swipe gesture, S5353 can be executed.

[0345] S5353, based on the recognition strategy of flipping gesture, determines whether there are consecutive multi-frame images in the frame information recording queue that match the hand shape change characteristics of the flipping gesture.

[0346] Optionally, if there are multiple consecutive frames of images in the frame information recording queue that match the hand shape change characteristics of the flip gesture, S5357 can be executed.

[0347] Optionally, if there are no consecutive multi-frame images in the frame information recording queue that match the hand shape change characteristics of the flip gesture, S5354 can be executed.

[0348] S5354, based on the recognition strategy of pressing gesture, determines whether there are consecutive multi-frame images in the frame information recording queue that match the hand shape change characteristics of pressing gesture.

[0349] Optionally, if there are multiple consecutive frames of images in the frame information recording queue that match the hand shape change characteristics of the pressing gesture, S5358 can be executed.

[0350] Optionally, if there are no consecutive multi-frame images in the frame information recording queue that match the hand shape change characteristics of the pressing gesture, S5359 can be executed.

[0351] S5355, It is determined that there is currently an air gesture, and the type of the current air gesture is a grasping gesture.

[0352] S5356, It is determined that there is currently an air gesture, and the current air gesture type is a swipe gesture.

[0353] S5357, It is determined that there is currently an air gesture, and the current air gesture type is a flip gesture.

[0354] S5358, It is determined that an air gesture exists and the current air gesture type is a press gesture.

[0355] S5359, It is confirmed that there is currently no air gesture.

[0356] S536, the timeout detection unit determines whether the timeout condition is met; if the timeout condition is met, the air gesture recognition process is exited.

[0357] The air gesture recognition process can include all processes executed in the air gesture recognition module.

[0358] Optionally, the timeout detection unit may determine that the timeout condition is currently met under any of the following circumstances:

[0359] (1) The hand stability state detection unit did not detect the starting gesture image for N consecutive frames.

[0360] (2) The hand stability state detection unit detects stable state images for L consecutive frames.

[0361] (3) The gesture hovering state detection unit detected the gesture hovering state for the third consecutive time period.

[0362] The values ​​of N and L can be set according to the actual situation; for example, both N and L can be 10.

[0363] If the timeout condition is met, exiting the gesture recognition process can save phone power consumption.

[0364] Please see Figure 18 This is a schematic flowchart illustrating a gesture recognition method for air gestures, provided in another embodiment of this application. Figure 18 As shown, the air gesture recognition method may include steps S181 to S182, which are detailed below:

[0365] S181, after recognizing the swipe gesture at the first moment, the front camera captures the first image at the second moment; the second moment is later than the first moment.

[0366] The swipe gestures can include: swipe up, swipe down, swipe left, or swipe right.

[0367] S182, if the first image is detected to include a starting gesture at the third time, and the interval between the third time and the first time is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image is not displayed.

[0368] The starting gesture can include: a palm with fingers pointing upwards, a palm with fingers pointing to the left, a palm with fingers pointing to the right, a back of the hand with fingers pointing downwards, a back of the hand with fingers pointing to the left, or a back of the hand with fingers pointing to the right.

[0369] It should be noted that the specific details of S182 can be found in the relevant description in S533, and will not be repeated here.

[0370] In another embodiment of this application, after S181, the following may also be included:

[0371] If the first image is detected to include a starting gesture at the third time point, and the interval between the third time point and the first time point is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

[0372] It should be noted that the specific details of this step can be found in the relevant description in S533, and will not be repeated here.

[0373] In yet another embodiment of this application, after S181, the following may also be included:

[0374] If a starting gesture is detected in the first image at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is the same as the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image is displayed, the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information recording queue.

[0375] It should be noted that the specific details of this step can be found in the relevant description in S533, and will not be repeated here.

[0376] In yet another embodiment of this application, after S181, the following may also be included:

[0377] If a starting gesture is detected in the first image at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of a swipe gesture, and the gesture in the third image captured by the front camera within the second duration after the second moment is the same as the starting gesture in the first image, then a hand icon corresponding to the starting gesture in the first image is displayed, and a dynamic gesture recognition process is executed, and the information of the first image and the information of the third image are stored in the frame information recording queue; the second duration is less than the first duration.

[0378] It should be noted that the specific details of this step can be found in the relevant description in S533, and will not be repeated here.

[0379] In yet another embodiment of this application, after S181, the following may also be included:

[0380] If the interval between the third moment and the first moment is greater than the first duration, then the hand icon corresponding to the starting gesture in the first image is displayed, the dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information recording queue.

[0381] It should be noted that the specific details of this step can be found in the relevant description in S533, and will not be repeated here.

[0382] In yet another embodiment of this application, the air gesture recognition method may further include:

[0383] If other dynamic gestures besides swiping are recognized at the first moment, the dynamic gesture recognition process is executed.

[0384] It should be noted that the specific details of this step can be found in the relevant description in S533, and will not be repeated here.

[0385] In yet another embodiment of this application, prior to S181, the following may also be included:

[0386] Dynamic gestures are identified based on information from multiple frames in the frame information recording queue. The frame information recording queue is used to store information about the initial gesture image and the stable state image. The initial gesture image is an image that includes the initial gesture, and the stable state image is a non-initial gesture image with a stable hand shape.

[0387] Based on this, after S181, it may also include:

[0388] Delete information about all images related to the swipe gesture in the frame information record queue.

[0389] It should be noted that the details of this step can be found in the relevant description in S535, and will not be repeated here.

[0390] In yet another embodiment of this application, before recognizing dynamic gestures based on information from multiple frames of images in the frame information recording queue, the following may be included:

[0391] Acquire multiple consecutive frames of images, which are captured by the front-facing camera;

[0392] For each frame in a series of consecutive images, determine the hand feature information of the current frame image;

[0393] Based on the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable;

[0394] If the current frame image includes the initial gesture image, or if the hand shape in the current frame image is stable, the information of the current frame image is stored in the frame information recording queue; the information of the current frame image includes human hand feature information.

[0395] It should be noted that the details of this step can be found in the relevant descriptions in S51 to S53, and will not be repeated here.

[0396] In another embodiment of this application, determining whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable, based on the hand feature information of the current frame image, includes:

[0397] When the frame information recording queue is empty, determine whether the current frame image contains the starting gesture of any dynamic gesture based on the hand feature information of the current frame image.

[0398] If the frame information recording queue is not empty, determine whether the hand shape in the current frame image is stable based on the hand feature information of the current frame image.

[0399] It should be noted that the details of this step can be found in the relevant description in S532, and will not be repeated here.

[0400] In another embodiment of this application, the hand feature information includes hand type and finger orientation; based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture includes:

[0401] If the hand category corresponding to the current frame image is palm, and the fingers are pointing upwards, leftwards, or rightwards, then the current frame image is determined to contain the initial gesture; or...

[0402] If the hand category corresponding to the current frame image is the back of the hand, and the fingers are pointing downwards, to the left, or to the right, then the current frame image is determined to include the initial gesture.

[0403] In another embodiment of this application, the hand feature information includes hand type and finger orientation; based on the hand feature information of the current frame image, determining whether the current frame image includes the starting gesture of any dynamic gesture includes:

[0404] If the hand category corresponding to the current frame image is palm, and the fingers are not pointing upwards, leftwards, or rightwards, then the current frame image does not contain the initial gesture; or,

[0405] If the hand category corresponding to the current frame image is the back of the hand, and the fingers are not pointing downwards, to the left, or to the right, then the current frame image does not contain the initial gesture; or,

[0406] If the hand category corresponding to the current frame image is neither palm nor back of hand, it is determined that the current frame image does not include the starting gesture.

[0407] In another embodiment of this application, the hand feature information includes information about the hand detection box, hand category, coordinates of key points on the hand, and finger orientation; based on the hand feature information of the current frame image, determining whether the hand shape in the current frame image is stable includes:

[0408] Calculate the intersection-union ratio (IoU) between the hand detection bounding box in the current frame and the hand detection bounding box in the previous frame;

[0409] Calculate the standard deviation of the first-order difference between the hand key points in the current frame and the hand key points in the previous frame;

[0410] If the hand category in the current frame image is the same as the hand category in the previous frame image, and the finger orientation in the current frame image is the same as the finger orientation in the previous frame image, and the intersection-union ratio is greater than or equal to the first preset intersection-union ratio, and the standard deviation of the first difference is less than or equal to the preset standard deviation, then the hand shape in the current frame image is determined to be stable.

[0411] In another embodiment of this application, the air gesture recognition method further includes:

[0412] If the hand category in the current frame image is different from the hand category in the previous frame image, or the finger orientation in the current frame image is different from the finger orientation in the previous frame image, or the crossover ratio is less than the first preset crossover ratio, or the standard deviation of the first difference is greater than the preset standard deviation, the hand shape in the current frame image is determined to be unstable.

[0413] In yet another embodiment of this application, if it is determined that the hand shape in the current frame image is stable, the method further includes:

[0414] Based on the hand feature information of the current frame image and the hand feature information of the previous frame image, determine whether the current state is a hand hovering state.

[0415] If the gesture is not currently in a hover state, execute the dynamic gesture recognition process.

[0416] In another embodiment of this application, the hand feature information includes information about the hand detection box, hand category, coordinates of key points on the hand, and finger orientation; determining whether the hand is currently in a gesture hovering state based on the hand feature information of the current frame image and the hand feature information of the previous frame image includes:

[0417] Calculate the intersection-union ratio (IoU) between the hand detection bounding box in the current frame and the hand detection bounding box in the previous frame;

[0418] Based on the information of the hand detection box corresponding to the current frame image, calculate the first proportion of the hand detection box in the current frame image;

[0419] Based on the information of the hand detection box corresponding to the previous frame image, calculate the second proportion of the hand detection box corresponding to the previous frame image in the previous frame image;

[0420] Calculate the difference between the first percentage and the second percentage;

[0421] If the hand category in the current frame image is different from the hand category in the previous frame image, or the finger orientation in the current frame image is different from the finger orientation in the previous frame image, or the crossover ratio is less than the second preset crossover ratio, or the difference between the first ratio and the second ratio is greater than the preset difference, then it is determined that the current state is not in a gesture hovering state.

[0422] In yet another embodiment of this application, it further includes:

[0423] If the hand category in the current frame image is the same as the hand category in the previous frame image, and the finger orientation in the current frame image is the same as the finger orientation in the previous frame image, and the cross-union ratio is greater than or equal to the second preset cross-union ratio, and the difference between the first ratio and the second ratio is less than or equal to the preset difference, then it is determined that the current state is a gesture hovering state.

[0424] It should be noted that the details of this step can be found in the relevant description in S534, and will not be repeated here.

[0425] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, which, when invoked by a computer, causes the computer to perform one or more steps in any of the above method embodiments.

[0426] Based on the same technical concept, embodiments of this application also provide a chip system, including a processor coupled to a memory, which executes a computer-executable program stored in the memory to implement one or more steps in any of the above method embodiments. This chip system can be a single chip or a chip module composed of multiple chips.

[0427] Based on the same technical concept, this application also provides a computer executable program product that, when run on an electronic device, causes the electronic device to perform one or more steps in any of the above method embodiments.

[0428] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0429] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0430] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0431] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for recognizing air gestures, characterized in that, Applied to an electronic device including a front-facing camera, the method includes: After recognizing the swipe gesture at the first moment, the front-facing camera captures the first image at the second moment; the second moment is later than the first moment. If a starting gesture is detected in the first image at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the hand icon corresponding to the starting gesture in the first image will not be displayed, and dynamic gesture recognition will not be performed. If a starting gesture is detected in the first image at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, and the gesture in the third image captured by the front camera within the second duration after the second moment is the same as the starting gesture in the first image, then a hand icon corresponding to the starting gesture in the first image is displayed, and a dynamic gesture recognition process is executed, and the information of the first image and the information of the third image are stored in the frame information recording queue; the second duration is less than the first duration.

2. The air gesture recognition method according to claim 1, characterized in that, Also includes: If a starting gesture is detected in the first image at a third time point, and the interval between the third time point and the first time point is less than or equal to a first duration, and the starting gesture in the first image is different from the starting gesture of the swipe gesture, then the dynamic gesture recognition process is not executed, and the information of the first image is not stored.

3. The air gesture recognition method according to claim 1 or 2, characterized in that, Also includes: If a starting gesture is detected in the first image at the third moment, and the interval between the third moment and the first moment is less than or equal to the first duration, and the starting gesture in the first image is the same as the starting gesture of the swipe gesture, then a hand icon corresponding to the starting gesture in the first image is displayed, a dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information recording queue.

4. The air gesture recognition method according to claim 1 or 2, characterized in that, Also includes: If the interval between the third moment and the first moment is greater than the first duration, then a hand icon corresponding to the starting gesture in the first image is displayed, and a dynamic gesture recognition process is executed, and the information of the first image is stored in the frame information recording queue.

5. The air gesture recognition method according to claim 1 or 2, characterized in that, Also includes: If other dynamic gestures besides the swipe gesture are identified at the first moment, the dynamic gesture recognition process is executed.

6. The air gesture recognition method according to any one of claims 1-5, characterized in that, The swipe gestures include: swipe up, swipe down, swipe left, or swipe right.

7. The air gesture recognition method according to any one of claims 1-5, characterized in that, The starting gesture includes: a palm with fingers pointing upwards, or a palm with fingers pointing to the left, or a palm with fingers pointing to the right, or the back of the hand with fingers pointing downwards, or the back of the hand with fingers pointing to the left, or the back of the hand with fingers pointing to the right.

8. The air gesture recognition method according to any one of claims 1-7, characterized in that, Before recognizing the swipe gesture at the first moment, it also includes: Dynamic gestures are identified based on information from multiple frames in the frame information recording queue; the frame information recording queue is used to store information of the initial gesture image and information of the stable state image, wherein the initial gesture image is an image including the initial gesture, and the stable state image is a non-initial gesture image with a stable hand shape; After recognizing the swipe gesture at the first moment, it also includes: Delete information about all images related to the swipe gesture from the frame information record queue.

9. The air gesture recognition method according to claim 8, characterized in that, Before recognizing dynamic gestures based on information from multiple frames in the frame information recording queue, the process also includes: Acquire multiple consecutive frames of images, which are captured by the front-facing camera; For each frame in the consecutive multi-frame images, determine the hand feature information of the current frame image; Based on the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable; If the current frame image includes a starting gesture image, or if the hand shape in the current frame image is stable, the information of the current frame image is stored in a frame information recording queue; the information of the current frame image includes the human hand feature information.

10. The air gesture recognition method according to claim 9, characterized in that, Based on the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, or whether the hand shape in the current frame image is stable, including: When the frame information recording queue is empty, determine whether the current frame image contains the starting gesture of any dynamic gesture based on the hand feature information of the current frame image. If the frame information recording queue is not empty, determine whether the hand shape in the current frame image is stable based on the hand feature information of the current frame image.

11. The air gesture recognition method according to claim 10, characterized in that, The hand feature information includes hand type and finger orientation; Based on the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, including: If the hand category corresponding to the current frame image is palm, and the fingers are pointing upwards, to the left, or to the right, then the current frame image is determined to include the initial gesture; or, If the hand category corresponding to the current frame image is the back of the hand, and the fingers are pointing downwards, to the left, or to the right, then the current frame image is determined to include the initial gesture.

12. The air gesture recognition method according to claim 10, characterized in that, The hand feature information includes hand type and finger orientation; Based on the hand feature information of the current frame image, determine whether the current frame image includes the starting gesture of any dynamic gesture, including: If the hand category corresponding to the current frame image is palm, and the fingers are not pointing upwards, to the left, or to the right, then the current frame image is determined to not contain the initial gesture; or, If the hand category corresponding to the current frame image is the back of the hand, and the fingers are not pointing downwards, to the left, or to the right, then the current frame image is determined to not contain the initial gesture; or, If the hand category corresponding to the current frame image is neither palm nor back of hand, it is determined that the current frame image does not include the starting gesture.

13. The air gesture recognition method according to claim 10, characterized in that, The hand feature information includes information about the hand detection box, hand category, coordinates of key points on the hand, and finger orientation; Based on the hand feature information of the current frame image, determine whether the hand shape in the current frame image is stable, including: Calculate the intersection-union ratio (IoU) between the hand detection bounding box corresponding to the current frame image and the hand detection bounding box corresponding to the previous frame image; Calculate the standard deviation of the first-order difference between the hand key points corresponding to the current frame image and the hand key points corresponding to the previous frame image; If the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection-union ratio is greater than or equal to a first preset intersection-union ratio, and the standard deviation of the first difference is less than or equal to a preset standard deviation, then the hand shape in the current frame image is determined to be stable.

14. The air gesture recognition method according to claim 13, characterized in that, Also includes: If the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the intersection-union ratio is less than the first preset intersection-union ratio, or the standard deviation of the first-order difference is greater than the preset standard deviation, the hand shape in the current frame image is determined to be unstable.

15. The air gesture recognition method according to claim 10, characterized in that, If the hand shape in the current frame image is determined to be stable, the method further includes: Based on the hand feature information of the current frame image and the hand feature information of the previous frame image, determine whether the current state is a gesture hovering state. If the gesture is not currently in a hover state, execute the dynamic gesture recognition process.

16. The air gesture recognition method according to claim 15, characterized in that, The hand feature information includes information about the hand detection box, hand category, coordinates of key points on the hand, and finger orientation; Based on the hand feature information of the current frame image and the hand feature information of the previous frame image, determine whether the current state is a gesture hovering state, including: Calculate the intersection-union ratio (IoU) between the hand detection bounding box corresponding to the current frame image and the hand detection bounding box corresponding to the previous frame image; Based on the information of the hand detection box corresponding to the current frame image, calculate the first proportion of the hand detection box in the current frame image; Based on the information of the hand detection box corresponding to the previous frame image, calculate the second proportion of the hand detection box in the previous frame image. Calculate the difference between the first percentage and the second percentage; If the hand category corresponding to the current frame image is different from the hand category corresponding to the previous frame image, or the finger orientation corresponding to the current frame image is different from the finger orientation corresponding to the previous frame image, or the crossover ratio is less than the second preset crossover ratio, or the difference between the first ratio and the second ratio is greater than the preset difference, then it is determined that the current state is not in a gesture hovering state.

17. The air gesture recognition method according to claim 16, characterized in that, Also includes: If the hand category corresponding to the current frame image is the same as the hand category corresponding to the previous frame image, and the finger orientation corresponding to the current frame image is the same as the finger orientation corresponding to the previous frame image, and the intersection-union ratio is greater than or equal to the second preset intersection-union ratio, and the difference between the first percentage and the second percentage is less than or equal to the preset difference, then it is determined that the current state is a gesture hovering state.

18. An electronic device, characterized in that, include: One or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 17.

20. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 17.

Images