Hand direction detection method, electronic device and readable medium

Abstract

The present application provides a method for detecting the direction of a hand, an electronic device, and a readable medium. The display screen of the electronic device is lit, and the camera is triggered to take a picture. The image taken by the camera is obtained. If the image includes a human hand, the angle between the finger direction and the horizontal direction of the human hand in the image can be calculated, and then the starting gesture of the user inputting the human-computer interaction is determined. Afterwards, the electronic device determines the angle between the finger direction and the horizontal direction of the hand in the image, and if the condition that the finger is facing the first direction is satisfied, a hand icon with the finger facing the first direction is displayed on the display screen; if the angle between the finger direction and the horizontal direction of the hand in the image is satisfied, a hand icon with the finger facing the second direction is displayed on the display screen, and the user is reminded that the starting gesture has been input by displaying the hand icon on the display screen, and the user can input the air gesture for human-computer interaction.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and in particular to a hand direction detection method, electronic device, computer program product, and computer readable storage medium. Background Art

[0002] At present, electronic devices can support non-contact air gesture interaction technology. Air gesture interaction technology can perform human-computer interaction in usage scenarios where it is inconvenient for users to touch electronic devices, thereby improving the user experience.

[0003] There is an urgent need for a solution that can support electronic devices to respond to the starting gesture of human-computer interaction input by users and output feedback information to remind users that they can input air gestures for human-computer interaction. Summary of the invention

[0004] The present application provides a hand direction detection method, an electronic device, a computer program product and a computer-readable storage medium, the purpose of which is to enable the electronic device to output feedback information after detecting the user's input of a starting gesture for human-computer interaction, so as to remind the user that he can input an air gesture for human-computer interaction.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In the first aspect, the present application provides a method for detecting the direction of a hand applied to an electronic device, the electronic device including a display screen and a camera, the method for detecting the direction of a hand comprising: in an application scenario where the display screen of the electronic device is in a bright screen state, obtaining an image captured by the camera of the electronic device, and calculating the angle between the finger direction of the hand in the image and the horizontal direction; if it is determined that the angle between the finger direction of the hand in the image and the horizontal direction satisfies the condition that the finger is facing a first direction, a hand icon with the finger facing the first direction is displayed on the display screen; if it is determined that the angle between the finger direction of the hand in the image and the horizontal direction satisfies the condition that the finger is facing a second direction, a hand icon with the finger facing the second direction is displayed on the display screen. Wherein: the second direction is usually the opposite direction of the first direction, for example, if the first direction is an upward direction, the second direction is a downward direction.

[0007] It can be seen from the above content that: the display screen of the electronic device lights up, the camera is triggered to record an image, and the image captured by the camera is obtained. If the image includes a human hand, the angle between the finger direction and the horizontal direction of the human hand in the image can be calculated, and then the starting gesture of the user inputting the human-computer interaction is determined. Afterwards, the electronic device determines the angle between the finger direction and the horizontal direction of the hand in the image, and if the condition that the finger is facing the first direction is satisfied, a hand icon with the finger facing the first direction is displayed on the display screen; if the angle between the finger direction and the horizontal direction of the hand in the image is satisfied, a hand icon with the finger facing the second direction is displayed on the display screen, and the user is reminded that the starting gesture has been input by displaying the hand icon on the display screen, and the user can input the air gesture for human-computer interaction.

[0008] Furthermore, if the first direction is an upward direction and the second direction is a downward direction, a hand icon with a finger pointing upward is displayed on the display screen, which can remind the user that the starting gesture of the air gesture has been input and that air gestures such as sliding down or grasping can be input. A hand icon with a finger pointing downward is displayed on the display screen, which can remind the user that the starting gesture of the air gesture has also been input and that air gestures such as sliding up can be input. The user can also check whether the starting gesture he input is correct according to the hand icon displayed on the display screen.

[0009] Displaying a hand-shaped icon with the finger pointing upward on the display screen means that the electronic device predicts that the user is going to input an air gesture such as sliding down or grasping. The electronic device can call a software process that responds to air gestures such as sliding down or grasping while the hand-shaped icon with the finger pointing upward is displayed on the display screen, thereby improving the response speed of the electronic device. Similarly, displaying a hand-shaped icon with the finger pointing downward on the display screen means that the electronic device predicts that the user is going to input an air gesture such as swiping up. The electronic device can call a software process that responds to air gestures such as swiping up while the hand-shaped icon with the finger pointing downward is displayed on the display screen, thereby improving the response speed of the electronic device.

[0010] In a possible implementation, after acquiring the image captured by the camera of the electronic device, the method further includes: detecting whether the hand in the image is a palm or a back of the hand.

[0011] In the above possible implementation manner, after the electronic device obtains the image taken by the camera, it detects that the hand in the image is a palm and back of the hand type, indicating that the user's intention is to operate the electronic device through air gestures. The electronic device then performs an image analysis of the finger direction and horizontal angle of the hand in the image, thereby verifying the predicted result that the user's intention is to operate the electronic device through air gestures.

[0012] In one possible implementation, after acquiring the image captured by the camera of the electronic device, it also includes: cropping a hand area image from the image; detecting the coordinates of the hand key points of the hand area image; calculating the angle between the fingers of the hand in the image and the horizontal direction, including: using the coordinates of the hand key points to calculate the angle between the finger direction vector of the hand and the horizontal direction.

[0013] In one possible implementation, the coordinates of the key points of the hand are used to calculate the angle between the finger direction vector of the hand and the horizontal direction, including: calculating the vector from the finger base coordinates of the hand in the coordinates of the key points of the hand to the fingertip coordinates to obtain the finger direction vector of the hand in the image; calculating the angle between the finger direction vector of the hand in the image and the horizontal direction.

[0014] In this possible implementation, the hand key points include the bases and fingertips of the fingers, and the vector from the base of the finger to the fingertip can reflect the direction of the finger. Therefore, the finger direction vector of the hand in the image can be calculated using the base coordinates and fingertip coordinates of the hand in the coordinates of the hand key points.

[0015] In one possible implementation, a vector pointing from the finger root coordinates of the hand in the coordinates of the hand key points to the fingertip coordinates is calculated to obtain the finger direction vector of the hand in the image, including: using the finger root coordinates of multiple fingers in the coordinates of the hand key points to calculate the average finger root coordinates of the multiple fingers in the image, and using the fingertip coordinates of multiple fingers in the coordinates of the hand key points to calculate the average fingertip coordinates of the multiple fingers in the image, wherein the multiple fingers include at least the middle finger; calculating a vector pointing from the average finger root coordinates of the multiple fingers to the average fingertip coordinates of the multiple fingers to obtain the finger direction vector of the hand in the image.

[0016] In this possible implementation, the finger direction of the middle finger is usually the same as the overall direction of the finger, but in some specific hand shapes, the finger is facing upward as a whole, but the middle finger is facing other directions. In order to avoid the finger direction vector calculated by only using the root coordinates of the middle finger to point to the fingertip coordinates being inaccurate, the root coordinates of multiple fingers in the coordinates of the hand key points are first used to calculate the average root coordinates of the multiple fingers in the image, and the fingertip coordinates of multiple fingers in the coordinates of the hand key points are used to calculate the average fingertip coordinates of the multiple fingers in the image, and then the vector from the average root coordinates of multiple fingers to the average fingertip coordinates of multiple fingers is calculated to obtain the finger direction vector of the hand in the image.

[0017] In one possible implementation, the coordinates of the key points of the hand are used to calculate the angle between the finger direction vector of the hand and the horizontal direction, including: using the base coordinates and fingertip coordinates of multiple fingers in the coordinates of the key points of the hand to calculate the finger direction vectors of multiple fingers of the hand in the image, the finger direction vectors of the multiple fingers at least include the finger direction vector of the middle finger; calculating the angles between the finger direction vectors of the multiple fingers and the horizontal direction to obtain multiple angles; calculating the average of the multiple angles to obtain the angle between the finger direction of the hand in the image and the horizontal direction.

[0018] In this possible implementation manner, in order to avoid the finger direction vector calculated only by pointing the root coordinates of the middle finger to the fingertip coordinates being inaccurate, the root coordinates and fingertip coordinates of multiple fingers in the coordinates of the key points of the hand are used to calculate the finger direction vectors of multiple fingers of the hand in the image, and the angles between the finger direction vectors of the multiple fingers and the horizontal direction are calculated to obtain multiple angles; the average of the multiple angles is calculated to obtain the angle between the finger direction of the hand in the image and the horizontal direction.

[0019] In one possible implementation, a method for determining whether the angle between the finger direction of a hand in an image and the horizontal direction satisfies a condition that the finger is facing a first direction includes determining whether the angle between the finger direction of the hand in the image and the horizontal direction is within a range of angles corresponding to the finger facing upward.

[0020] In a possible implementation, if it is determined that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy the condition that the finger is facing a first direction before the condition that the finger is facing a first direction is satisfied, it also includes: determining that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy the condition that the finger is facing a third direction.

[0021] In this possible implementation, the third direction is other directions, for example, the first direction is the upward direction, the second direction is the downward direction, and the third direction may refer to the left or right, or both the left and right directions. Based on this, before judging the angle between the finger direction of the hand in the image and the horizontal direction, the angle between the finger direction of the hand in the image and the horizontal direction is judged to satisfy the condition of the finger pointing to the first direction. If the condition of the finger pointing to the third direction is not satisfied, it can be avoided that the angle between the finger direction of the hand in the image and the horizontal direction satisfies the condition of the third direction, but is misjudged as satisfying the condition of the first direction, resulting in the display of a hand icon on the display screen.

[0022] In one possible implementation, determining that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy the condition that the finger is facing a third direction includes: determining that the angle between the finger direction of the hand in the image and the horizontal direction is not within the angle range corresponding to the finger facing left and right.

[0023] In one possible implementation, if it is determined that the angle between the finger direction and the horizontal direction of the hand in the image meets the condition that the finger is facing the second direction, a hand icon with the finger facing the second direction is displayed on the display screen, including: if it is determined that the angle between the finger direction and the horizontal direction of the hand in the image does not meet the condition that the finger is facing the first direction, a hand icon with the finger facing the second direction is displayed on the display screen.

[0024] In a second aspect, the present application provides an electronic device comprising: one or more processors, a memory and a display screen; the memory and the display screen are coupled to the one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the hand direction detection method described in any one of the first aspects.

[0025] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the hand direction detection method as described in any one of the first aspects.

[0026] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer executes the hand direction detection method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An application scenario diagram of a hand direction detection method provided in an embodiment of the present application;

[0028] Figure 2 A diagram of an application scenario of another hand direction detection method provided in an embodiment of the present application;

[0029] Figure 3 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0030] Figure 4 A software framework diagram of an electronic device provided in an embodiment of the present application;

[0031] Figure 5 A signaling diagram of a hand direction detection method provided in an embodiment of the present application;

[0032] Figure 6 A display diagram of 21 key points of a human hand provided in an embodiment of the present application;

[0033] Figure 7 A diagram showing the angle between the vector of the average coordinate of the finger root pointing to the average coordinate of the fingertip and the horizontal direction provided in an embodiment of the present application;

[0034] Figure 8A signaling diagram of another hand direction detection method provided in an embodiment of the present application;

[0035] Fig. 9 A diagram showing the angles between the direction vectors of multiple fingers and the horizontal direction provided in an embodiment of the present application;

[0036] Fig.10 A flowchart of a method for detecting hand direction provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0038] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0039] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0040] At present, electronic devices can support contactless air gesture interaction technology. Air gesture interaction technology can perform human-computer interaction in usage scenarios where it is inconvenient for users to contact electronic devices, thereby improving the user experience. In addition, before a user uses air gestures to perform human-computer interaction with an electronic device, the user can trigger the electronic device to prepare for air gesture human-computer interaction with the user by inputting a starting gesture. Correspondingly, after the user inputs the starting gesture, the electronic device can also output feedback information to prompt the user that the starting gesture has been completed and that air gestures can be used for human-computer interaction operations.

[0041] Take mobile phones as an example of electronic devices. Figure 1 and Figure 2 Two ways of prompting users to perform air gestures are shown. After the phone's display is on and unlocked, the user's hand is within the FOV of the phone's front camera. The FOV refers to the range covered by the front camera lens. The phone's display can display a hand icon to prompt the user to perform air gestures to interact with the phone.

[0042] Figure 1 In the example shown in (a) of FIG. 1 , the display screen of the mobile phone displays the setting interface, and the user Figure 1 As shown in (b), the fingers of the hand are facing upward, the palm is facing the display screen, the hand is within 20cm-40cm away from the mobile phone and within the field of view (FOV) of the front camera of the mobile phone, and the mobile phone displays a small hand icon 101 on the setting interface, and the fingers of the small hand icon 101 are also facing upward.

[0043] Figure 2 In the example shown in (a) of FIG. 1 , the mobile phone display also displays the settings interface. The user Figure 2 As shown in (b), the fingers of the hand are facing downward, the back of the hand is facing the display screen, the hand is within 20cm-40cm away from the mobile phone and within the field of view FOV of the front camera of the mobile phone, and the mobile phone displays a small hand icon 102 on the setting interface, and the fingers of the small hand icon 102 are facing downward.

[0044] The user sees the display screen of the mobile phone Figure 1 (b) shows a small hand icon 101, or shows Figure 2 After clicking the small hand icon 102 shown in (b) in the figure, you can make air gestures to interact with the mobile phone and control the mobile phone.

[0045] It should be noted that the fingers pointing upwards proposed in the present application means that the fingers are pointing toward the top of an electronic device such as a mobile phone, which can be understood as the fingertips are close to the top of the device, where a camera is usually provided, and the fingers pointing downwards means that the fingers are pointing toward the bottom of an electronic device such as a mobile phone, which can be understood as the fingertips are close to the bottom of the device.

[0046] Figure 1 (b) and Figure 2 The small hand icon shown in (b) of FIG. 1 can also be called a hand-shaped icon. Its style is an exemplary style and does not constitute a limitation on the display of the small hand icon on the display screen. It can be understood that when the finger is facing upwards facing the display screen of the mobile phone, the display screen of the mobile phone displays the small hand icon with the finger facing upwards, and when the finger is facing downwards facing the display screen of the mobile phone, the display screen of the mobile phone displays the small hand icon with the finger facing downwards.

[0047] Trigger mobile phones and other electronic devices to display Figure 1 The hand shape of the user of the small hand icon 101 shown in (b) is not limited to Figure 1 The hand shape shown in (b) in the figure. The hand shape of the user's hand can be: fingers facing upward, fingers partially or fully open, or five fingers together; the palm of the user's hand can face the display screen, or the back of the hand can face the display screen. It only needs to ensure that the hand shape of the user's hand meets the requirements: the hand faces the display screen with most fingers facing upward, and the hand is within the FOV of the front camera of the mobile phone.

[0048] Trigger mobile phones and other electronic devices to display Figure 2 The hand shape of the user of the small hand icon 102 shown in (b) is not limited to Figure 2 The hand shape shown in (b) in the figure. The hand shape of the user's hand can be: fingers facing downward, fingers partially or fully open, or five fingers together; the palm of the user's hand can face the display screen, or the back of the hand can face the display screen. It only needs to ensure that the hand shape of the user's hand meets the requirements: the hand faces the display screen with most fingers facing downward, and the hand is within the FOV of the front camera of the mobile phone.

[0049] To achieve Figure 1 and Figure 2 The invention shows a small hand icon on the display screen to prompt the user to make a gesture in the air. The embodiment of the present application provides a method for detecting the hand direction. The method for detecting the hand direction provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, personal digital assistants (PDAs), desktops, laptops, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices. The present application does not impose any special restrictions on the specific forms of the above electronic devices.

[0050] Take mobile phones as an example. Figure 3 This is an example of the composition of an electronic device provided in an embodiment of the present application. Figure 3As shown, the electronic device 300 may include a processor 310, an internal memory 320, a sensor module 330, a mobile communication module 340, a wireless communication module 350, a display screen 360, an audio module 370, and the like.

[0051] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 300. In other embodiments, the electronic device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 310 may include one or more processing units, for example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a smart sensor hub and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0053] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0054] The internal memory 320 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 310 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the internal memory 320. The internal memory 320 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 310 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the internal memory 320, and / or the instructions stored in the memory provided in the processor.

[0055] In some embodiments, the internal memory 320 stores instructions for executing hand direction detection. The processor 310 can control the electronic device to detect that the hand is within the FOV of the front camera and display the small hand icon by executing the instructions stored in the internal memory 320.

[0056] The electronic device implements the display function through a GPU, a display screen 360, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 360 and the application processor. The GPU is used for image rendering by performing mathematical and geometric calculations. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0057] The display screen 360 is used to display images, video interfaces, etc. The display screen 360 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 360, where N is a positive integer greater than 1.

[0058] The wireless communication function of the electronic device 300 can be implemented through the antenna 1, the antenna 2, the mobile communication module 340, the wireless communication module 350, the modem processor and the baseband processor.

[0059] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 300 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0060] The mobile communication module 340 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 300. The mobile communication module 340 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 340 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 340 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 340 can be set in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 340 can be set in the same device as at least some of the modules of the processor 310.

[0061] The wireless communication module 350 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 300. The wireless communication module 350 can be one or more devices integrating at least one communication processing module. The wireless communication module 350 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 310. The wireless communication module 350 can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0062] In the sensor module 330, the pressure sensor 330A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 330A can be set on the display screen 360. There are many types of pressure sensors 330A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 330A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 360, the electronic device detects the intensity of the touch operation based on the pressure sensor 330A. The electronic device can also calculate the position of the touch based on the detection signal of the pressure sensor 330A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0063] The touch sensor 330B is also called a "touch control device". The touch sensor 330B can be set on the display screen 360, and the touch sensor 330B and the display screen 360 form a touch screen, also called a "touch control screen". The touch sensor 330B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 360. In other embodiments, the touch sensor 330B can also be set on the surface of the electronic device, which is different from the position of the display screen 360.

[0064] The electronic device 300 can implement audio functions such as music playing and recording through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0065] The audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 370 can also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be arranged in the processor 310, or some functional modules of the audio module 370 can be arranged in the processor 310.

[0066] The speaker 370A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 370A.

[0067] The receiver 370B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be received by placing the receiver 330B close to the human ear.

[0068] Microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to the microphone 370C to input the sound signal into the microphone 370C. The electronic device can be provided with at least one microphone 370C. In other embodiments, the electronic device can be provided with two microphones 370C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 370C to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording function, etc.

[0069] The earphone interface 370D is used to connect a wired earphone. The earphone interface 330D may be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0070] In addition, the electronic device runs an operating system on top of the above components. operating system, operating system, Operating system, etc. Application programs can be installed and run on the operating system.

[0071] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application.

[0072] The layered architecture divides the operating system of the electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system of the electronic device is an Android system. The Android system, from top to bottom, includes an application (APP) layer, an application framework layer (FWK for short), a system library, a kernel layer, and a hardware layer.

[0073] The application layer can include a series of application packages. Figure 4 As shown, the application package may include applications such as gallery, map, calendar, call, camera, and system user interface (SystemUI).

[0074] SystemUI is a system-level application. In some embodiments, SystemUI can be used to draw Figure 1 The hand icon shown in (b) 101 or Figure 2 The small hand icon 102 shown in (b) above controls the display of the small hand icon 101 or the small hand icon 102 on the display screen.

[0075] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 4 As shown, the application framework layer may include a window management service (WindowManagerService, WMS), an activity management service (ActivityManagerService, AMS), a content provider, a phone manager, and a resource manager, etc.

[0076] The window management service is used to manage window programs. The window management service can realize the addition, deletion, display and hiding control of windows, etc.

[0077] The activity management service is used to manage activity behavior, control the life cycle of the activity, dispatch message time, and memory management.

[0078] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0079] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0080] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0081] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. In some embodiments of the present application, the application cold start will run in the Android runtime, and the Android runtime will obtain the optimized file status parameters of the application. Then, the Android runtime can determine whether the optimized file is outdated due to system upgrades based on the optimized file status parameters, and return the judgment result to the application management module.

[0082] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0083] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0084] The system library may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0085] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0086] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0087] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.

[0088] A 2D graphics engine is a drawing engine for 2D drawings.

[0089] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms.

[0090] In some embodiments, the hardware abstraction layer may include an image processing module, which is used to perform type recognition and hand direction recognition on the image captured by the camera and obtain recognition results. The specific working method of the image processing module can be found in the following content.

[0091] The kernel layer is a layer between hardware and software. The kernel layer includes at least display driver, sensor driver, camera driver, etc. In some embodiments, the display driver is used to control the display screen to display images; the sensor driver is used to control the operation of multiple sensors, such as controlling the operation of pressure sensors and touch sensors. The camera driver is used to drive the camera to operate.

[0092] Under the above five-layer architecture, the electronic device is also provided with a hardware layer, which may include the above-mentioned electronic device hardware components. Figure 4 The display and camera are shown.

[0093] It should be noted that although the embodiments of the present application are based on The system is used as an example, but the basic principles are also applicable to Electronic devices with other operating systems.

[0094] The following takes a mobile phone as an example, and introduces that when a mobile phone or other electronic device detects that a hand is within the field of view (FOV) of the front camera of the mobile phone, the display screen of the mobile phone can display a hand icon to prompt the user that the user can perform air gestures to interact with the mobile phone.

[0095] Figure 5 A signaling diagram of a hand direction detection method provided in an embodiment of the present application is shown.

[0096] like Figure 5 As shown, the hand direction detection method provided in the embodiment of the present application includes:

[0097] S501: A camera captures a first image.

[0098] After the display screen of the mobile phone is lit and unlocked, the camera of the mobile phone is triggered to start capturing images. In some embodiments, the camera of the mobile phone can capture images at intervals of a certain length, and the length of the interval can be set.

[0099] An image captured by a camera of a mobile phone may be referred to as a first image. A camera of a mobile phone generally refers to a front camera of the mobile phone. Of course, the present application is not limited to the front camera.

[0100] In some embodiments, the display screen of the mobile phone is on and in the lock screen interface, and the camera of the mobile phone can also be triggered to start capturing images.

[0101] S502: An image processing module obtains a first image captured by a camera.

[0102] The image processing module is arranged in the hardware abstraction layer, and a data transmission channel is arranged between the image processing module and the camera. After the camera takes a first image, the image processing module obtains the first image taken by the camera through the data transmission channel.

[0103] S503: The image processing module detects whether the hand in the first image is a palm or back of hand type.

[0104] After the image processing module acquires the first image, it performs hand detection on the first image to detect the type of hand in the first image. The types of hands may generally include: palm and back of hand type, fist type, other gesture types, and background type. The palm and back of hand type means that the first image includes part or all of the back of the hand or the palm of the hand; the fist type means that the hand shape of the hand included in the first image is a fist shape; the background type means that the first image does not include the hand, and the other gesture type means that the first image includes the hand, but the hand is not in a fist shape, and is not in a shape showing the palm or back of the hand.

[0105] In some embodiments, the image processing module calls a hand type detection model to perform hand detection on the first image to obtain confidences of multiple hand types, and the hand type with the highest confidence is the category of the hand in the first image.

[0106] The hand type detection model can usually be obtained by training a convolutional neural network using sample images including hands. This application does not provide a detailed introduction to the training process of the convolutional neural network, and reference can be made to conventional techniques.

[0107] The image processing module detects that the hand in the first image is not a palm and back type, and the following step S504 can be executed. The image processing module detects that the hand in the first image is a palm and back type, and the following steps S505 to S513 can be executed.

[0108] In some embodiments, during the process of the image processing module detecting the hand on the first image, the position information and confidence of the hand in the first image can also be obtained; the position information of the hand is used to indicate the bounding box of the hand in the first image, which can be called the hand detection box, and the confidence is used to indicate the credibility of the position information of the hand. The higher the confidence, the higher the feasibility of the position information of the hand. In some embodiments, the position information of the hand can be the coordinate information of the four vertices of the matrix of the hand detection box.

[0109] S504: The image processing module sends a first message to SystemUI, where the first message indicates that the direction of the hand in the first image is unclear.

[0110] SystemUI is a system application at the application layer. The image processing module detects that the hand in the first image is not a palm or back of hand type using step S503. The image processing module can use transparent transmission to send the first message to the module at the application framework layer. The module at the application framework layer uses an interface callback method to send the first message to SystemUI. The first message can indicate that the direction of the hand in the first image is unclear, that is, the direction of the hand in the first image is UNKNOW. SystemUI receives the first message and usually does not perform any operation.

[0111] The image processing module uses step S503 to detect that the hand in the first image is not a palm or back of the hand type, which means that the first image taken by the camera does not include a hand, or the hand in the first image is a fist gesture or other gesture. It can be inferred that the user's intention is not to operate the mobile phone through air gestures, and the mobile phone can avoid executing the following steps S505 to S513. When the user does not need to operate the mobile phone through air gestures, a small hand icon is displayed on the display screen to remind the user to input air gestures, which causes trouble to the user and saves power consumption.

[0112] In some embodiments, the first message sent by the image processing module to the SystemUI may also indicate that the hand in the first image is facing left or right. The SystemUI displays a small hand icon with a finger facing left or right on the display screen.

[0113] S505: The image processing module crops the first image to obtain a hand area image.

[0114] The image processing module uses step S503 to detect that the hand in the first image is a palm and back of the hand type, indicating that the user's intention is to operate the mobile phone through air gestures. The image processing module can further identify the hand area image to verify whether the prediction of the user's intention to operate the mobile phone through air gestures is correct.

[0115] In some embodiments, the image processing module performs hand detection on the first image to obtain a hand detection frame. The image processing module may crop the hand detection frame from the first image to obtain a hand region image.

[0116] S506: The image processing module detects and obtains the coordinates of the hand key points in the hand area image.

[0117] The key points of the hand can be located at one or more finger joints, fingertips, the connection points between fingers and palms, and wrists of different fingers. They usually include the fingertips, joints, and bases of multiple fingers, and may also include the center point of the wrist.

[0118] For example, the number of key points of a human hand may be 21. Figure 6 Shows 21 hand key points of the human hand.

[0119] like Figure 6 As shown in FIG. 1 , the 21 key points of the human hand include: the center point of the wrist 0, the bases of the five fingers 1, 5, 9, 13 and 17, the fingertips of the five fingers 4, 8, 12, 16 and 19, and the joints of the five fingers 2, 3, 6, 7, 10, 11, 14, 15, 18 and 19. This step detects the coordinates of the key points of the hand in the hand area image, which can be understood as: the image processing module obtains the coordinates of the key points of the hand in the hand area image. Figure 6 The coordinate values ​​of each hand key point in the displayed xy coordinate system.

[0120] The embodiment of the present application takes the coordinates of 21 hand key points detected by the image processing module as an example for introduction, but the image processing module is not limited to only detecting the coordinates of 21 hand key points. In some embodiments, the image processing module can obtain hand key points including fingertips and finger roots of multiple fingers. For example, the hand key points include finger roots 1, 5, 9, 13 and 17, and fingertips 4, 8, 12, 16 and 19.

[0121] In some embodiments, the image processing module calls a pre-trained key point detection model to identify the hand area image and obtain the coordinates of the hand key points in the hand area image. In some embodiments, the key point detection model can use a deep neural network model, and exemplary deep neural network models such as ResNet network structure, MobileNet network structure, NASNet network structure, etc. can be used.

[0122] S507: The image processing module calculates a first vector using the coordinates of the key points. The first vector is a vector of average coordinates of the finger base and average coordinates of the fingertip.

[0123] The direction of the fingers reflected by the directions of the index finger, middle finger and ring finger of a human hand is usually the same as the overall direction of the fingers of the human hand. In this way, the image processing module can calculate the direction of the fingers reflected by the directions of the index finger, middle finger and ring finger of a human hand.

[0124] In some embodiments, the image processing module obtains the coordinates of the bases and fingertips of the index finger, middle finger, and ring finger from the coordinates of the key points of the hand. Figure 7As shown, the image processing module calculates the average coordinate P1 of the root 5 of the index finger, the root 9 of the middle finger, and the root 13 of the ring finger, and the average coordinate P1 is the average coordinate of the root; the image processing module calculates the average coordinate P2 of the fingertip 8 of the index finger, the fingertip 12 of the middle finger, and the fingertip 16 of the ring finger, and the average coordinate P2 is the average coordinate of the fingertip. The image processing module uses the average coordinates P1 and the average coordinates P2 to calculate the vector from the average coordinate P1 to the average coordinate P2, which is called the first vector. The direction of the first vector can be understood as the overall direction of the finger.

[0125] After the image processing module calculates the first vector, it can also calculate the angle θ between the first vector and the horizontal x-axis.

[0126] In some cases, the direction of the middle finger of the hand is also the same as the overall direction of the fingers of the human hand, so the image processing module can calculate the direction of the middle finger. In some embodiments, the image processing module uses the coordinates of the root 9 of the middle finger and the coordinates of the fingertip 12 to calculate the vector from the root 9 of the middle finger to the fingertip 12, which is also called the first vector.

[0127] In some embodiments, the image processing module may also calculate the average coordinates of the base 5 of the index finger and the base 9 of the middle finger, and calculate the average coordinates of the fingertip 8 of the index finger and the fingertip 12 of the middle finger, and the image processing module calculates the vector pointing from the average coordinate to the average coordinate as the first vector.

[0128] In other embodiments, the image processing module may also calculate the average coordinates of the root 9 of the middle finger and the root 13 of the ring finger, and calculate the average coordinates of the fingertip 12 of the middle finger and the fingertip 16 of the ring finger, and the image processing module calculates the vector pointing from the average coordinate to the average coordinate as the first vector.

[0129] When a user inputs a starting gesture, he may input a hand shape with the middle finger pointing downward and all or part of the other fingers pointing upward. Therefore, the image processing module calculates the vector from the root 9 of the middle finger to the fingertip 12 as the first vector, and the mobile phone executes the following steps S508 to S513, obtaining an erroneous detection result of the finger pointing downward in the first image. The image processing module calculates the vector from the root of at least two fingers among the index finger, middle finger and ring finger to the fingertip as the first vector, and executes the following steps S508 to S513, which can ensure the accuracy of the hand direction detection in the first image.

[0130] It should be noted that the coordinates of the fingertips of the index finger, middle finger and ring finger proposed in the aforementioned embodiments can be replaced by the coordinates of the first joint points of the index finger, middle finger and ring finger, or by the coordinates of the second joint points of the index finger, middle finger and ring finger.

[0131] S508. The image processing module determines whether the angle between the first vector and the horizontal direction is within the range of (-30°, 30°), (-180°, -150°) or (150°, 180°).

[0132] The image processing module calculates the angle θ between the first vector and the horizontal x-axis. The image processing module compares the angle θ with three intervals: (-30°, 30°), (-180°, -150°) and (150°, 180°) to determine whether the angle θ is within (-30°, 30°).

[0133] (-180°,-150°) or (150°,180°) range.

[0134] The three intervals (-30°, 30°), (-180°, -150°) and (150°, 180°) are exemplary descriptions, and the endpoints of the three intervals can be adjusted to other values. The image processing module determines whether the angle θ between the first vector and the horizontal x-axis is within the range of the three intervals, and can infer whether the fingers of the human hand are facing left or right.

[0135] Therefore, the image processing module determines whether the angle between the first vector and the horizontal direction is at (-30°, 30°), (-180°, -150°) and (150°, 180°), which can be understood as: determining whether the angle θ between the first vector and the horizontal x-axis is in an angle range used to indicate that the fingers of a human hand are facing left or right.

[0136] The image processing module determines that the angle between the first vector and the horizontal direction is (-30°, 30°), (-180°, -150°) and (150°, 180°), and step S504 may be executed. The image processing module determines that the angle between the first vector and the horizontal direction is not (-30°, 30°), (-180°, -150°) and (150°, 180°), and the following steps S509 to S513 may be executed.

[0137] The image processing module determines that the angle between the first vector and the horizontal direction is at (-30°, 30°), (-180°, -150°) and (150°, 180°), and can infer whether the fingers of the human hand are facing to the left or to the right, and further infer that the user's intention is not to operate the mobile phone through air gestures. The mobile phone can avoid executing the following steps S510 to S513. When the user does not need to operate the mobile phone through air gestures, a small hand icon is displayed on the display screen to remind the user to input air gestures, which causes inconvenience to the user and saves power consumption.

[0138] S509: The image processing module determines whether the angle between the first vector and the horizontal direction is within the range of [30°, 150°].

[0139] The image processing module determines that the angle between the first vector and the horizontal direction is not in the range of (-30°, 30°), (-180°, -150°) or (150°, 180°), indicating that the angle between the first vector and the horizontal direction is [30°, 150°] or [-150°, -30°]; wherein, if the angle between the first vector and the horizontal direction is [30°, 150°], it can be inferred that the fingers of the human hand are facing upward, and if the angle between the first vector and the horizontal direction is [-150°, -30°], it can be inferred that the fingers of the human hand are facing downward.

[0140] In this embodiment, the image processing module needs to determine whether the angle between the first vector and the horizontal direction is [30°, 150°] or [-150°, -30°] to determine the direction of the fingers of the human hand. This step is introduced by taking the image processing module determining whether the angle between the first vector and the horizontal direction is [30°, 150°] as an example. In some embodiments, the image processing module may also determine whether the angle between the first vector and the horizontal direction is [-150°, -30°]; in other embodiments, the image processing module may also determine whether the angle between the first vector and the horizontal direction is [30°, 150°] and whether it is [-150°, -30°].

[0141] When the image processing module determines that the angle between the first vector and the horizontal direction is [30°, 150°], steps S510 and S511 may be executed. When the image processing module determines that the angle between the first vector and the horizontal direction is not [30°, 150°], steps S512 and S513 may be executed.

[0142] S510. The image processing module sends a second message to SystemUI, where the second message indicates that the hand in the first image is facing upward.

[0143] The image processing module determines that the angle between the first vector and the horizontal direction is in the range of [30°, 150°], then it can be inferred that the fingers of the human hand are facing upward, and the image processing module can send a second message to SystemUI in a transparent transmission manner.

[0144] S511. System UI displays a small hand icon with a finger pointing upward on the display.

[0145] When the SystemUI receives the second message and knows that the finger of the person's hand is pointing upward, it controls the display of a small hand icon with the finger pointing upward on the display screen of the mobile phone. For example, the small hand icon with the finger pointing upward can be as follows: Figure 1 The small hand icon 101 in (b).

[0146] S512. The image processing module sends a third message to SystemUI, where the third message indicates that the hand in the first image is facing downward.

[0147] The image processing module determines that the angle between the first vector and the horizontal direction is not in the range of [30°, 150°], then it can be inferred that the fingers of the human hand are facing downward, and the image processing module can send a third message to SystemUI in a transparent manner.

[0148] S513. System UI displays a small hand icon with a finger pointing downward on the display.

[0149] SystemUI receives the third message and knows that the finger of the human hand is pointing downward, then controls the display of a small hand icon with the finger pointing downward on the display screen of the mobile phone. For example, the small hand icon with the finger pointing upward can be as follows: Figure 2 The small hand icon 102 in (b).

[0150] In step S511 and step S513, SystemUI displays a small hand icon on the interface currently displayed on the display screen. In addition, the implementation method of SystemUI displaying a small hand icon with a finger pointing up and a small hand icon with a finger pointing down can refer to conventional technology and will not be described in detail here.

[0151] SystemUI displays the small hand icon on the display screen only when it receives the second message or the third message and determines that the hand in the first image is facing upward or downward. When it receives the first message and determines that the direction of the hand in the first image is unclear, it does not display the small hand on the display screen. This can avoid the situation where the mobile phone mistakenly detects that the hand in the first image is a palm or back of the hand through step S503, and then displays the small hand icon on the display screen to remind the user to input an air gesture, which causes inconvenience to the user and saves power consumption.

[0152] Furthermore, SystemUI displays a small hand icon with the finger pointing upwards on the display screen through step S511, which can remind the user that the starting gesture of the air gesture has been input, and the air gestures such as sliding down or grabbing can be input. SystemUI displays a small hand icon with the finger pointing downwards on the display screen through step S513, which can remind the user that the starting gesture of the air gesture has also been input, and the air gestures such as sliding up can be input. The user can also check whether the starting gesture he input is correct according to the small hand icon displayed on the display screen.

[0153] SystemUI displays a small hand icon with the finger pointing upward on the display screen through step S511, which means that the phone predicts that the user is going to input air gestures such as sliding down or grasping. The phone can call the software process that responds to air gestures such as sliding down or grasping during the display screen displaying the small hand icon with the finger pointing upward, thereby improving the response speed of the phone. Similarly, SystemUI displays a small hand icon with the finger pointing downward on the display screen through step S513, which means that the phone predicts that the user is going to input air gestures such as swiping up. The phone can call the software process that responds to air gestures such as swiping up during the display screen displaying the small hand icon with the finger pointing downward, thereby improving the response speed of the phone.

[0154] The above embodiment provides an implementation method of detecting the direction of the hand using the coordinates of the key points of the hand. There are other implementation methods of detecting the direction of the hand using the coordinates of the key points of the hand. The hand direction detection method provided in the following embodiment of the present application discloses another implementation method of detecting the direction of the hand using the coordinates of the key points of the hand.

[0155] like Figure 8 As shown, the hand direction detection method provided in this embodiment includes:

[0156] S801: A camera captures a first image.

[0157] S802: An image processing module obtains a first image captured by a camera.

[0158] S803: The image processing module detects whether the hand in the first image is a palm or back of hand type.

[0159] The image processing module detects that the hand in the first image is not a palm and back type, and the following step S804 can be executed. The image processing module detects that the hand in the first image is a palm and back type, and the following steps S805 to S814 can be executed.

[0160] S804: The image processing module sends a first message to SystemUI, where the first message indicates that the direction of the hand in the first image is unclear.

[0161] In some embodiments, the first message may also be used to indicate whether the hand in the first image is facing left or right.

[0162] S805: The image processing module crops the first image to obtain a hand area image.

[0163] S806: The image processing module detects and obtains the coordinates of the hand key points in the hand area image.

[0164] The specific implementation of steps S801 to S806 can refer to the contents of steps S501 to S506 in the aforementioned embodiment, which will not be repeated here.

[0165] S807: The image processing module calculates the direction vectors of multiple fingers using the coordinates of the key points of the hand.

[0166] In some embodiments, multiple fingers may refer to all fingers of a human hand. The image processing module calculates the direction vectors of multiple fingers using the coordinates of the key points, as shown in FIG9 , where the image processing module calculates the vector v1 from the base 1 of the thumb to the fingertip 4, the vector v2 from the base 5 of the index finger to the fingertip 8, the vector v3 from the base 9 of the middle finger to the fingertip 12, the vector v4 from the base 13 of the ring finger to the fingertip 16, and the vector v5 from the base 17 of the little finger to the fingertip 20.

[0167] In some other embodiments, the multiple fingers may refer to the index finger, middle finger and ring finger of a human hand. The image processing module uses the coordinates of the key points to calculate the direction vectors of the multiple fingers: the image processing module calculates the vector v2 from the root 5 of the index finger to the fingertip 8, the vector v3 from the root 9 of the middle finger to the fingertip 12, and the vector v4 from the root 13 of the ring finger to the fingertip 16.

[0168] In some other embodiments, the multiple fingers may refer to the middle finger and the ring finger of a human hand. The image processing module uses the coordinates of the key points to calculate the direction vectors of the multiple fingers: the image processing module calculates the vector v3 from the root 9 of the middle finger to the fingertip 12, and the vector v4 from the root 13 of the ring finger to the fingertip 16.

[0169] In some other embodiments, the multiple fingers may refer to the index finger and the middle finger of a human hand. The image processing module uses the coordinates of the key points to calculate the direction vectors of the multiple fingers: the image processing module calculates the vector v2 from the base 5 of the index finger to the fingertip 8, and the vector v3 from the base 9 of the middle finger to the fingertip 12.

[0170] In other embodiments, the multiple fingers only refer to the middle finger of a human hand. The image processing module uses the coordinates of the key points to calculate the direction vectors of the multiple fingers: The image processing module calculates the vector v3 from the root 9 of the middle finger to the fingertip 12.

[0171] In some scenarios, the direction of the middle finger of a person's hand is the same as the overall direction of the hand. The image processing module calculates the finger direction vector of the middle finger, and the overall direction of the hand can be determined using the finger direction vector of the middle finger. In some scenarios, the direction of the middle finger of a person's hand is different from the overall direction of the hand. The image processing module can calculate the finger direction vectors of at least two of all the fingers. Of course, it is best for the image processing module to calculate the direction vectors of the index finger, middle finger, and ring finger.

[0172] S808. The image processing module calculates an average value of angles between the direction vectors of the multiple fingers and the horizontal direction.

[0173] In some embodiments, Fig. 9 As shown, the image processing module calculates the vector v1 from the base 1 of the thumb to the fingertip 4 and the horizontal angle θ1, the vector v2 from the base 5 of the index finger to the fingertip 8 and the horizontal angle θ2, the vector v3 from the base 9 of the middle finger to the fingertip 12 and the horizontal angle θ3, the vector v4 from the base 13 of the ring finger to the fingertip 16 and the horizontal angle θ4, and the vector v5 from the base 17 of the little finger to the fingertip 20 and the horizontal angle θ5.

[0174] The image processing module calculates an average value of angles θ1 to θ5 between the vector of each finger and the horizontal direction as an average value θ of the angles between the direction vectors of the five fingers and the horizontal direction.

[0175] In other embodiments, the image processing module calculates the vector v2 pointing from the base 5 of the index finger to the fingertip 8 and the horizontal angle θ2, the vector v3 pointing from the base 9 of the middle finger to the fingertip 12 and the horizontal angle θ3, and the vector v4 pointing from the base 13 of the ring finger to the fingertip 16 and the horizontal angle θ4.

[0176] The image processing module calculates an average value of angles θ2 to θ4 between the vectors of the index finger, the middle finger and the ring finger and the horizontal direction as an average value θ of the angles between the direction vectors of the plurality of fingers and the horizontal direction.

[0177] In other embodiments, the image processing module calculates the angle θ2 between the vector v2 from the base 5 of the index finger to the fingertip 8 and the horizontal direction, and the angle θ3 between the vector v3 from the base 9 of the middle finger to the fingertip 12 and the horizontal direction.

[0178] The image processing module calculates the average of the angles θ2 and θ3 between the vectors of the index finger and the middle finger and the horizontal direction as the average θ of the angles between the direction vectors of the multiple fingers and the horizontal direction.

[0179] In other embodiments, the image processing module calculates the vector v3 from the base 9 of the middle finger to the fingertip 12 and the angle θ3 in the horizontal direction, and the vector v4 from the base 13 of the ring finger to the fingertip 16 and the angle θ4 in the horizontal direction.

[0180] The image processing module calculates an average value of angles θ2 to θ4 between the vectors of the middle finger and the ring finger and the horizontal direction as an average value θ of angles between the direction vectors of the plurality of fingers and the horizontal direction.

[0181] In some scenarios, the middle finger of a person's hand can also indicate the overall direction of the hand. The image processing module can also calculate the vector v3 pointing from the base 9 of the middle finger to the fingertip 12 and the angle θ3 in the horizontal direction, and use the vector v3 pointing from the base 9 of the middle finger to the fingertip 12 and the angle θ3 in the horizontal direction as the average value θ of the direction vectors of multiple fingers and the angles in the horizontal direction.

[0182] S809, the image processing module determines whether the average value of the angle is within the range of (-30°, 30°), (-180°, -150°) or (150°, 180°).

[0183] The average value θ of the angles between the direction vectors of multiple fingers and the horizontal direction can indicate the overall direction of the human hand, that is, the direction of the human hand. The image processing module determines whether the average value θ of the angles between the direction vectors of multiple fingers and the horizontal direction is (-30°, 30°), (-180°, -150°) or (150°, 180°), and can infer whether the human hand is facing right or left.

[0184] The image processing module determines that the average value θ of the angles between the direction vectors of multiple fingers and the horizontal direction is at (-30°, 30°), (-180°, -150°) or (150°, 180°), indicating that the hand is facing left or right; the image processing module determines that the average value θ of the angles between the direction vectors of multiple fingers and the horizontal direction is not at (-30°, 30°), (-180°, -150°) or (150°, 180°), indicating that the hand is facing up or down.

[0185] The image processing module determines that the average value of the angles between the direction vectors of the multiple fingers and the horizontal direction is (-30°, 30°), (-180°, -150°) or (150°, 180°), and step S804 may be executed. The image processing module determines that the average value of the angles between the direction vectors of the multiple fingers and the horizontal direction is not (-30°, 30°), (-180°, -150°) or (150°, 180°), and the following steps S810 to S814 may be executed.

[0186] The three intervals (-30°, 30°), (-180°, -150°) and (150°, 180°) are exemplary descriptions, and the endpoints of the three intervals can be adjusted to other values. The image processing module determines whether the average value of the angles between the direction vectors of multiple fingers and the horizontal direction is within the range of the three intervals, and can infer whether the fingers of the human hand are facing left or right.

[0187] Therefore, the image processing module determines whether the average value of the angles between the direction vectors of multiple fingers and the horizontal direction is between (-30°, 30°), (-180°, -150°) and (150°, 180°). This can be understood as: determining whether the average value of the angles between the direction vectors of multiple fingers and the horizontal direction is in an implementation method for indicating that the fingers of a human hand are facing left or right.

[0188] S810: The image processing module determines whether the average value of the angle is within the range of [30°, 150°].

[0189] The image processing module determines whether the average value θ of the angles between the direction vectors of the multiple fingers and the horizontal direction is in [30°, 150°]. The average value θ of the angles between the direction vectors of the multiple fingers and the horizontal direction can indicate the overall direction of the human hand, that is, the direction of the human hand. If the image processing module determines that the average value θ of the angles between the direction vectors of the multiple fingers and the horizontal direction is in [30°, 150°], it can be inferred that the fingers of the human hand in the first image are facing upward; if the image processing module determines that the average value θ of the angles between the direction vectors of the multiple fingers and the horizontal direction is not in [30°, 150°], it can be inferred that the fingers of the human hand in the first image are facing downward.

[0190] When the image processing module determines that the average value of the angles between the direction vectors of multiple fingers and the horizontal direction is [30°, 150°], steps S811 and S812 can be executed. When the image processing module determines that the average value of the angles between the direction vectors of multiple fingers and the horizontal direction is not [30°, 150°], steps S813 and S814 can be executed.

[0191] In some embodiments, the image processing module may also determine whether the angle between the first vector and the horizontal direction is in the range of [-150°, -30°]; in other embodiments, the image processing module may also determine whether the angle between the first vector and the horizontal direction is in the range of [30°, 150°] and whether it is in the range of [-150°, -30°].

[0192] S811. The image processing module sends a second message to SystemUI, where the second message indicates that the hand in the first image is facing upward.

[0193] S812, SystemUI displays a small hand icon with a finger pointing upward on the display.

[0194] S813. The image processing module sends a third message to SystemUI, where the third message indicates that the hand in the first image is facing downward.

[0195] S814, SystemUI displays a small hand icon with a finger pointing downward on the display.

[0196] The specific implementation of steps S811 to S814 can refer to the contents of steps S510 to S513 of the aforementioned embodiment, which will not be repeated here.

[0197] It can be seen from the above two embodiments that in order to realize the human-computer interaction of the user inputting the starting gesture to trigger the electronic device to prepare for the air gesture with the user, the hand direction detection method provided in the embodiment of the present application can be as follows: Fig.10 As shown, including:

[0198] S1001. The electronic device obtains an image captured by a camera.

[0199] After the screen of the electronic device is turned on, the camera is activated to capture images, and the electronic device obtains the images captured by the camera.

[0200] S1002: The electronic device detects whether the hand in the image is a palm or back of hand type.

[0201] If the electronic device detects that the hand in the image is a palm and back type, step S1003 is executed; if the electronic device detects that the hand in the image is not a palm and back type, steps S1004 to S1010 are executed.

[0202] The electronic device uses a hand detection algorithm to detect whether the hand in the image taken by the camera is a palm and back of the hand type. The implementation method of the electronic device detecting whether the hand in the image taken by the camera is a palm and back of the hand type can be referred to the content of step S503 in the aforementioned embodiment, which will not be repeated here.

[0203] S1003: The electronic device determines that the direction of the human hand in the image is unclear.

[0204] When the electronic device detects that the hand in the image is not a palm or back of the hand type, it can be inferred that the user's intention is not to interact with the electronic device through air gestures. Therefore, the electronic device may not further identify the finger direction of the hand in the image.

[0205] S1004: The electronic device crops the image to obtain an image of the hand area of ​​the human hand.

[0206] For the specific implementation of this step, please refer to the content of step S505 in the above embodiment, which will not be repeated here.

[0207] S1005: The electronic device detects and obtains the coordinates of the key points of the hand in the hand area image.

[0208] For the specific implementation of this step, please refer to the content of step S506 in the above embodiment, which will not be repeated here.

[0209] S1006. The electronic device calculates the angle between the finger direction vector of the human hand and the horizontal direction using the coordinates of the key points of the hand.

[0210] Step S507 in the aforementioned embodiment, and steps S807 and S808 in the aforementioned embodiment are two implementation methods of the electronic device performing this step. For the specific implementation method of this step, please refer to the content of the aforementioned embodiment, which will not be repeated here.

[0211] S1007, the electronic device determines whether the angle between the finger direction vector of the human hand and the horizontal direction is (-30°, 30°).

[0212] The range is (-180°, -150°) or (150°, 180°).

[0213] If the electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is (-30°, 30°), (-180°, -150°) or (150°, 180°), then step S1003 is executed; if the electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is (-30°, 30°), (-180°, -150°) or (150°, 180°), then step S1008 to step S1010 are executed.

[0214] The specific implementation of this step can refer to the contents of step S508 and step S809 in the above-mentioned embodiment, which will not be repeated here.

[0215] S1008. The electronic device determines whether the angle between the finger direction vector of the human hand and the horizontal direction is within the range of [30°, 150°].

[0216] When the electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is within the range of [30°, 150°], the electronic device executes step S1009; when the electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is not within the range of [30°, 150°], the electronic device executes step S1010.

[0217] The specific implementation of this step can refer to the contents of step S509 and step S810 in the above-mentioned embodiment, which will not be repeated here.

[0218] S1009. The electronic device displays a small hand icon with a finger pointing upward on the display screen.

[0219] The electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is within the range of [30°, 150°], and it can be inferred that the fingers of the human hand in the image captured by the camera are facing upward. Based on this, the electronic device displays a small hand icon with fingers facing upward on the display screen. For example, the small hand icon is as follows Figure 1 As shown in (b) in .

[0220] S1010. The electronic device displays a small hand icon with a finger pointing downward on the display screen.

[0221] The electronic device determines that the angle between the finger direction vector of the human hand and the horizontal direction is not within the range of [30°, 150°], and it can be inferred that the finger of the human hand in the image captured by the camera is facing downward. Based on this, the electronic device displays a small hand icon with the finger facing downward on the display screen. For example, the small hand icon is as follows Figure 2 As shown in (b) in .

[0222] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions, and when the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0223] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0224] Another embodiment of the present application further provides a computer program product including instructions. When the computer program product is run on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

Claims

1. A method for detecting hand direction, characterized in that: include: The display screen of the electronic device is in a bright screen state, and an image captured by a camera of the electronic device is obtained; Calculating the angle between the finger direction of the hand in the image and the horizontal direction; If it is determined that the angle between the finger direction of the hand in the image and the horizontal direction satisfies the condition that the finger is facing the first direction, a hand icon with the finger facing the first direction is displayed on the display screen; In the process of displaying a hand-shaped icon with a finger pointing in a first direction, calling a software process that responds to a first air gesture, and executing the software process that responds to the first air gesture after the user inputs the first air gesture, wherein the first air gesture includes sliding down or grasping; and the first direction is an upward direction; If it is determined that the angle between the finger direction of the hand in the image and the horizontal direction meets the condition that the finger is facing a second direction, a hand icon with the finger facing the second direction is displayed on the display screen, and the first direction and the second direction are different. During the process of displaying the hand icon with the finger facing the second direction on the display screen, a software process responding to the second air gesture is called, and after the user inputs the second air gesture, the software process responding to the second air gesture is executed, and the second air gesture includes swiping upwards; the second direction is a downward direction.

2. The method for detecting hand direction according to claim 1, characterized in that: After acquiring the image captured by the camera of the electronic device, the method further includes: detecting whether the hand in the image is a palm or a back of the hand.

3. The method for detecting hand direction according to claim 1 or 2, characterized in that: After acquiring the image captured by the camera of the electronic device, the method further includes: cropping a hand region image from the image; Detecting and obtaining the coordinates of the hand key points of the hand area image; The calculating the angle between the fingers of the hand in the image and the horizontal direction includes: using the coordinates of the key points of the hand to calculate the angle between the finger direction vector of the hand and the horizontal direction.

4. The method for detecting hand direction according to claim 3, characterized in that: The method of calculating the angle between the finger direction vector of the hand and the horizontal direction by using the coordinates of the key points of the hand includes: Calculate the vector from the finger base coordinates of the hand to the finger tip coordinates in the coordinates of the key points of the hand to obtain the finger direction vector of the hand in the image; The angle between the finger direction vector of the hand in the image and the horizontal direction is calculated.

5. The method for detecting hand direction according to claim 4, characterized in that: The step of calculating a vector from the finger base coordinates of the hand in the coordinates of the key points of the hand to the finger tip coordinates to obtain the finger direction vector of the hand in the image includes: Using the finger base coordinates of multiple fingers in the coordinates of the hand key points, calculate the average finger base coordinates of multiple fingers in the image, and using the finger tip coordinates of multiple fingers in the coordinates of the hand key points, calculate the average finger tip coordinates of multiple fingers in the image, the multiple fingers at least including the middle finger; The vector from the average coordinate of the bases of the multiple fingers to the average coordinate of the fingertips of the multiple fingers is calculated to obtain the finger direction vector of the hand in the image.

6. The method for detecting hand direction according to claim 3, characterized in that: The method of calculating the angle between the finger direction vector of the hand and the horizontal direction by using the coordinates of the key points of the hand includes: Calculate finger direction vectors of multiple fingers of the hand in the image using the finger base coordinates and fingertip coordinates of multiple fingers in the coordinates of the hand key points, wherein the finger direction vectors of the multiple fingers at least include the finger direction vector of the middle finger; Calculating angles between the finger direction vectors of the plurality of fingers and the horizontal direction to obtain a plurality of angles; The average of the multiple angles is calculated to obtain the angle between the finger direction of the hand in the image and the horizontal direction.

7. The method for detecting hand direction according to claim 1 or 2, characterized in that: The method of determining whether the angle between the finger direction of the hand in the image and the horizontal direction satisfies the condition that the finger is facing the first direction includes: The angle between the finger direction of the hand in the image and the horizontal direction is determined to be within the angle range corresponding to the finger pointing upward.

8. The method for detecting hand direction according to claim 1 or 2, characterized in that: If it is determined that the angle between the finger direction of the hand in the image and the horizontal direction is equal to the angle between the finger direction and the horizontal direction, before the condition that the finger is facing the first direction is satisfied, the method further includes: It is determined that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy the condition that the finger is facing the third direction.

9. The method for detecting hand direction according to claim 8, characterized in that: The determining that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy a condition that the finger is facing the third direction includes: It is determined that the angle between the finger direction of the hand in the image and the horizontal direction is not within the angle range corresponding to the finger facing left and right.

10. The method for detecting hand direction according to claim 9, characterized in that: If it is determined that the angle between the finger direction of the hand in the image and the horizontal direction meets the condition that the finger is facing the second direction, displaying a hand icon with the finger facing the second direction on the display screen includes: If it is determined that the angle between the finger direction of the hand in the image and the horizontal direction does not satisfy the condition that the finger is facing the first direction, a hand icon with the finger facing the second direction is displayed on the display screen.

11. An electronic device, characterized in that: include: one or more processors, memory, and display screens; The memory and the display screen are coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the hand direction detection method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the hand direction detection method as described in any one of claims 1 to 10.

Images