Gesture sensing method and device and storage medium

Abstract

The invention provides a gesture sensing method and device and a storage medium, relates to the technical field of terminals and the field of artificial intelligence, and further relates to the technical fields of intelligent sensing, intelligent control and the like. The method comprises the following steps: after the electronic equipment starts a gesture sensing function, a camera runs in a first mode; and when the electronic equipment detects that a gesture exists in the range of the camera, the camera is controlled to operate in a second mode. The electronic device obtains the image collected by the camera in the second mode, recognizes the specific type of the gesture in the image, and then executes the preset operation corresponding to the gesture type, such as screen capture, pause or video playing, next or previous short video playing and the like, so that the function of intelligently sensing the gesture is realized, and the use experience of a user is improved.

Description

Gesture perception method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 30, 2022, with application number 202211521522.7 and application name “Gesture Perception Method, Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a gesture perception method, device, and storage medium. Background Art

[0003] Traditional contact-based human-computer interaction involves users interacting with the computer through a touch keyboard, mouse, tablet, touch screen, or physical buttons. With the advancement of human-computer interaction technology for smart terminals, users can now control smart terminals through gestures in the air, a form of contactless human-computer interaction.

[0004] In related technologies, users can open the camera app and use air gestures to control the camera to take photos. This function requires the terminal to continuously perform gesture recognition, which places high demands on the camera device and processor. In addition, users also need air gestures in more scenarios where it is inconvenient to touch the terminal screen, such as when driving a vehicle or when their fingers are dirty.

[0005] Therefore, a gesture perception solution is urgently needed to meet more user needs while reducing device power consumption.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a gesture perception method, device, and storage medium, which can realize intelligent perception of air gestures and control of the device while reducing the power consumption of the device.

[0008] In a first aspect, an embodiment of the present application proposes a gesture perception method, which is applied to an electronic device, in which the camera of the electronic device operates in a first mode; the electronic device obtains a first image captured by the camera in the first mode, and detects whether there is a gesture in the first image; the electronic device detects the gesture in the first image, and controls the camera to switch from the first mode to the second mode; the electronic device obtains a second image captured by the camera in the second mode, and identifies the category of the gesture in the second image; the electronic device recognizes the gesture in the second image as a target gesture, and controls the execution of a preset operation corresponding to the target gesture.

[0009] As an example, the resolution of the first image is smaller than the resolution of the second image, and / or the frame rate of the first image is smaller than the frame rate of the second image.

[0010] In this solution, the camera continuously captures a first image in the first mode. If a gesture is detected in the first image, it switches to the second mode, continuously capturing a second image in the second mode and identifying the gesture category in the second image. This allows the execution of operations corresponding to the gesture category, enabling intelligent sensing of air gestures and device control, providing convenience when it is inconvenient for the user to touch the device. Furthermore, by acquiring the first image captured by the camera in the first mode, detecting the presence of a gesture, and then switching the camera mode to acquire the second image captured by the camera in the second mode and identifying the specific category of the gesture, device power consumption can be reduced to a certain extent.

[0011] In an optional embodiment of the first aspect, before the camera of the electronic device operates in the first mode, the method further includes: the electronic device responding to a first operation of enabling a gesture sensing function.

[0012] In this solution, the conditions for turning on the camera of the device are defined. If the conditions are met, the device is triggered to execute the gesture perception solution.

[0013] In an optional embodiment of the first aspect, before the camera of the electronic device operates in the first mode, the method also includes: detecting that the state of the electronic device satisfies a first condition; the first condition includes at least one of the following: the screen state of the electronic device is a bright screen state; the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0014] This solution further limits the conditions under which the device's camera must be activated. Besides enabling gesture sensing, a first condition is added to prevent the camera from continuously capturing the first image when not necessary, thus reducing device power consumption. "Unnecessary" can be understood as any scenario where the camera cannot capture gestures.

[0015] In an optional embodiment of the first aspect, the electronic device is a foldable device, the foldable device includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera; the camera of the electronic device operates in a first mode, including: detecting that the outer screen of the electronic device is in a bright screen state and the electronic device is in a folded state, and controlling the second camera to operate in the first mode; or detecting that the inner screen of the electronic device is in a bright screen state and the electronic device is in an unfolded state, and controlling the first camera to operate in the first mode.

[0016] This solution can be applied to foldable devices. If the device is in the folded or unfolded state and the corresponding screen (inner screen or outer screen) is in the bright screen state, the camera on the bright screen can be controlled to turn on to detect whether there is a gesture within the camera range and realize gesture perception.

[0017] In an optional embodiment of the first aspect, before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, the method further includes: detecting that the state of the electronic device satisfies a second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the inner screen or outer screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0018] In this solution, the conditions for turning on the camera of a foldable device are further limited. In addition to the corresponding screen lighting up when the user turns on the gesture sensing function and the electronic device screen is in a folded or unfolded state, a second condition is added to avoid the camera from continuously collecting the first image when it is not necessary, thereby reducing the power consumption of the device.

[0019] In an optional embodiment of the first aspect, the method further includes: when the second camera is operating in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera; when the first camera is operating in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to turn off and controls the second camera to operate in the first mode.

[0020] In this solution, when other conditions remain unchanged, if the user changes the physical state of the device screen, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the first image can be continuously captured by switching the camera so that the device screen can still realize the functions of intelligently sensing air gestures and controlling the device in the new physical state.

[0021] In an optional embodiment of the first aspect, the target gesture includes a first gesture and a second gesture, the first gesture is the starting gesture of the target gesture, and the second gesture is the ending gesture of the target gesture; the electronic device recognizes that the gesture in the second image is the first gesture, and the electronic device controls the execution of a preset operation corresponding to the target gesture, including: the electronic device recognizes that the gesture in the second image is the first gesture, and controls the camera to switch from the second mode to the third mode; the electronic device obtains a third image captured by the camera in the third mode, and identifies the category of the gesture in the third image; the electronic device recognizes that the gesture in the third image is the second gesture, and the electronic device controls the execution of a preset operation corresponding to the target gesture.

[0022] As an example, the resolution of the second image is less than or equal to the resolution of the third image, and / or the frame rate of the second image is less than the frame rate of the third image.

[0023] In this solution, the target gesture includes a start gesture and an end gesture. If the start gesture and the end gesture are recognized in sequence, the preset operation corresponding to the target gesture can be controlled to be executed, such as taking a screenshot, controlling audio and video playback, etc.

[0024] In an optional embodiment of the first aspect, the camera of the electronic device operates in a first mode, including: a perception module of the electronic device sends a first indication to a second processing module of the electronic device, the first indication being used to instruct the second processing module to detect whether there is a gesture within the range of the camera; the second processing module sends a first shooting instruction to the camera; and the camera operates in the first mode in response to the first shooting instruction.

[0025] In an optional embodiment of the first aspect, the electronic device obtains a first image captured by the camera in the first mode, and detects whether there is a gesture in the first image, including: a second processing module of the electronic device obtains the first image captured by the camera in the first mode, and detects whether there is a gesture in the first image.

[0026] In an optional embodiment of the first aspect, the electronic device detects a gesture in a first image and controls the camera to switch from the first mode to the second mode, including: the second processing module of the electronic device detects a gesture in the first image, the second processing module sends a first message to the perception module of the electronic device, and the first message is used to notify the perception module that there is a gesture within the camera range; the perception module sends a second indication to the first processing module of the electronic device, and the second indication is used to instruct the first processing module to identify the category of the gesture in the second image; the first processing module sends a second shooting instruction to the camera in response to the second indication, and the second shooting instruction is used to instruct the camera to operate in the second mode.

[0027] In an optional embodiment of the first aspect, the electronic device obtains a second image captured by the camera in the second mode and identifies the category of the gesture in the second image, including: the first processing module of the electronic device obtains the second image captured by the camera in the second mode, and identifies the category of the gesture in the second image.

[0028] In an optional embodiment of the first aspect, the electronic device recognizes that the gesture in the second image is a target gesture, and controls the execution of a preset operation corresponding to the target gesture, including: the first processing module of the electronic device recognizes that the gesture in the second image is a target gesture, the first processing module sends a second message to the perception module of the electronic device, and the second message is used to indicate that the target gesture is recognized in the second image; the perception module sends a third indication to the target application of the electronic device, and the third indication is used to indicate the target gesture; the target application controls the execution of the preset operation corresponding to the target gesture.

[0029] In an optional embodiment of the first aspect, the second processing module of the electronic device detects a state of the electronic device.

[0030] In an optional embodiment of the first aspect, the electronic device controls the camera to switch from the second mode to the third mode, including: the first processing module of the electronic device controls the camera to switch from the second mode to the third mode; the electronic device obtains a third image captured by the camera in the third mode, and identifies the category of the gesture in the third image, including: the first processing module obtains the third image captured by the camera in the third mode, and identifies the category of the gesture in the third image; if the gesture in the third image is the second gesture, the electronic device controls the execution of a preset operation corresponding to the target gesture, including: if the gesture in the third image is the second gesture, the first processing module sends a third message to the perception module of the electronic device, the third message is used to indicate that the target gesture is recognized; the perception module sends a fourth indication to the target application of the electronic device, the fourth indication is used to indicate the target gesture; the target application controls the execution of a preset operation corresponding to the target gesture.

[0031] The above-mentioned optional embodiments illustrate the interaction process between the underlying modules of electronic devices to enable the device to perceive air gestures and control the execution of corresponding operations, thereby improving the user experience.

[0032] In a second aspect, an embodiment of the present application provides an electronic device, which includes: a camera, a memory, and a processor; the camera is used to capture images with different frame rates and / or resolutions, and the processor is used to call a computer program in the memory to execute the gesture perception method as described in any one of the first aspects.

[0033] In an optional embodiment of the second aspect, the processor includes a first processing module and a second processing module, the power consumption of the first processing module is higher than the power consumption of the second processing module; the second processing module is used to detect whether there is a gesture within the camera range; and the first processing module is used to identify the category of the gesture within the camera range.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, which includes a unit, module or circuit for executing the method as described in any one of the first aspects.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method as described in any one of the first aspects.

[0036] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute any method as described in the first aspect.

[0037] In a sixth aspect, a computer program product comprises a computer program, which, when executed, enables a computer to execute the method as described in any one of the first aspects.

[0038] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a schematic diagram of a gesture perception method provided in an embodiment of the present application;

[0040] Figures 2 to 4 are schematic diagrams of target gestures provided in embodiments of the present application;

[0041] FIG5 is a schematic diagram of an interface provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of a gesture perception method provided in an embodiment of the present application;

[0043] FIG7 is a schematic diagram of a dynamic frame adjustment provided by an embodiment of the present application;

[0044] FIG8 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0045] FIG9 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0046] FIG10 is a schematic diagram of the structure of a SoC provided in an embodiment of the present application;

[0047] FIG11 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0048] FIG12 is a flow chart of a gesture perception method provided in an embodiment of the present application;

[0049] FIG13 is a schematic diagram of a hand node provided in an embodiment of the present application;

[0050] FIG14 is a flow chart of a gesture perception method provided in an embodiment of the present application;

[0051] FIG15 is a flow chart of a gesture perception method provided in an embodiment of the present application;

[0052] FIG16 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0053] FIG17 is a schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0054] FIG18a is a schematic diagram showing changes in a mobile phone interface provided by an embodiment of the present application;

[0055] FIG18b is a schematic diagram of a mobile phone interface change according to an embodiment of the present application;

[0056] FIG18c is a schematic diagram showing changes in a mobile phone interface provided by an embodiment of the present application;

[0057] FIG19 is a schematic structural diagram of a foldable screen mobile phone provided in an embodiment of the present application;

[0058] FIG20 is a flow chart of a gesture perception method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first image and the second image are merely used to distinguish images with different frame rates and / or resolutions, and do not limit their order. For another example, the first indication and the second indication are merely used to distinguish different indications. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity or execution order, and words such as "first" and "second" do not necessarily mean that they are different.

[0060] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0061] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items (kind / individual)" or similar expressions refers to any combination of these items, including any combination of single items (kind / individual) or plural items (kind / individual). For example, at least one of a, b or c (kind / individual) can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0062] Some of the terms used in this application are explained below to facilitate understanding by those skilled in the art.

[0063] Frame rate refers to the number of images a camera captures or transmits in one second, typically expressed in fps (frames per second). In an embodiment of the present application, the camera captures images at a first frame rate in a first mode and at a second frame rate in a second mode, where the first frame rate is less than the second frame rate. In some embodiments, the camera captures images at a third frame rate in a third mode, where the third frame rate is greater than the second frame rate.

[0064] Resolution, also known as image resolution, refers to the amount of information stored in an image, which is the number of pixels per inch in the image. The units of resolution include: dpi (dots per inch), ppi (pixels per inch), etc.

[0065] A lightweight neural network is a lighter model that has performance comparable to heavier models, thereby achieving a hardware-friendly neural network. The weight here usually refers to the scale or number of parameters of the model. Commonly used lightweight neural network technologies include: distillation, pruning, quantization, weight sharing, low-rank decomposition, lightweight attention module, dynamic network architecture / training method, lighter network architecture design, etc., which are not limited in the embodiments of this application.

[0066] Background subtraction is a widely used method in motion target detection technology. Its basic concept is similar to the inter-frame difference method, which uses the difference operation of different images to extract the target area. Unlike the inter-frame difference method, background subtraction does not subtract the current frame image from the adjacent frame image, but subtracts the current frame image from a continuously updated background model to extract the moving target in the difference image. In an embodiment of the present application, the moving target can be the user's hand. Based on the background subtraction operation, the changes in the user's gestures in consecutive image frames are detected to determine the gesture category.

[0067] Compared to traditional contact-based human-computer interaction, handheld gestures allow users to operate with their bare hands, providing a contactless human-computer interaction method that eliminates the inconvenience of gesture interaction. Handheld gestures include pointing, waving, clenching a fist, and rotating the palm. Users can naturally express their interaction intentions through finger, wrist, and arm movements, offering a wide interaction space, greater flexibility, and a better interactive experience.

[0068] In related technologies, in some scenarios, after the user opens the camera application, he can use air gestures, such as spreading his palm, to control the camera to take photos or record videos. However, as user needs continue to increase, there are more and more scenarios for using air gestures: in one possible scenario, the user is wearing gloves or his fingers are dirty and he does not want to touch the phone screen, but he has the need to control the phone, for example, when the user is eating crayfish, he needs to watch short videos; in another possible scenario, when the user is driving a vehicle, he is a little far away from the phone screen and needs to call other applications, for example, when the user drives to the vicinity of the office and needs to punch in for work. In order to meet the user's needs for air operations in different scenarios, the device needs to continuously perform gesture recognition, which places higher requirements on the device's camera components and processors.

[0069] For this reason, there is an urgent need for a gesture perception solution that can reduce the power consumption of the device while meeting more user needs. The embodiment of the present application proposes a gesture perception method, device and storage medium. The electronic device provided by the embodiment of the present application can obtain images with different frame rates and / or resolutions by adjusting the working mode of the camera to detect whether there is a gesture in the image step by step, identify whether the gesture is a target gesture, etc. The frame rate and / or resolution of the image captured by the camera in different working modes are different. In the above scheme, by setting multiple working modes for the camera, the working mode of the camera is automatically switched in different detection stages to meet the image quality requirements of different detection stages, which can reduce the power consumption of the device. In addition, by improving the processing module of the device, different processing modules can perform different image detection tasks, such as detecting whether there is a gesture in the image, identifying gestures, etc., to further reduce the power consumption of the device.

[0070] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0071] It should be noted that the following description uses a mobile phone as an electronic device, and this example does not constitute a limitation on the embodiments of the present application.

[0072] Figure 1 is a schematic diagram of a gesture perception method provided by an embodiment of the present application. As shown in Figure 1, if the conditions for turning on the camera are met, for example, the user has turned on the gesture perception function, the camera of the mobile phone is triggered to work in the first mode, and the camera in the first mode continuously captures the first image of the first resolution at a first frame rate. At the same time, the mobile phone can automatically identify whether there is a gesture in the first image (that is, whether there is a hand in the first image). If a gesture is recognized in the first image, the camera is triggered to work in the second mode, and the camera in the second mode continuously captures the second image of the second resolution at a second frame rate, identifies the gesture in the second image, and determines whether the gesture is a target gesture. If the gesture is recognized as a target gesture, the mobile phone can run the preset operation corresponding to the target gesture, such as swiping up or down the interface, adjusting the screen brightness, volume or vibration intensity, calling applications, etc. The first frame rate of the first image is less than the second frame rate of the second image, and the first resolution of the first image is less than the second resolution of the second image.

[0073] In the embodiments of the present application, unless otherwise specified, the camera refers to the front camera of the mobile phone.

[0074] The gesture sensing method illustrated in this embodiment allows the camera to continuously capture images with low power consumption to detect whether a hand is within camera range. If a hand is detected, the camera switches its shooting mode to capture images at a higher resolution and / or frame rate to identify the gesture. If the gesture is the target gesture, the preset operation corresponding to the gesture is executed, thereby implementing the device's gesture sensing function and meeting the user's needs for remote operation. By adjusting the camera operating mode, device power consumption can be reduced.

[0075] In an embodiment of the present application, the target gesture includes at least one gesture action. Accordingly, the mobile phone's recognition of the target gesture includes recognizing at least one gesture action. For example, Figures 2 to 4 are schematic diagrams of target gestures provided in an embodiment of the present application. The target gestures and their application scenarios are described in detail below with reference to the accompanying drawings.

[0076] In a possible implementation, the target gesture is the first gesture, that is, the number of target gestures is one.

[0077] For example, the gesture shown in Figure 2 (a) is extending the index finger. For example, if the current interface of the phone is the short video interface, the phone recognizes this gesture and then likes or collects the short video. For another example, if the current interface of the phone is the selfie interface, the phone recognizes this gesture and starts a 3-second countdown to shoot.

[0078] Exemplarily, the gesture shown in (b) in Figure 2 is five fingers spread out. For example, if the current screen state of the mobile phone is the bright screen state and the interface is any interface, after the mobile phone recognizes the gesture, it will capture the current interface. For another example, if the current interface of the mobile phone is the selfie interface, after the mobile phone recognizes that the user makes a five-fingered spread action, it can control the front camera of the mobile phone to capture an image, or start / pause video recording. For another example, if the current screen state of the mobile phone is black, after the mobile phone recognizes the gesture, it will light up the screen to check the time / message, or open the office application and perform the work clocking task. For another example, if the current interface is the navigation interface, after the mobile phone recognizes the gesture, it can automatically broadcast the real-time traffic conditions.

[0079] In a possible implementation, the target gesture includes a first gesture and a second gesture, the first gesture is a starting gesture of the gesture change, and the second gesture is a terminating gesture of the gesture change.

[0080] For example, the gesture shown in Figure 3 (a) is a grasping gesture, where the gesture changes from five fingers spread out to five fingers clenched into a fist, that is, the starting gesture is the palm facing the front camera of the phone, and the ending gesture is the fist. For example, if the phone's screen is in the on state and the interface is any interface, the phone will capture the current interface after recognizing the gesture.

[0081] Exemplarily, the gesture shown in (b) in FIG3 is a pressing action, which is a movement of approaching the mobile phone. The starting gesture and the ending gesture are both with the palm facing the front camera of the mobile phone, and the ending gesture is closer to the screen. The preset operation corresponding to the gesture can be pause or continue playing. For example, if the current interface of the mobile phone is a video / music playback interface and is currently in the playback state, the mobile phone will pause the video / music after recognizing the gesture. As shown in FIG18a, the user can pause the current video screen by using an air pressing gesture within the range of the front camera of the mobile phone; if the current interface of the mobile phone is a video / music playback interface and is currently in the pause state, the mobile phone will continue to play the video / music after recognizing the gesture.

[0082] For example, the gesture shown in Figure 3(c) is a two-finger (thumb and index finger) zoom-in gesture, where the distance between the thumb and index finger in the initial gesture is smaller than the distance between the thumb and index finger in the final gesture. For example, if a picture is currently displayed on the phone interface, the phone will zoom in on the picture after recognizing the two-finger zoom-in gesture. As shown in Figure 18b, a user can use the two-finger zoom-in gesture in the air within the range of the phone's front camera to zoom in on the picture on the display interface.

[0083] For example, the gesture shown in Figure 3 (d) is a pinch-out gesture using two fingers (thumb and index finger), where the distance between the thumb and index finger in the initial gesture is greater than the distance between the thumb and index finger in the final gesture. For example, if a picture is currently displayed on the mobile phone interface, the phone will zoom out after recognizing the pinch-out gesture.

[0084] In a possible implementation, the target gesture includes a first gesture, a second gesture, and a third gesture, the first gesture being a starting gesture of a gesture change, the second gesture being an intermediate gesture of the gesture change, and the third gesture being an ending gesture of the gesture change.

[0085] Exemplarily, the gesture shown in (a) of FIG4 is an upward swipe gesture, the starting gesture of which is the back of the hand facing the front camera of the mobile phone, the middle gesture is the flattening of the palm, and the ending gesture is the palm facing the front camera of the mobile phone, that is, the gesture goes from bending downward to flattening the palm and then flipping the palm to face the camera. For example, if the mobile phone currently displays a data stream interface, the mobile phone will scroll down a fixed length after recognizing the gesture. For another example, if the mobile phone currently displays a page, the mobile phone will turn down a page after recognizing the gesture. For another example, if the mobile phone is currently playing a video, the mobile phone will turn up the volume or brightness after recognizing the gesture. For another example, if the current interface of the mobile phone is a short video interface, the mobile phone will play the next short video after recognizing the gesture. As shown in FIG18c, the user uses an air swipe gesture within the range of the front camera of the mobile phone to control the playback of the next short video.

[0086] Exemplarily, the gesture shown in (b) of FIG4 is a sliding gesture, the starting gesture of which is the palm facing the front camera of the mobile phone, the middle gesture is the palm flattened, and the ending gesture is the back of the hand facing the front camera of the mobile phone, that is, the gesture goes from the palm facing the front camera of the mobile phone to the palm clasped perpendicular to the mobile phone screen, and then to the palm bent with the back of the hand facing the front camera. For example, if the mobile phone currently displays a data stream interface, the mobile phone will scroll up a fixed length after recognizing the gesture. For another example, if the mobile phone currently displays a page, the mobile phone will turn up a page after recognizing the gesture. For another example, if the mobile phone is currently playing a video, the mobile phone will lower the volume or brightness after recognizing the gesture. For another example, if the current interface of the mobile phone is a short video interface, the mobile phone will play the previous short video after recognizing the gesture.

[0087] Based on the above embodiments, it can be seen that in different application scenarios, users can achieve diversified control of the mobile phone through preset air gestures. Application scenarios include, for example, the mobile phone screen is black, the user is watching a video or short video, the user is browsing the web or photo album, the user is driving and navigating, and the user is taking pictures or recording videos.

[0088] It should be noted that the embodiments of the present application do not limit the application scenarios of the gesture perception method. In addition to the application scenarios described in the above embodiments, it can also be applied to any other scenarios where a mobile phone can be controlled through air gesture operations.

[0089] In the above embodiment, the condition for turning on the camera includes that the gesture sensing function is turned on. In some embodiments, in response to the first operation of turning on the gesture sensing function, the camera of the mobile phone operates in a first mode (background operation, for example, when a third-party application is currently opened on the mobile phone and the camera is running in the background) to capture the above-mentioned first image.

[0090] Among them, the first operation can be a click operation of the user on the system setting interface, and the first operation can also be a voice operation, which is not limited in this embodiment of the present application.

[0091] For example, Figure 5 is a schematic diagram of an interface provided by an embodiment of the present application. As shown in Figure 5 (a) and (b), the user can choose to turn on or off the gesture sensing function in the settings interface of the system application. If the user chooses to turn on the gesture sensing function, the mobile phone camera operates in low-power shooting mode and continuously captures low-resolution images to detect whether there are hands in the image.

[0092] In some embodiments, the user can also choose to perceive one or more gestures in the settings interface. Each gesture has a preset corresponding operation, for example, gesture A is used to slide down the interface / flip down the page, gesture B is used to slide up the interface / flip up the page, gesture C is used for screenshots, and gesture D is used to pause / resume playback.

[0093] For example, as shown in (c) and (d) of Figure 5, the user can choose to turn on the perception of gesture A. If the phone recognizes gesture A, the operation corresponding to gesture A is executed: sliding down the interface / flipping down the page; the user can also choose to turn on the perception of gesture E. If the phone recognizes gesture E, the operation corresponding to gesture E is executed: calling the preset application.

[0094] In some embodiments, the user can customize the application corresponding to gesture E. As shown in Figure 5 (d), (e), and (f), the user selects gesture E on the setting interface of gesture E to call application A. For example, gesture E is five fingers spread out, and application A is a clock-in applet for an office application. When the user drives near the office, they can use the air gesture E to quickly launch the clock-in applet for the office application and complete the work clock-in, realizing gesture perception and improving the user's mobile experience.

[0095] In some embodiments, the conditions for turning on the camera also include a first condition, and the first condition includes at least one of the following: the screen status of the mobile phone is on; the mobile phone is unlocked; the time difference between the light signal emitted by the proximity light sensor of the mobile phone and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the mobile phone is greater than a third threshold; the screen of the mobile phone is facing a preset direction; the mobile phone runs a preset application.

[0096] In this embodiment, if the gesture sensing function is enabled on the mobile phone and the first condition is met, the mobile phone's camera is triggered to continuously capture the first image. By adding the first condition, the mobile phone camera is prevented from continuously capturing the first image when not necessary, further reducing device power consumption.

[0097] Based on the aforementioned embodiments, various possible implementation methods of triggering the camera to capture the first image are described in detail below.

[0098] In a possible implementation, if the gesture sensing function of the mobile phone is turned on, the mobile phone camera is triggered to continuously capture the first image.

[0099] In one possible implementation, if the gesture sensing function is enabled on the mobile phone, the screen status of the mobile phone is detected. If the screen status of the mobile phone is on, the mobile phone camera is triggered to continuously capture the first image. The interfaces displayed in the on-screen state of the mobile phone include, for example, the lock screen interface, the home screen, and the third-party application interface.

[0100] In a possible implementation, if the gesture sensing function of the mobile phone is turned on, it is detected whether the mobile phone is unlocked. If the mobile phone is unlocked, the mobile phone camera is triggered to continuously capture the first image.

[0101] In one possible implementation, if the gesture sensing function of the mobile phone is turned on, it is detected whether the light signal emitted by the proximity light sensor of the mobile phone is blocked. If it is determined that the light signal emitted by the proximity light sensor is not blocked, the mobile phone camera is triggered to continue capturing the first image.

[0102] As an example, if the time difference between the light signal emitted by the proximity light sensor of the mobile phone and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the proximity light sensor does not receive the reflected signal, it can be determined that the light signal emitted by the proximity light sensor is not blocked.

[0103] It is understandable that if the user makes or receives a call through the handset, or the phone screen is upside down, or the phone is in a handbag or pocket, the light signal emitted by the phone's proximity light sensor will be blocked, and the image captured by the front camera will not be able to detect gestures. At this time, the camera can stop continuous image capture to reduce device power consumption.

[0104] In one possible implementation, if the gesture sensing function is enabled on the mobile phone, the detection data of the mobile phone's ambient light sensor is detected to determine whether it is greater than a third threshold. If the detection data of the ambient light sensor is determined to be greater than the third threshold, the mobile phone camera is triggered to capture the first image. The detection data primarily refers to the ambient light brightness. It should be understood that if the detection data of the mobile phone's ambient light sensor is greater than the third threshold, it indicates that the electronic device is not in a dark environment, such as if the phone is in a pocket or it is currently night time.

[0105] In one possible implementation, if the gesture sensing function is enabled on the mobile phone, the screen orientation of the mobile phone is detected. If the screen is facing a preset direction, the mobile phone camera is triggered to continuously capture the first image. In this implementation, the preset direction includes the direction in which the user is holding the mobile phone to browse the interface content, the direction in which the mobile phone is placed on a table to browse the interface content, etc. This direction can be determined by detecting the posture data of the mobile phone, where the posture data includes the pitch angle, yaw angle, and roll angle.

[0106] In one possible implementation, if the mobile phone has turned on the gesture perception function, it detects whether the mobile phone is running a preset application. If it is determined that the mobile phone is running the preset application, the mobile phone camera is triggered to continuously capture the first image. For example, the preset application is a short video application. If the mobile phone detects that the user is watching a short video, the mobile phone camera is triggered to continuously capture the first image to detect whether the user is using the preset air gesture to meet the user's need to browse short videos through the air. For another example, the preset application is a navigation application. If the mobile phone detects that the user is using the driving navigation of the navigation application, the mobile phone camera is triggered to continuously capture the first image to detect whether the user is using the preset air gesture to meet the user's need to listen to traffic reports through the air. The embodiment of the present application does not limit the preset application. In addition, it should be noted that the mobile phone running the preset application includes displaying the interface of the preset application on the current mobile phone screen, or the preset application is running in the background.

[0107] In one possible implementation, if the gesture perception function of the mobile phone has been turned on, and at least two of the following conditions are met: the screen of the mobile phone is in the bright screen state, the mobile phone is unlocked, the light signal emitted by the proximity light sensor of the mobile phone is not blocked, and the mobile phone runs a preset application, then the mobile phone camera is triggered to continuously capture the first image.

[0108] The above embodiment illustrates a gesture perception method. If the conditions for turning on the camera are met, the mobile phone camera is triggered to scan permanently in a first mode and continuously capture a first image with a lower resolution. The conditions for turning on the camera include at least that the gesture perception function is turned on. If a gesture is detected in the first image, the camera is triggered to capture a second image with a higher resolution in a second mode. If the gesture in the second image is recognized as a target gesture, the preset operation corresponding to the target gesture is executed. The above method automatically switches the camera mode in different detection stages by setting two modes for the camera to meet the image quality requirements in different detection stages and reduce device power consumption.

[0109] In another embodiment, if the gesture sensing function is enabled on the mobile phone and the mobile phone is detected to be in motion, the mobile phone camera may be triggered to capture the aforementioned first image. Conversely, if the gesture sensing function is enabled on the mobile phone and the mobile phone is detected to be stationary, the camera may be controlled to be turned off to reduce device power consumption. The mobile phone in motion includes, for example, the following scenarios: a user using mobile phone navigation while driving a vehicle, or a user using mobile phone navigation while riding a bicycle.

[0110] In another embodiment, if the gesture sensing function is enabled on the mobile phone and the mobile phone is detected to be stationary, the gesture camera can be triggered to capture the first image to sense the user's possible airborne control needs at any time. The stationary state of the mobile phone includes, for example, the mobile phone being placed on a table.

[0111] In some embodiments, if the mobile phone is a folding screen mobile phone, the folding screen mobile phone includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera. For example, the first camera is camera 3 in Figure 19, and the second camera is camera 1 in Figure 19.

[0112] As an example, if it is detected that the external screen of the mobile phone is in the bright screen state and the mobile phone is in the folded state, the second camera is controlled to operate in the first mode. As an example, if it is detected that the external screen of the mobile phone is in the bright screen state and the mobile phone is in the unfolded state, the second camera is controlled to operate in the first mode. Based on these two examples, in some embodiments, before controlling the second camera to operate in the first mode, it also includes: detecting that the state of the mobile phone meets at least one of the following: the mobile phone is unlocked; the time difference between the light signal emitted by the proximity light sensor of the mobile phone (on the external screen) and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the mobile phone is greater than the third threshold; the external screen of the mobile phone is facing a preset direction; the mobile phone runs a preset application.

[0113] As an example, if it is detected that the inner screen of the mobile phone is in the bright screen state and the mobile phone is in the unfolded state, the first camera is controlled to operate in the first mode. Before controlling the first camera to operate in the first mode, it also includes: detecting that the state of the mobile phone meets at least one of the following: the mobile phone is unlocked; the time difference between the light signal emitted by the proximity light sensor of the mobile phone (on the inner screen) and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the mobile phone is greater than the third threshold; the inner screen of the mobile phone is facing a preset direction; the mobile phone runs a preset application.

[0114] Based on the above examples, before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, it also includes: detecting that the state of the electronic device meets the second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the inner screen or outer screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0115] As an example, when the second camera operates in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera.

[0116] As an example, when the first camera is operating in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to turn off and controls the second camera to operate in the first mode; or, the electronic device controls the first camera to turn off.

[0117] The following is a detailed description of the gesture perception method for a foldable screen mobile phone in conjunction with Figure 19.

[0118] For example, FIG19 is a schematic structural diagram of a folding screen mobile phone provided in an embodiment of the present application. As shown in FIG19 , the screen of the folding screen mobile phone includes a first screen, a second screen and a third screen. The first screen is the outer screen of the folding screen mobile phone, and the second screen and the third screen are the inner screens of the folding screen mobile phone. The folding screen includes the second screen and the third screen. The folding screen is folded according to the folding edge shown in FIG19 (4) to form the second screen and the third screen. The virtual axis where the folding screen is located is a common axis. Among them, the inner screen refers to the screen located inside when the folding screen is in a folded state, and the outer screen refers to the screen located outside when the folding screen is in a closed state. The angle β between the second screen and the third screen is the hinge angle of the folding screen mobile phone. The physical state of the folding screen can be determined by determining the hinge angle. The physical state includes the folding state shown in FIG19 (3), the unfolded state shown in FIG19 (4), or the bracket state shown in FIG19 (2). The folding screen mobile phone shown in FIG19 includes three groups of cameras, which are respectively recorded as camera 1, camera 2 and camera 3. As shown in (1) of Figure 19, camera 1 is set in the middle of the upper part of the first screen, and camera 2 is set on the back panel. As shown in (4) of Figure 19, camera 3 is set in the middle of the upper part of the third screen. When the folding screen mobile phone is in the folded state, camera 1 can be regarded as the front camera, and camera 2 can be regarded as the rear camera, as shown in (3) of Figure 19. When the folding screen mobile phone is in the unfolded state, camera 3 can be regarded as the front camera, and cameras 1 and 2 can be regarded as the rear cameras.

[0119] Taking the foldable screen mobile phone shown in Figure 19 as an example, the following examples illustrate under what circumstances the camera of the foldable screen mobile phone is turned on and which cameras are turned on.

[0120] In one possible implementation, if the gesture sensing function of the foldable screen mobile phone is turned on, when it is detected that the mobile phone is in a folded state and the external screen (such as the first screen in Figure 19) is in a bright screen state, the camera 1 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a gesture within the camera range.

[0121] In one possible implementation, if the gesture sensing function of the foldable screen mobile phone is turned on, when it is detected that the mobile phone is in a folded state and the external screen is in a bright screen state, and at least one of the above-mentioned second conditions is met, the camera 1 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a gesture within the camera range.

[0122] In one possible implementation, if the gesture sensing function of the foldable screen mobile phone is turned on, when it is detected that the mobile phone is in the unfolded state and the inner screen (such as the second screen and the third screen in Figure 19) is in the bright screen state, the camera 3 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a gesture within the camera range.

[0123] In one possible implementation, if the gesture sensing function of the foldable screen mobile phone is turned on, when it is detected that the mobile phone is in the unfolded state and the inner screen is in the bright screen state, and at least one of the above-mentioned second conditions is met, the camera 3 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a gesture within the camera range.

[0124] In one possible implementation, if the foldable phone meets the camera activation requirements, the phone is currently in the folded state and camera 1 is activated. Upon detecting that the phone has transitioned from the folded state to the unfolded state, and if other camera activation requirements remain unchanged, camera 1 may be deactivated and camera 3 activated to detect gestures within the range of camera 3.

[0125] In one possible implementation, if the foldable phone meets the camera activation conditions, the phone is currently in the unfolded state, and camera 3 is already activated. When it is detected that the phone has switched from the unfolded state to the folded state, and other camera activation conditions remain unchanged, camera 3 may be deactivated and camera 1 activated to detect whether there is a gesture within the range of camera 1.

[0126] It should be noted that other forms of foldable screen mobile phones can implement gesture perception methods with reference to the foldable screen mobile phone shown in Figure 19. The implementation principles and technical effects are similar. The embodiments of this application do not impose any restrictions on the structural style of the foldable screen mobile phone.

[0127] FIG6 is a schematic diagram of a gesture perception method provided by an embodiment of the present application. As shown in FIG6, in one possible embodiment, the target gesture includes a first gesture and a second gesture. If the conditions for starting the camera are met, the mobile phone camera is triggered to operate in the first mode. In the first mode, the camera continuously captures a first image of a first resolution at a first frame rate. If the mobile phone recognizes a gesture in the first image, the camera is triggered to operate in the second mode (switching from the first mode to the second mode). In the second mode, the camera continuously captures a second image of a second resolution at a second frame rate. If the gesture in the second image is recognized as the first gesture (i.e., the starting gesture of the target gesture), the camera is triggered to operate in the third mode (switching from the second mode to the third mode). In the third mode, the camera continuously captures a third image of a third resolution at a third frame rate. If the gesture in the third image is recognized as the second gesture (i.e., the ending gesture of the target gesture), the mobile phone executes the preset operation corresponding to the target gesture. In another possible embodiment, the target gesture includes the first gesture, the second gesture, and the third gesture. Unlike the previous embodiment, if the gestures in the third image are recognized as the second gesture and the third gesture (i.e., the middle gesture of the target gesture), the mobile phone executes the preset operation corresponding to the target gesture.

[0128] In this embodiment, the first frame rate of the first image is less than the second frame rate of the second image, and the second frame rate of the second image is less than the third frame rate of the third image. For example, the first frame rate can be set to 2 fps, the second frame rate can be set to 5 fps, and the third frame rate can be selected within the range of [10 fps, 30 fps], for example, the third frame rate is 10 fps.

[0129] In this embodiment, the first resolution of the first image is less than the second resolution of the second image, and the second resolution of the second image is less than or equal to the third resolution of the third image. For example, the first resolution can be set to 120×180 or 640×480. The second and third resolutions can be set to the same, such as 1920×1080.

[0130] This embodiment demonstrates a gesture sensing method in which the camera can switch between three modes. In the first mode, a first image is captured to detect whether a gesture is within camera range. In the second mode, a second image is captured to detect whether the gesture is the starting gesture of a target gesture. In the third mode, a third image is captured to detect gesture changes, specifically whether the gesture changes include intermediate and ending gestures of the target gesture. By gradually adjusting the camera's operating mode, the resolution and / or frame rate of the captured image are increased, thereby improving gesture recognition accuracy while minimizing device power consumption.

[0131] The above embodiments all involve switching of camera working modes. Switching of camera working modes is closely related to the image frame rate. The following describes a solution for adjusting the frame rate of images captured by the camera.

[0132] For example, FIG7 is a schematic diagram of a dynamic frame adjustment provided by an embodiment of the present application. As shown in FIG7 (a), in a possible frame adjustment scheme, the frame rate of the camera capturing / transmitting images is adjusted by adjusting the clock frequency. Specifically, data transmission is based on the change of the clock, and data is usually collected / transmitted at the rising or falling edge of the clock. If the frame rate of the camera capturing images needs to be doubled, the clock frequency can be doubled, and the corresponding data transmission speed will also be doubled; if the frame rate of the camera capturing images needs to be halved, the clock frequency can be halved, and the corresponding data transmission speed will also be halved. As shown in FIG7 (b), in a possible frame adjustment scheme, the frame rate of the camera capturing / transmitting images is adjusted by adjusting the ratio of the effective area / blanking area of ​​the data. Specifically, the frame rate can be adjusted by adjusting the configuration of the blanking area length. Increasing the length of the blanking area can reduce the frame rate, and conversely, reducing the length of the blanking area can increase the frame rate.

[0133] It should be noted that since the first frame adjustment scheme mentioned above requires shutting down the clock and powering it on again to modify the clock frequency, dynamic adjustment between frames cannot be achieved. Therefore, the embodiment of the present application adopts the second frame adjustment scheme mentioned above to achieve the purpose of dynamic frame adjustment.

[0134] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiment of the present application is introduced below. For example, Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in Figure 8, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0135] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110.

[0136] In some embodiments, processor 110 may include one or more interfaces.

[0137] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0138] The charging management module 140 is configured to receive charging input from a charger. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160.

[0139] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor. The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The wireless communication module 160 can provide solutions for wireless communications such as wireless local area networks (WLAN), Bluetooth, global navigation satellite systems (GNSS), frequency modulation (FM), NFC, and infrared technology (IR) applied to the electronic device 100.

[0140] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute instructions to generate or modify display information.

[0141] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0142] The electronic device 100 can implement a shooting function through an ISP, one or more cameras 193, a video codec, a GPU, one or more display screens 194, and an application processor.

[0143] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0144] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, data files such as music, photos, and videos can be stored on the external memory card.

[0145] The internal memory 121 may be used to store one or more computer programs, which include instructions. The processor 110 may execute the instructions stored in the internal memory 121 to enable the electronic device 100 to perform various functional applications and data processing.

[0146] The sensors 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0147] The gyro sensor 180B can be used to determine the motion of the electronic device 100. In some embodiments, the gyro sensor 180B can be used to determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes). The gyro sensor 180B can also be used for image stabilization. The gyro sensor 180B can also be used for navigation, somatosensory gaming, and other applications.

[0148] The magnetic sensor 180D is used to detect the magnetic field strength of the magnet, obtain magnetic data, and detect the physical state of the folding screen of the electronic device 100 through the magnetic data. The magnet is used to generate a magnetic field. In an embodiment of the present application, the magnetic sensor 180D can be set in a body corresponding to the back panel shown in (1) of Figure 19, and the magnet can be set in a body corresponding to the first screen shown in (1) of Figure 19. The magnet allows the magnetic sensor 180D to detect magnetic data. As the opening and closing state of the folding screen changes, the distance between the magnetic sensor 180D and the magnet changes accordingly, and the magnetic field strength of the magnet detected by the magnetic sensor 180D also changes. Then, the smart sensor hub can judge the physical state of the folding screen based on the magnetic data obtained by the magnetic sensor 180D under the magnetic field of the magnet. The physical state includes, for example, an unfolded state, a bracket state, or a folded state (closed state). In some embodiments, the sensor 180 can also include a Hall sensor, which can also be used to detect the magnetic field strength of the magnet and output a high / low level. The physical state of the folding screen of the electronic device 100 is determined by the high / low level.

[0149] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0150] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0151] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0152] The keys 190 include a power button, a volume button, etc. The keys 190 may be mechanical keys or touch keys. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0153] The above-mentioned electronic devices may also be referred to as terminal devices, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic devices may be mobile phones with touch screens, smart TVs, wearable devices, tablet computers (Pads), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms used by the electronic devices.

[0154] In the embodiment of the present application, in order to realize the gesture sensing function of the electronic device, it is necessary to make hardware and software improvements to the camera and processor of the electronic device. The hardware improvements of the electronic device are first described below.

[0155] For example, FIG9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in FIG9 , the electronic device may include an improved camera 901 and an improved processor 902. The improved camera 901 refers to adding a control circuit and a working circuit corresponding to the newly added shooting mode in the existing camera module to achieve a low-power configuration. For example, the shooting mode of the existing camera module is mode 1, and the camera module includes a working circuit corresponding to mode 1. If mode 2 is added to the shooting mode, correspondingly, switching between mode 1 and mode 2 is also involved. For this, the improved camera module includes, in addition to the working circuit corresponding to mode 1, a working circuit corresponding to the newly added mode 2, and a control circuit corresponding to the switching between the two modes. It should be understood that more than two shooting modes can be set according to actual application requirements, and the embodiment of the present application does not impose any restrictions on this.

[0156] In one possible implementation, the improved camera 901 includes two operating modes: a first mode and a second mode. The resolution of images captured by the camera 901 in the first mode is lower than the resolution of images captured in the second mode, and the frame rate of images captured by the camera 901 in the first mode is lower than the frame rate of images captured in the second mode. The camera 901 can switch between these two modes.

[0157] For example, in a gesture perception scenario, if the conditions for starting the camera are met, camera 901 operates in a first mode and continuously captures a first image of a first resolution at a first frame rate (i.e., the camera is in a permanent scan) to detect whether there is a gesture within the range of camera 901, i.e., to detect whether there is a gesture in the first image. If a gesture is detected within the range of camera 901, camera 901 switches from the first mode to a second mode and continuously captures a second image of a second resolution at a second frame rate to identify whether the gesture in the second image is the target gesture. In this example, the first frame rate is lower than the second frame rate, and the first resolution is lower than the second resolution.

[0158] In one possible implementation, the improved camera 901 includes three operating modes: a first mode, a second mode, and a third mode. The resolution of images captured by the camera 901 in the first mode is lower than the resolution of images captured in the second mode, and the resolution of images captured in the second mode is lower than or equal to the resolution of images captured in the third mode. The frame rate of images captured by the camera 901 in the first mode is lower than the frame rate of images captured in the second mode, and the frame rate of images captured in the second mode is lower than the frame rate of images captured in the third mode.

[0159] Exemplarily, in a gesture perception scenario, if the conditions for starting the camera are met, camera 901 operates in a first mode and continuously captures a first image with a first resolution at a first frame rate to detect whether there is a gesture within the range of camera 901, that is, to detect whether there is a gesture in the first image; if a gesture is detected within the range of camera 901, camera 901 switches from the first mode to a second mode and continuously captures a second image with a second resolution at a second frame rate to identify whether the gesture in the second image is the starting gesture of the target gesture; if the gesture in the second image is recognized as the starting gesture, camera 901 switches from the second mode to a third mode and continuously captures a third image with a third resolution at a third frame rate to identify whether the gesture in the third image is the ending gesture of the target gesture, or to identify whether the gestures in the third image are, in sequence, an intermediate gesture and an ending gesture of the target gesture. In this example, the first frame rate is less than the second frame rate, the second frame rate is less than the third frame rate, the first resolution is less than the second resolution, and the second resolution is less than or equal to the third resolution.

[0160] Based on the above examples, the camera 901 can dynamically adjust the frame rate and resolution of the captured image to adapt to different detection requirements. For example, the camera 901 captures images at a lower frame rate and resolution to detect whether there are gestures within the range of the camera 901, and the camera 901 captures images at a higher frame rate and resolution to identify the specific category of gestures in the image. For example, the minimum frame rate of the camera 901 can be 1fps, and the maximum frame rate can be 30fps. The minimum resolution of the camera 901 can be 120×180, and the maximum resolution can be 480×640. In other examples, the maximum frame rate can also be 240fps, and the maximum resolution can also be 2736×3648.

[0161] It should be noted that the embodiments of this application do not impose specific restrictions on the frame rate range and resolution range of the images captured by the camera. In other words, there are no restrictions on the minimum and maximum frame rates of the images captured by the camera, nor are there any restrictions on the minimum and maximum resolutions of the images captured by the camera. In actual applications, the frame rate range and resolution range can be reasonably set according to needs.

[0162] It should be noted that the camera 901 can be a front camera or a rear camera of the electronic device, and this embodiment does not limit this.

[0163] The improved processor 902 may be a system on chip (SoC). In an embodiment of the present application, when the electronic device triggers the camera 901 to perform a resident scan to acquire an image of the current scene, the camera 901 may send the image to the SoC for image analysis to detect whether there is a gesture in the image and the type of gesture.

[0164] To achieve low power consumption, the SoC can support the low-power AON ISP (Always On ISP). Referring to Figure 10, the camera 901 transmits the image to the AON ISP. The AON ISP does not perform any image effects processing except for format conversion. The converted image is then stored in the on-chip static random-access memory (on-chip SRAM). The SoC can also support an extremely low-power core, with computing, algorithm execution, and image storage all operating in low-power mode. In addition, the SoC can also support a low-power embedded neural network processor (eNPU) (emdedded NPU).

[0165] The SoC involved in the embodiments of the present application is described in detail below.

[0166] For example, Figure 10 is a schematic diagram of the structure of a SoC provided in an embodiment of the present application. As shown in Figure 10, the SoC includes a first processing unit and a second processing unit. The first processing unit includes an image signal processing ISP, a neural network processor NPU and a central processing unit CPU, and the second processing unit includes an I2C bus interface, an AON ISP, an on-chip SRAM, a digital signal processor DSP and an eNPU. In the SoC, the power consumption of the second processing unit is lower than that of the first processing unit. Specifically, the power consumption of the eNPU in the second processing unit is lower than that of the NPU in the first processing unit, and the power consumption of the AON ISP in the second processing unit is lower than that of the ISP in the first processing unit.

[0167] As an example, the first processing unit can be used to process the second image of the second resolution captured by the camera 901. Exemplarily, in the normal shooting mode, the camera 901 captures the second image of the second resolution, and after IPS processing, the NPU recognizes the processed second image of the second resolution, for example, identifying the category of the gesture in the second image. Before the first processing unit sends the data (such as image data) to the memory, it can perform security processing (such as encryption processing) and store the securely processed data in a security buffer of the memory.

[0168] As an example, the second processing unit can be used to process the above-mentioned first image of the first resolution captured by the camera 901. Exemplarily, in the low-power shooting mode, the camera 901 captures the first image of the first resolution, the AON ISP obtains the first image of the first resolution through the I2C bus interface, and after processing by the AON ISP, the eNPU detects the processed first image of the first resolution, for example, detecting whether there is a gesture in the first image. The on-chip SRAM in the second processing unit can be used to store the processed first image of the first resolution, and the DSP can be used to notify the eNPU to perform image detection, receive the detection results reported by the eNPU, and report the detection results to the upper-layer application. The second processing unit adopts a low-power configuration to reduce the power consumption of the electronic device.

[0169] It should be noted that the embodiments of the present application do not limit the format of the image data transmitted between the first processing unit or the second processing unit and the camera. Exemplarily, the image data may be camera serial interface (CSI) or mobile industry processor interface (MIPI) data.

[0170] The software system of an electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system as an example of a software system with a layered architecture to illustrate the software structure of an electronic device. Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The layered architecture divides the software system of an electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0171] 11 , the electronic device of an embodiment of the present application includes an application layer (Applications), an application framework layer (Application Framework), a hardware abstraction layer (HAL), a kernel layer (Kernel), a sensor control center (Sensorhub) and a hardware layer.

[0172] The application layer may include a series of applications, and the application layer runs the applications by calling the application programming interface (API) provided by the application framework layer.

[0173] In an embodiment of the present application, the application layer may include a gesture perception application and a perception module. The gesture perception application is connected to the perception module, and the gesture perception application is registered in the perception module. The perception module performs status management and data transmission. For example, when the perception module learns from the second processing module in the Sensorhub that there is a gesture within the camera range, the perception module notifies the first processing module in the HAL so that the first processing module can identify the category of the gesture, and finally the perception module reports the recognition result to the gesture perception application.

[0174] In some embodiments, the application layer also includes other applications (not shown in Figure 11), such as an always-on display application and an always-on display (AOD) application. In one possible scenario, multiple applications correspond to the same algorithm. For example, the always-on display application and the AOD application correspond to the gaze detection algorithm. The perception module can be used to uniformly schedule and manage the gaze detection algorithm. In another possible scenario, different applications correspond to different algorithms. For example, the gesture perception application corresponds to the gesture recognition algorithm (including two algorithms for identifying the presence or absence of gestures and the category of gestures), and the always-on display application corresponds to the gaze detection algorithm. Both algorithms involve obtaining image data from the underlying camera. The perception module can be used to schedule and manage the priorities of multiple algorithms. There is a possibility that the gesture recognition algorithm and the gaze detection algorithm have the same priority. The perception module can notify the underlying camera to report the image data to the gesture perception application and the always-on display application at the same time. In this embodiment, the algorithm for identifying the presence or absence of gestures is deployed in the second processing module, and the algorithm for identifying the category of gestures is deployed in the first processing module. The algorithms shown in this embodiment are only examples.

[0175] In some embodiments, if the electronic device includes a foldable screen, the application layer further includes a third processing module, which is configured to obtain the physical state of the foldable screen and the state of the internal and external screen cameras (on or off) reported by the second processing module. In addition, the third processing module is further configured to notify the first processing module of the state of the internal and external screen cameras.

[0176] In some embodiments, the application may also include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which may be system applications or third-party applications. The embodiments of the present application do not limit this.

[0177] The application framework layer provides an API and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 11, the application framework layer can include a camera service (CameraService), which is used to prioritize and manage all applications that require camera access.

[0178] In some embodiments, the application framework layer may also include, for example, a window manager, a content provider, a resource manager, a notification manager, a view system, etc., which is not limited in the embodiments of the present application.

[0179] The hardware abstraction layer may include an AO (always on) service and a first processing module. The AO service may be used to control the on or off of the algorithm for identifying gesture categories in the first processing module, and the on or off of the algorithm for identifying the presence or absence of gestures in the second processing module, as well as upper and lower layer data transmission. The first processing module may be used to process images with higher resolution and / or higher frame rate, such as the second image mentioned above, to detect the category of gestures in the second image. The first processing module is also used for camera mode switching. For example, the first processing module receives a second indication from the perception module and controls the camera to switch from the first mode to the second mode. For another example, the first processing module detects that the gesture is the starting gesture of the target gesture and controls the camera to switch from the second mode to the third mode.

[0180] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to enable it to operate. In an embodiment of the present application, the kernel layer may include a camera driver, which is used to drive the camera in the electronic device to operate in a first mode or a second mode to capture images at different frame rates and / or resolutions.

[0181] In addition, the core layer may also include display drivers, audio drivers, sensor drivers, motor drivers, etc., which are not limited in the embodiments of the present application. Among them, the sensor driver can drive, for example, a proximity light sensor to emit a light signal to detect whether the user is currently holding the electronic device close to the ear to make a call, etc. The sensor driver can also drive, for example, a gyroscope sensor to detect the posture data of the electronic device; the sensor driver can also drive, for example, an ambient light sensor to detect the ambient light brightness to detect whether the electronic device is in a dark environment, such as a mobile phone in a pocket.

[0182] The Sensorhub is used to centrally control sensors, reducing CPU load. The Sensorhub is equivalent to a microprogrammed control unit (MCU), which can run programs that drive multiple sensors. This means the Sensorhub can support multiple sensors. It can be placed as a standalone chip between the CPU and various sensors, or integrated into the CPU's application processor (AP).

[0183] In an embodiment of the present application, the Sensorhub may include a second processing module, which may be used to process images with lower resolution and / or lower frame rate, such as the first image mentioned above, to detect whether there is a gesture in the first image. Compared with the first processing module, the second processing module is a low-power processing module, and the second processing module runs permanently or in a low-power form. As an example, when the gesture perception function is turned on, the second processing module is also used to obtain data reported by various sensors, and determine the various states of the electronic device based on the various sensor data, such as screen state, unlock state, usage state, etc. If the device state meets the first condition, the second processing module may send a first shooting instruction to the camera, instructing the camera to scan the image permanently in a low-power shooting mode (such as the first mode) to detect whether there is a gesture within the camera range. As an example, for a foldable device, the second processing module may determine whether to send the first shooting instruction to the camera (the camera on the inner screen or the outer screen) by detecting whether the physical state of the folding screen, the screen state, and the device state meet the second condition. As an example, when the second processing module detects that the physical state of the screen of the foldable screen mobile phone has changed, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the second processing module can control the camera of the foldable screen mobile phone (such as the camera on the external screen of the mobile phone, and / or the camera on the internal screen of the mobile phone) to be turned on or off.

[0184] The hardware layer may include, for example, cameras, various sensors, and AON ISP.

[0185] It will be understood that the layers in the hierarchical structure shown in FIG11 and the modules or components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in this application. The modules included in each layer shown in FIG11 are the modules involved in the embodiments of this application, and the modules included in each layer do not constitute a limitation on the structure of the electronic device and the hierarchy of module deployment (example description). In some embodiments, the modules shown in FIG11 can be deployed separately, or several modules can be deployed together, and the division of modules in FIG11 is an example. In some embodiments, the names of the modules shown in FIG11 are examples.

[0186] Based on the structure of the electronic device shown above, the gesture perception method provided by the embodiment of the present application is described below in conjunction with specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0187] FIG12 is a flow chart of a gesture perception method provided in an embodiment of the present application. As shown in FIG12 , the gesture perception method provided in this embodiment includes:

[0188] Step 1201: The target application registers the gesture perception function in the perception module.

[0189] In this embodiment, the target application is a gesture perception application at the application layer. In some embodiments, the target application obtains information about the gesture perception function from a server (or cloud). After obtaining the information about the gesture perception function, the target application can register the gesture perception function in the perception module so that the perception module can run resident and execute matters related to gesture perception. The information about the gesture perception function includes code information for implementing the gesture perception function, and can also include information such as preset operations corresponding to different air gesture operations.

[0190] Step 1202: The perception module determines whether to start the gesture perception function.

[0191] If the perception module determines that the gesture perception function is activated, step 1203 is executed.

[0192] For example, referring to (a) and (b) in FIG5 , if the user chooses to turn on the gesture perception function in the setting interface of the system application, the system application may send a notification to the perception module to inform the perception module that the gesture perception function has been turned on in the mobile phone.

[0193] Step 1203: The perception module sends a first instruction to the second processing module.

[0194] The first indication is used to instruct the second processing module to detect whether there is a recognizable code within the camera range of the electronic device. The perception module sends the first indication to control the activation of the second processing module, including module power-up, work scenario delivery, resource preparation, etc.

[0195] Step 1204: The second processing module sends a first shooting instruction to the camera in response to the first instruction, where the first shooting instruction is used to instruct the camera to operate in the first mode.

[0196] The second processing module sends a first shooting instruction to control the camera startup, including camera power-on, mode switching, image output resolution and frame rate setting, etc. The first mode is a low-power shooting mode, in which the camera continuously captures an image at a lower frame rate and / or resolution, namely, the first image.

[0197] Step 1205: The camera sends the first image to the second processing module.

[0198] In response to the first shooting instruction, the camera captures a first image of a first resolution at a first frame rate in a first mode. For example, the first frame rate may be set to 5 fps, and the first resolution may be set to 120×180 or 640×480.

[0199] Step 1206: The second processing module identifies whether the first image contains a gesture.

[0200] If the second processing module recognizes a gesture in the first image, step 1207 is executed. Otherwise, the second processing module continues to execute step 1206 unless the camera is controlled to be turned off.

[0201] In some embodiments, the second processing module is pre-installed with a first gesture recognition model. The second processing module recognizes the first image through the first gesture recognition model and determines whether there is a gesture in the first image according to the recognition result output by the first gesture recognition model.

[0202] In some embodiments, the first gesture recognition model can be trained using a lightweight neural network model to identify whether there is a gesture in the image. The first gesture recognition model can be deployed on the eNPU of the second processing module and has good real-time performance.

[0203] As an example, the training process of the first gesture recognition model may include:

[0204] Step a: construct a training set and a test set for the first gesture recognition model. The training set or the test set includes sample images and the annotation results corresponding to the sample images (i.e., the presence or absence of gestures in the sample images). The sample images in the training set and the test set are different.

[0205] Step b: Training the first gesture recognition model based on the initial first gesture recognition model and the training set. Specifically, the first gesture recognition model is trained using sample images from the training set as input to the initial first gesture recognition model and the annotation results corresponding to the sample images from the training set as output from the initial first gesture recognition model.

[0206] Step c: Based on the first gesture recognition model trained in step b and the test set, the prediction result of the first gesture recognition model is verified. When the model loss function converges, the training of the first gesture recognition model is stopped.

[0207] Step 1207: The second processing module sends a first message to the perception module, where the first message is used to notify the perception module that a gesture exists within the camera range of the electronic device.

[0208] Step 1208: The perception module sends a second instruction to the first processing module.

[0209] The second indication is used to instruct the first processing module to identify the type of gesture within the camera range. The second processing module sends the second indication to control the activation of the first processing module, including powering on the module, issuing the work scenario, and preparing resources. In this embodiment, after the second processing module identifies a gesture within the camera range, the first processing module determines whether the gesture is a target gesture. Target gestures include, for example, the starting gesture, intermediate gesture, and ending gesture of the target gesture.

[0210] Step 1209: The first processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to operate in the second mode.

[0211] The first processing module sends a second shooting instruction to control the camera to switch modes, that is, switch from the first mode to the second mode. The camera continuously captures images at a higher frame rate and / or resolution in the second mode, that is, the second image.

[0212] Step 1210: The camera sends the second image to the first processing module.

[0213] In response to the second capture instruction, the camera captures a second image at a second frame rate and a second resolution in a second mode, and sends the second image to the first processing module. In this embodiment, the second image includes multiple images. For example, the second frame rate can be set to 30 fps, and the second resolution can be set to 1920×1080.

[0214] Step 1211: The first processing module determines whether there is a hand in the second image.

[0215] If it is determined that there is a hand in the second image, execute step 1213; if it is determined that there is no hand in the second image, execute step 1212.

[0216] Step 1212: The first processing module sends a fourth message to the perception module, where the fourth message is used to notify the perception module that there is no hand within the camera range.

[0217] Step 1213: The first processing module recognizes the gesture in the second image.

[0218] Step 1214: The first processing module determines whether the gesture is a target gesture.

[0219] If the gesture is determined to be a target gesture, step 1215 is executed;

[0220] If it is determined that the gesture is not the target gesture, jump back to step 1211.

[0221] In some embodiments, the first processing module is preset with a second gesture recognition model. The first processing module recognizes the gesture in the second image through the second gesture recognition model, and determines whether the gesture in the second image is a target gesture based on the recognition result output by the second gesture recognition model.

[0222] As an example, the second gesture recognition model can be trained using a lightweight neural network model based on deep learning methods and is used to identify the category of gestures in images. As an example, the second gesture recognition model can include a first model, a second model, and a third model. The first model is used to identify the overall frame and orientation of the hand, the second model is used to identify 3D hand nodes, and the third model is used to classify the identified hand nodes to determine the specific category of the gesture.

[0223] Among them, the first model can be a model that can recognize a single frame image, mainly providing accurately cropped palm images for the second model. In addition, it can recognize a variety of different palm sizes, recognize hand occlusion, and accurately locate the hand by recognizing arms, torsos or personal features. The second model can be a landmark model, which can recognize 21 hand nodes and the position occlusion between these nodes. For example, Figure 13 shows a schematic diagram of the hand nodes, and the second model can recognize the coordinates of the 21 nodes of the hand. The third model can infer the actions of each finger straightening or bending based on the data of the hand nodes extracted by the second model, and match these actions with preset gestures to predict the category of the gesture.

[0224] As an example, the second gesture recognition model can be deployed on the NPU of the first processing module, which has good real-time performance.

[0225] In some embodiments, the first processing module recognizes gestures in the second image, which may include: identifying a hand area from the second image to remove non-hand areas in the second image; performing gesture analysis on the identified hand area, such as inputting the image of the identified hand area into the above-mentioned second gesture recognition model to determine the category of the gesture in the image.

[0226] In some embodiments, the first processing module may use background subtraction to separate the foreground (ie, the hand area) and the background (ie, the non-hand area) in the second image.

[0227] Step 1215: The first processing module sends a second message to the perception module, where the second message is used to indicate that the target gesture is recognized in the second image.

[0228] As an example, the second message includes an identifier corresponding to the target gesture.

[0229] Step 1216: The perception module sends a third indication to the target application, where the third indication is used to indicate that the target gesture is recognized in the second image.

[0230] As an example, the third indication includes an identifier corresponding to the target gesture.

[0231] Step 1217: The target application controls execution of a preset operation corresponding to the target gesture.

[0232] The target application stores a correspondence between the target gesture and the preset operation (for details, please refer to the examples in the above embodiment). In response to the third instruction, the target application executes the preset operation corresponding to the target gesture.

[0233] In the above embodiment, if the electronic device has registered the gesture perception function and has turned on the gesture perception function, the perception module can send a first shooting instruction to the camera through the second processing module (low-power processing module) to enable the camera to capture the first image at a lower frame rate and / or resolution. If the second processing module recognizes that there is a gesture in the first image, it can notify the first processing module so that the first processing module sends a second shooting instruction to the camera to enable the camera to capture the second image at a higher frame rate and / or resolution. The first processing module recognizes the gesture category in the second image. If the gesture in the second image is a target gesture, the application is notified to perform a preset operation corresponding to the target gesture.

[0234] In the above solution, after the user turns on the gesture sensing function, the electronic device automatically detects gestures through the air with low power consumption, that is, detects the presence or absence of gestures. After detecting the gesture, a higher-resolution image is captured and the type of gesture is detected based on the higher-resolution image. If the gesture is a target gesture, the preset operation corresponding to the target gesture is executed, enabling the device to automatically sense and recognize gestures to meet the user's needs for remote control of the device. Because both the camera and the second processing module adopt a low-power configuration, the power consumption of executing this solution is extremely low.

[0235] FIG14 is a flow chart of a gesture perception method provided in an embodiment of the present application. Based on the embodiment shown in FIG12 , as shown in FIG14 , the gesture perception method provided in this embodiment includes:

[0236] Step 1401: The target application registers the gesture perception function in the perception module.

[0237] Step 1402: The perception module determines whether to start the gesture perception function.

[0238] If the perception module determines that the gesture perception function is activated, step 1403 is executed.

[0239] Step 1403: The perception module sends a first indication to the second processing module, where the first indication is used to instruct the second processing module to detect whether there is a gesture within the camera range of the electronic device.

[0240] Step 1404: The second processing module determines whether the first condition is met.

[0241] If the second processing module determines that the first condition is met, step 1405 is executed. Otherwise, the second processing module continues the low power consumption detection to determine whether the first condition is met.

[0242] In this embodiment, the first condition includes at least one of the following:

[0243] The screen of the electronic device is in a bright state;

[0244] The electronic device is unlocked;

[0245] The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal;

[0246] Detection data of the ambient light sensor of the electronic device is greater than a third threshold;

[0247] The screen of the electronic device faces a preset direction;

[0248] The electronic device runs the target application.

[0249] In this embodiment, if the electronic device has the gesture sensing function enabled and the first condition is met, the camera of the electronic device is triggered to continuously capture the first image. By adding the first condition, the electronic device is prevented from continuously capturing the first image when not necessary, further reducing the power consumption of the device.

[0250] Step 1405: The second processing module sends a first shooting instruction to the camera, where the first shooting instruction is used to instruct the camera to operate in a first mode.

[0251] In some embodiments, if the electronic device is a foldable device, the foldable device includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera.

[0252] In one possible scenario, step 1404 may be replaced by: the second processing module detects that the inner screen of the electronic device is in a bright screen state, and the electronic device is in an unfolded state. Accordingly, step 1405 may be: the second processing module sends a first shooting instruction to the first camera. In some embodiments, before the second processing module sends the first shooting instruction to the first camera, it also includes: detecting that the state of the electronic device meets the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the inner screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the inner screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0253] In one possible scenario, step 1404 may be replaced by: the second processing module detects that the external screen of the electronic device is in a bright screen state, and the electronic device is in a folded state. Accordingly, step 1405 may be: the second processing module sends a first shooting instruction to the second camera. In some embodiments, before the second processing module sends the first shooting instruction to the second camera, it also includes: detecting that the state of the electronic device meets the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the external screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the external screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0254] Step 1406: The camera sends the first image to the second processing module.

[0255] Step 1407: The second processing module identifies whether the first image contains a gesture.

[0256] If the second processing module recognizes a gesture in the first image, step 1408 is executed;

[0257] If the second processing module does not recognize the gesture in the first image, the process jumps back to step 1404 .

[0258] Step 1408: The second processing module sends a first message to the perception module, where the first message is used to notify the perception module that a gesture exists within the camera range of the electronic device.

[0259] Step 1409: The perception module sends a second instruction to the first processing module, where the second instruction is used to instruct the first processing module to identify the category of gestures within the camera range.

[0260] Step 1410: The first processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to operate in the second mode.

[0261] Step 1411: The camera sends the second image to the first processing module.

[0262] Step 1412: The first processing module determines whether there is a hand in the second image.

[0263] If it is determined that there is a hand in the second image, go to step 1414 ; if it is determined that there is no hand in the second image, go to step 1413 .

[0264] Step 1413: The first processing module sends a fourth message to the perception module, where the fourth message is used to notify the perception module that there is no hand within the camera range.

[0265] Step 1414: The first processing module recognizes the gesture in the second image.

[0266] Step 1415: The first processing module determines whether the gesture is a target gesture.

[0267] If the gesture is determined to be a target gesture, step 1416 is executed;

[0268] If it is determined that the gesture is not the target gesture, jump back to step 1412.

[0269] Step 1416: The first processing module sends a second message to the perception module, where the second message is used to indicate that the target gesture is recognized in the second image.

[0270] Step 1417: The perception module sends a third indication to the target application, where the third indication is used to indicate that the target gesture is recognized in the second image.

[0271] Step 1418: The target application controls execution of a preset operation corresponding to the target gesture.

[0272] In the above embodiment, if the electronic device has turned on the gesture perception function and meets the above-mentioned first condition, the perception module can send a first shooting instruction to the camera through the second processing module, so that the camera captures the first image at a lower frame rate and / or resolution. If the second processing module recognizes that there is a gesture in the first image, it can notify the first processing module, and the first processing module sends a second shooting instruction to the camera, so that the camera captures the second image at a higher frame rate and / or resolution. The first processing module identifies whether the gesture in the second image is a target gesture. If it is a target gesture, the preset operation corresponding to the target gesture is executed to realize automatic perception and recognition of gestures by the device, so as to meet the function of the user to operate the device remotely. On the one hand, since both the camera and the second processing module adopt low-power configuration, the power consumption of executing the above scheme is extremely low. On the other hand, setting the first condition can avoid the electronic device from performing unnecessary gesture detection, thereby further reducing the power consumption of the device.

[0273] In some embodiments, the target gesture includes a first gesture and a second gesture, where the first gesture is the starting gesture of the target gesture and the second gesture is the ending gesture of the target gesture. If the first processing module recognizes that the gesture in the second image is the first gesture, the first processing module sends a third shooting instruction to the camera. The third shooting instruction is used to instruct the camera to operate in the third mode, that is, the first processing module controls the camera to switch from the second mode to the third mode. The first processing module obtains a third image captured by the camera in the third mode and identifies the category of the gesture in the third image. This recognition process is consistent with the process of identifying the second image described above and will not be repeated here. If the first processing module recognizes that the gesture in the third image is the second gesture, the first processing module sends a third message to the perception module, so that after receiving the third message, the perception module sends a fourth instruction to the target application, instructing the target application to control the execution of a preset operation corresponding to the target gesture. In this embodiment, the third message is used to indicate the recognition of the target gesture, and the fourth instruction is used to indicate the target gesture (the identifier corresponding to the target gesture). If the target gesture includes three gestures, similar to this embodiment, the execution principle of two gestures can be referred to.

[0274] The embodiment of the present application also provides a gesture perception method, which is described below in conjunction with FIG15 .

[0275] FIG15 is a flow chart of a gesture sensing method provided in an embodiment of the present application. As shown in FIG15 , the electronic device determines whether the gesture sensing function is enabled. If the gesture sensing function is enabled, the electronic device determines whether the preset conditions are met. The preset conditions include: the electronic device screen is on, unlocked, the proximity light is not blocked, the ambient light brightness is greater than a third threshold, and the screen is facing a preset direction. In this embodiment, the proximity light is not blocked by detecting the light signal emitted by the proximity light sensor. For details, please refer to the previous embodiment and will not be elaborated here. The preset direction includes the direction in which the user holds the mobile phone to browse the interface content, the direction in which the mobile phone is placed on the desktop to browse the interface content, etc. If the electronic device determines that the gesture sensing function is enabled and the above-mentioned preset conditions are met, the camera is controlled to capture a first image and detect whether there is a gesture in the first image, wherein the frame rate of the first image can be 2fps. If a gesture is detected in the first image, the camera is controlled to capture a second image and detect whether there is a preset starting gesture in the second image, wherein the frame rate of the second image can be 5fps. Otherwise, the process jumps to the step of whether the preset conditions are met. If a preset starting gesture is detected in the second image, the camera is controlled to capture the third image, wherein the frame rate of the third image can be any value of 10-30fps, otherwise jump to the step of whether the preset conditions are met. If a preset target gesture is detected in the third image, the preset operation corresponding to the target gesture is responded to, otherwise jump to the step of whether the preset conditions are met. The target gesture in this embodiment includes the above-mentioned starting gesture, and may also include an intermediate gesture and an ending gesture, that is, the target gesture corresponds to a gesture change. The preset operation in this embodiment may be one of the operations described in the above embodiment.

[0276] Based on the above solution, after turning on the above gesture perception function, users can use air gestures directly within the range of the mobile phone camera under certain conditions to achieve the effect of controlling the mobile phone through the air, meet the user's needs for air operation in various scenarios, and improve the user's mobile phone experience.

[0277] The present application also provides a gesture sensing method for an electronic device with a flexible screen. The following uses a foldable screen mobile phone as an example to illustrate the method. A set of cameras is provided on both the inner and outer screens of the foldable screen mobile phone to capture image data. The gesture sensing method of this embodiment involves the processing logic of the device's underlying modules when the foldable screen changes state, as described below in conjunction with Figure 20.

[0278] For example, FIG20 is a flowchart of a gesture perception method provided by an embodiment of the present application. As shown in FIG20 , the gesture perception method of this embodiment may include the following steps:

[0279] Step 2010: The second processing module obtains sensor data to determine the change in the physical state of the folding screen.

[0280] In this embodiment, sensor data includes, for example, magnetic sensors and Hall sensors, and is used to determine whether the physical state of the foldable screen has changed. The change in the physical state of the foldable screen includes a change from a folded state to an unfolded state, or vice versa.

[0281] Step 2020a: The second processing module controls the camera on the inner screen to be turned on or off based on the change in the physical state of the folding screen.

[0282] Step 2020b: The second processing module controls the camera on the external screen to be turned on or off based on the change in the physical state of the folding screen.

[0283] In a possible implementation, if the physical state of the folding screen changes from a folded state to an unfolded state, the second processing module may control the camera on the inner screen to be turned on.

[0284] In one possible implementation, the camera on the outer screen is turned on. If the physical state of the folding screen changes from a folded state to an unfolded state, the second processing module can control the camera on the outer screen to be turned off, and at the same time control the camera on the inner screen to be turned on.

[0285] In one possible implementation, the camera on the inner screen is turned on. If the physical state of the folding screen changes from an unfolded state to a folded state, the second processing module can control the camera on the inner screen to be turned off and the camera on the outer screen to be turned on at the same time.

[0286] Exemplarily, the camera on the outer screen may be camera 1 shown in FIG. 19 , and the camera on the inner screen may be camera 3 shown in FIG. 19 .

[0287] Step 2020c: The second processing module reports the physical state of the folding screen and the state of the internal and external screen cameras to the third processing module.

[0288] It should be noted that this embodiment does not impose any limitation on the execution order of steps 2020a to 2020c.

[0289] Step 2030: The third processing module sends a notification to the first processing module to inform the status of the internal and external screen cameras.

[0290] This embodiment shows the interaction process between various modules inside a foldable screen mobile phone when the physical state of the screen changes. Through the above interaction, precise control of the low-power continuous scanning function of the camera is achieved, so that when the user uses the foldable screen mobile phone, the mobile phone can recognize the user's air gesture operations, and then realize the control of the mobile phone in different usage scenarios, such as audio and video playback control, picture display control, etc., to enhance the user experience of the phone.

[0291] Figure 16 is a structural diagram of an electronic device provided in an embodiment of the present application. As shown in Figure 16, the electronic device includes a camera 1606, a processor 1601, a communication line 1604 and at least one communication interface (communication interface 1603 is used as an example in Figure 16).

[0292] The camera 1606 may be used to capture images with different frame rates and / or resolutions, and the processor 1601 may be used to detect the presence of gestures in the images and the specific type of gestures.

[0293] Processor 1601 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control program execution of the present application. In some embodiments, the processor includes a first processing module and a second processing module, wherein the first processing module consumes more power than the second processing module; the second processing module is used to detect whether a gesture exists within the camera range; and the first processing module is used to identify the type of gesture within the camera range.

[0294] Communications link 1604 may include circuitry to transmit information between the aforementioned components.

[0295] The communication interface 1603 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0296] In some embodiments, the electronic device may further include a memory 1602 .

[0297] The memory 1602 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a communication line 1604. The memory may also be integrated with the processor.

[0298] The memory 1602 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1601. The processor 1601 is used to execute the computer-executable instructions stored in the memory 1602, thereby implementing the gesture perception method provided by the embodiment of the present application.

[0299] In some embodiments, the electronic device further includes a display screen 1607 , which may be a foldable screen.

[0300] The computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0301] As an example, the processor 1601 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 16 .

[0302] As an example, an electronic device may include multiple processors, such as processor 1601 and processor 1605 in FIG16 . Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0303] FIG17 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. As shown in FIG17 , a chip 1700 includes one or more (including two) processors 1720 and a communication interface 1730 .

[0304] In some embodiments, the memory 1740 stores the following elements: executable modules or data structures, or a subset of executable modules or data structures, or an extended set of executable modules or data structures.

[0305] In the embodiment of the present application, the memory 1740 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1720. A portion of the memory 1740 may also include a non-volatile random access memory (NVRAM).

[0306] In the embodiment of the present application, the memory 1740, the communication interface 1730, and the memory 1740 are coupled together via a bus system 1710. In addition to a data bus, the bus system 1710 may also include a power bus, a control bus, and a status signal bus. For ease of description, each bus is labeled as the bus system 1710 in FIG. 17 .

[0307] The method described in the above embodiment of the present application can be applied to the processor 1720, or implemented by the processor 1720. The processor 1720 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the hardware integrated logic circuit in the processor 1720 or the instructions in the form of software. The above-mentioned processor 1720 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1720 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present invention.

[0308] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product. The computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0309] The present application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on an electronic device, the electronic device executes the technical solution in the above-mentioned embodiment. The implementation principle and technical effects thereof are similar to those in the above-mentioned related embodiments and are not further described here.

[0310] Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0311] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0312] Computer-readable storage media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. Computer-readable storage media may include: compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable storage media may include magnetic disk storage or other magnetic disk storage devices. Furthermore, any connecting line may also be appropriately referred to as a computer-readable storage medium. For example, if a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) is used to transmit software from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0313] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0314] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A gesture perception method, characterized in that: Applied to electronic equipment, the method includes: The camera of the electronic device operates in a first mode; The electronic device obtains, by the camera in the first mode, a first image captured, and detects whether there is a gesture in the first image; The electronic device detects a gesture in the first image and controls the camera to switch from the first mode to the second mode; The electronic device obtains a second image captured by the camera in the second mode, and identifies a category of a gesture in the second image; The electronic device recognizes that the gesture in the second image is a target gesture, and controls execution of a preset operation corresponding to the target gesture.

2. The method according to claim 1, characterized in that Before the camera of the electronic device operates in the first mode, the method further includes: the electronic device responding to a first operation of enabling a gesture sensing function.

3. The method according to claim 1 or 2, characterized in that Before the camera of the electronic device operates in the first mode, the method further includes: It is detected that the state of the electronic device satisfies a first condition; the first condition includes at least one of the following: The screen state of the electronic device is a bright screen state; The electronic device is unlocked; The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; Detection data of the ambient light sensor of the electronic device is greater than a third threshold; The screen of the electronic device faces a preset direction; The electronic device runs a preset application.

4. The method according to claim 1 or 2, characterized in that The electronic device is a foldable device, comprising an inner screen and an outer screen, the inner screen being provided with a first camera, and the outer screen being provided with a second camera; the camera of the electronic device operating in a first mode, including: detecting that the outer screen of the electronic device is in a light-on state and the electronic device is in a folded state, and controlling the second camera to operate in a first mode; or It is detected that the inner screen of the electronic device is in a bright screen state and the electronic device is in an unfolded state, and the first camera is controlled to operate in a first mode.

5. The method according to claim 4, characterized in that Before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, the method further includes: It is detected that the state of the electronic device satisfies a second condition; the second condition includes at least one of the following: The electronic device is unlocked; The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; Detection data of the ambient light sensor of the electronic device is greater than a third threshold; The inner screen or outer screen of the electronic device faces a preset direction; The electronic device runs a preset application.

6. The method according to claim 4 or 5, characterized in that The method further comprises: When the second camera operates in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera; When the first camera operates in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to turn off and controls the second camera to operate in the first mode.

7. The method according to any one of claims 1 to 6, characterized in that The target gesture includes a first gesture and a second gesture, the first gesture is a starting gesture of the target gesture, and the second gesture is a terminating gesture of the target gesture; The electronic device recognizes that the gesture in the second image is the first gesture, and the electronic device controls execution of a preset operation corresponding to the target gesture, including: The electronic device recognizes that the gesture in the second image is the first gesture, and controls the camera to switch from the second mode to the third mode; The electronic device acquires a third image captured by the camera in the third mode, and identifies a category of a gesture in the third image; The electronic device recognizes that the gesture in the third image is the second gesture, and controls execution of a preset operation corresponding to the target gesture.

8. The method according to any one of claims 1 to 7, characterized in that The camera of the electronic device operates in a first mode, including: The sensing module of the electronic device sends a first instruction to the second processing module of the electronic device, where the first instruction is used to instruct the second processing module to detect whether there is a gesture within the camera range; The second processing module sends a first shooting instruction to the camera; The camera operates in the first mode in response to the first shooting instruction.

9. The method according to any one of claims 1 to 8, characterized in that The electronic device acquires a first image captured by the camera in the first mode, and detects whether there is a gesture in the first image, including: The second processing module of the electronic device obtains the first image captured by the camera in the first mode, and detects whether there is a gesture in the first image.

10. The method according to any one of claims 1 to 9, characterized in that The electronic device detects a gesture in the first image and controls the camera to switch from the first mode to the second mode, including: The second processing module of the electronic device detects a gesture in the first image, and the second processing module sends a first message to the perception module of the electronic device, where the first message is used to notify the perception module that there is a gesture within the camera range; The perception module sends a second instruction to the first processing module of the electronic device, where the second instruction is used to instruct the first processing module to identify the category of the gesture in the second image; The first processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to operate in a second mode.

11. The method according to any one of claims 1 to 10, characterized in that The electronic device acquires a second image captured by the camera in the second mode, and identifies a category of a gesture in the second image, including: The first processing module of the electronic device obtains a second image captured by the camera in the second mode, and identifies a category of a gesture in the second image.

12. The method according to any one of claims 1 to 11, characterized in that The electronic device recognizes that the gesture in the second image is a target gesture, and controls execution of a preset operation corresponding to the target gesture, including: The first processing module of the electronic device recognizes that the gesture in the second image is a target gesture, and the first processing module sends a second message to the perception module of the electronic device, where the second message is used to indicate that the target gesture is recognized in the second image; The perception module sends a third indication to the target application of the electronic device, where the third indication is used to indicate the target gesture; The target application controls execution of a preset operation corresponding to the target gesture.

13. The method according to any one of claims 3 to 6, characterized in that The second processing module of the electronic device detects the state of the electronic device.

14. The method according to claim 7, wherein: The electronic device controls the camera to switch from the second mode to the third mode, comprising: a first processing module of the electronic device controls the camera to switch from the second mode to the third mode; The electronic device acquires a third image captured by the camera in the third mode and identifies a category of a gesture in the third image, including: the first processing module acquires the third image captured by the camera in the third mode and identifies a category of a gesture in the third image; If the gesture in the third image is the second gesture, the electronic device controls the execution of a preset operation corresponding to the target gesture, including: if the gesture in the third image is the second gesture, the first processing module sends a third message to the perception module of the electronic device, and the third message is used to indicate that the target gesture is recognized; the perception module sends a fourth indication to the target application of the electronic device, and the fourth indication is used to indicate the target gesture; the target application controls the execution of a preset operation corresponding to the target gesture.

15. An electronic device, characterized in that: The electronic device includes: a camera, a memory and a processor; The camera is used to capture images with different frame rates and / or resolutions, and the processor is used to call the computer program in the memory to execute the gesture perception method according to any one of claims 1 to 14.

16. The electronic device according to claim 15, characterized in that The processor includes a first processing module and a second processing module, wherein the power consumption of the first processing module is higher than the power consumption of the second processing module; The second processing module is used to detect whether there is a gesture within the camera range; The first processing module is used to identify the category of the gesture within the camera range.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 14.

18. A chip, characterized in that: The chip includes a processor, and the processor is configured to call a computer program in a memory to execute the method according to any one of claims 1 to 14.

Images