Gesture sensing method and electronic equipment

Abstract

The invention provides a gesture sensing method and electronic equipment, relates to the technical field of terminals and the field of artificial intelligence, and further relates to the fields of intelligent sensing, intelligent control and the like. In the method, the electronic equipment controls a camera to collect a first image of a first resolution, if an effective gesture is detected in the first image, the camera is controlled to be switched from a first mode to a second mode, and a second image of a second resolution is collected through the second mode. Furthermore, gesture detection is carried out on the second image and a third image obtained through upward interpolation of the first image, and whether a subsequent gesture sensing process is triggered or not is determined according to a gesture detection result so as to detect a target dynamic gesture. According to the scheme, the electronic equipment can achieve the gesture sensing function with low power consumption, the gesture made by the user can be responded more quickly when the function is achieved, and the gesture sensing efficiency and the gesture sensing success rate are improved.

Description

Technical Field

[0001] The present application relates to the field of electronic devices, and in particular to a gesture sensing method and electronic device. Background Art

[0002] Traditional contact-based human-computer interaction involves users interacting with the computer through a touch keyboard, mouse, tablet, touch screen, or physical buttons. With the advancement of human-computer interaction technology for smart terminals, users can now control smart terminals through gestures in the air, a form of contactless human-computer interaction.

[0003] In related technologies, users can open the camera app and use air gestures to control the camera to take photos. This function requires the terminal to continuously recognize gestures. In addition, users also need air gestures in more scenarios where it is inconvenient to touch the terminal screen, such as when driving a car or when their fingers are dirty.

[0004] However, in some scenarios, electronic devices may fail to recognize user gestures, affecting the user experience. Improving the success rate of gesture recognition is an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of the present application provide a gesture perception method and an electronic device, which can enable the electronic device to implement a gesture perception function with lower power consumption, and when implementing this function, can respond more quickly to gestures made by the user.

[0006] In a first aspect, a gesture perception method is provided, which is applied to an electronic device, wherein the electronic device includes a camera, and the method includes: controlling the camera to operate in a first mode, wherein the first mode is used to capture images of a first resolution; obtaining a frame of a first image captured by the camera in the first mode, and detecting the first image to obtain a first detection result, wherein the first detection result is used to indicate whether a first valid gesture exists in the first image, wherein the first valid gesture is any one of the starting gestures of one or more preset dynamic gestures; when the first valid gesture is detected from the first image, controlling the camera to switch from the first mode to the second mode, wherein the second mode is used to capture images of a second resolution, wherein the first resolution is lower than the second resolution; obtaining a frame of a second image captured by the camera in the second mode, and detecting the second image A detection is performed to obtain a second detection result, and the second detection result is used to indicate whether a second valid gesture exists in the second image, and the category of the second valid gesture when the second valid gesture exists, and the second valid gesture is any one of the starting hand postures of one or more preset dynamic gestures; the first image is interpolated to obtain a third image, and the third image is detected to obtain a third detection result, and the third detection result is used to indicate the category of the first valid gesture, and the resolution of the third image is the same as the resolution of the second image or the resolution of the third image is greater than a preset value; based on the second detection result and the third detection result, it is determined whether to execute the gesture perception process, and the gesture perception process is used to detect the target dynamic gesture and execute the preset operation corresponding to the target dynamic gesture, and the target dynamic gesture is one of one or more preset dynamic gestures.

[0007] Based on the above solution, the electronic device controls the camera to capture images in a first mode to detect whether a valid gesture exists within the camera's range. Because the first mode is a low-resolution mode, the electronic device can continuously capture images with low power consumption. After detecting a valid gesture in the first image (a frame captured in the first mode), the electronic device switches the capture mode and controls the camera to capture a second image in a second mode to detect whether a valid gesture exists within the camera's range and the type of valid gesture. Furthermore, the electronic device interpolates the first image to generate a high-resolution third image and detects the type of valid gesture in the third image. Finally, based on the detection results of the second and third images, it determines whether to execute the gesture detection process. By setting the camera in both high-resolution and low-resolution modes, this solution automatically switches the camera mode between different detection stages to meet the image quality requirements of each detection stage, thereby reducing device power consumption. Furthermore, this solution can determine whether to trigger the gesture detection process based on only two frames of images, ensuring a high detection success rate even when the user performs gestures quickly, thus achieving a balance between power consumption and performance.

[0008] In combination with the first aspect, in certain implementations of the first aspect, interpolating the first image to obtain a third image includes: after determining that a second valid gesture exists in the second image based on a second detection result, interpolating the first image to obtain the third image.

[0009] Based on the above scheme, when the electronic device detects the second valid gesture from the second image, it interpolates the first image to obtain the third image. As an example, the electronic device can extract the first image from the cache. This implementation method is conducive to reducing power consumption. Specifically, after the electronic device detects the first valid gesture from the first image, it will cache the first image and control the camera to switch to the second mode to capture the second image. Since the first valid gesture detected in the first image may not be the gesture made by the user when performing an air gesture operation, for example, the first valid gesture may be the gesture made by the user when he accidentally passes his hand across the camera, in this case, the second valid gesture is likely not to be detected in the second image. Therefore, in this implementation method, this step is only triggered when it is determined that the second valid gesture has been detected from the second image, which can reduce unnecessary interpolation operations, save resources, and reduce power consumption.

[0010] In combination with the first aspect, in certain implementations of the first aspect, interpolating the first image to obtain the third image includes: after determining that the first image includes a first valid gesture based on the first detection result, directly interpolating the first image to obtain the third image.

[0011] Based on the above solution, when the electronic device detects the first valid gesture from the first image, it directly interpolates the first image. Because in some scenarios, the starting gestures of preset dynamic gestures are relatively special gestures, the probability of user error is relatively low. Therefore, after detecting the first valid gesture from the first image, interpolating the first image in advance can improve the efficiency of gesture perception and further increase the success rate of gesture perception when the user performs gestures quickly.

[0012] In combination with the first aspect, in certain implementations of the first aspect, detecting the first image to obtain a first detection result includes: inputting the first image into a first model and outputting the first detection result, and the first model is a binary classification model; detecting the second image to obtain a second detection result includes: inputting the second image into a second model and outputting the second detection result, and the second model is a multi-classification model; detecting the third image to obtain a third detection result includes: inputting the third image into the second model and outputting the third detection result.

[0013] Based on the above solution, the electronic device performs effective gesture detection on the first image through a binary classification model, that is, the first detection result only displays two results, namely, the first image includes the first effective gesture, or the first image does not include the first effective gesture, without indicating the specific category of the first effective gesture. Compared with the multi-classification model, the power consumption caused by the binary classification model is lower, and because the electronic device needs to continuously capture images of the first resolution through the camera in the first mode and perform effective gesture detection on these first resolution images, the detection frequency is very high. The present application detects images of the first resolution through a binary classification model, which can effectively reduce the power consumption of the electronic device.

[0014] Based on the above solution, the electronic device uses a multi-classification model to detect valid gestures in the second and third images. Taking the second detection result as an example, the second detection result can display multiple results, namely, the second image does not include the second valid gesture, or the second valid gesture in the second image is classified as category #1, or the second valid gesture in the second image is classified as category #2, or the second valid gesture in the second image is classified as category #3. Compared to a binary classification model, a multi-classification model can output the specific category of the detected valid gesture, allowing the electronic device to determine which starting gesture the user performed based on the detection result, so as to perform the subsequent gesture perception process.

[0015] In combination with the first aspect, in certain implementations of the first aspect, when the category of the second valid gesture is different from the category of the first valid gesture, the method also includes: obtaining n consecutive frames of images captured by the camera in the second mode, and detecting the n frames of images in turn to obtain n fourth detection results, the n fourth detection results corresponding one-to-one to the n frames of images, each fourth detection result being used to indicate whether the corresponding image has a third valid gesture, and the category of the third valid gesture when the third valid gesture exists, and n is an integer greater than or equal to 1; when the n fourth detection results indicate that the third valid gesture does not exist in any of the n frames of images, controlling the camera to switch from the second mode back to the first mode.

[0016] In the above solution, if the category of the first valid gesture is different from the category of the second valid gesture, the electronic device reacquires the high-resolution (i.e., second-resolution) image captured by the camera in the second mode and detects the captured high-resolution image to determine whether a valid gesture has been detected. If the number of detections exceeds a preset threshold (e.g., n) and no valid gesture is detected, the camera is controlled to switch from the second mode back to the first mode to save power.

[0017] In combination with the first aspect, in some implementations of the first aspect, the first image and the second image are two adjacent frames of images, or there is one frame of image between the first image and the second image.

[0018] In the above scheme, an invalid frame is allowed to exist between two valid frames. That is to say, in one implementation, if the electronic device obtains two consecutive frames of images through the camera, the first frame of the two frames is an image of the first resolution, and the second frame is an image of the second resolution, and valid gestures are detected in both frames, then the subsequent detection process is triggered. In another implementation, if the electronic device obtains three consecutive frames of images through the camera, the first frame of the three frames is an image of the first resolution, the second and third frames are images of the second resolution, and valid gestures are detected in the first and third frames, but no valid gesture is detected in the second frame, in this case, the subsequent detection process is still triggered, that is, it is compatible with the situation where the three consecutive frames of images are <valid low resolution, invalid high resolution, valid high resolution> in sequence. In this way, more scenarios can be compatible and the detection success rate can be improved.

[0019] In combination with the first aspect, in certain implementations of the first aspect, when a first valid gesture is detected from a first image, the method further includes: caching the first image; after determining that the first detection result indicates that a second valid gesture exists in the second image, interpolating the first image to obtain a third image, including: after determining that the first detection result indicates that a second valid gesture exists in the second image, extracting the first image from the cache, and then interpolating the first image to obtain the third image.

[0020] In combination with the first aspect, in some implementations of the first aspect, when no first valid gesture is detected from the first image, the method further includes: discarding the first image.

[0021] In the above solution, only images in which the first valid gesture was detected can be cached, rather than all images at the first resolution. This is because if an image does not detect the first valid gesture, it will not be used in subsequent detection processes, so discarding it directly saves storage space. By caching the first image in which the first valid gesture was detected, the first image can be directly retrieved from the cache during the procedure detection process.

[0022] In combination with the first aspect, in certain implementations of the first aspect, determining whether to execute the gesture perception process is based on the first detection result and the second detection result, including: when the second detection result indicates that the second image includes a second valid gesture, and the category of the second valid gesture is the same as the category of the first valid gesture indicated by the third detection result, executing the gesture perception process, and the first valid gesture and the second valid gesture are the starting gestures of the target dynamic gesture.

[0023] In the above scheme, the electronic device determines that the starting gesture has been detected through a dual-frame confirmation mechanism, that is, the starting gesture is detected in both frames of the image before continuing to execute the subsequent gesture perception process. This can reduce redundant detection for certain scenarios, such as the scenario where the user inadvertently passes his hand across the camera, thereby reducing power consumption.

[0024] In combination with the first aspect, in certain implementations of the first aspect, when the category of the second valid gesture indicated by the first detection result is the same as the category of the first valid gesture indicated by the second detection result, the method also includes: displaying an indicator icon, where the indicator icon is used to prompt the user to continue inputting the target dynamic gesture.

[0025] This indicator icon can prompt the user to continue performing other gestures after the electronic device confirms the detection of the starting gesture of the target dynamic gesture, thereby improving the user experience. For example, after the user observes the indicator icon, the user knows that the electronic device has detected the starting gesture of the target dynamic gesture and is ready to detect other gestures in the target dynamic gesture. At this time, the user can continue to perform other gestures to trigger the preset operation corresponding to the target dynamic gesture.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the target dynamic gesture includes a first gesture and a second gesture, the first gesture is the starting gesture of the target dynamic gesture, and the second gesture is the terminating gesture of the target dynamic gesture; executing the gesture perception process, including: obtaining a frame of a fourth image captured by the camera in the second mode, and detecting the fourth image to obtain a fifth detection result, the fifth detection result is used to indicate whether there is a fourth valid gesture in the fourth image, and the category of the fourth valid gesture when there is a fourth valid gesture, the fourth valid gesture is any one of the terminating gestures of one or more preset dynamic gestures; when the fifth detection result indicates that the fourth valid gesture is the second gesture, controlling the execution of the preset operation corresponding to the target dynamic gesture.

[0027] In combination with the first aspect, in certain implementations of the first aspect, the target dynamic gesture includes a third gesture, a fourth gesture, and a fifth gesture, the third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the intermediate gesture of the target dynamic gesture, and the fifth gesture is the terminating gesture of the target dynamic gesture; executing a gesture perception process, including: obtaining a frame of a fifth image and a frame of a sixth image captured by the camera in the second mode, and detecting the fifth image and the sixth image respectively to obtain a sixth detection result and a seventh detection result, the sixth detection result is used to indicate whether there is a fifth valid gesture in the fifth image, and the category of the fifth valid gesture when the fifth valid gesture exists, the fifth valid gesture is any one of the intermediate gestures of one or more preset dynamic gestures, the seventh detection result is used to indicate whether there is a sixth valid gesture in the sixth image, and the category of the sixth valid gesture when the sixth valid gesture exists, the sixth valid gesture is any one of the terminating gestures of one or more preset dynamic gestures; when the sixth detection result indicates that the fifth valid gesture is the fourth gesture, and the seventh detection result indicates that the sixth valid gesture is the fifth gesture, controlling the execution of a preset operation corresponding to the target dynamic gesture.

[0028] The above two solutions are respectively compatible with the case where the target dynamic gesture consists of two gestures and the case where the target dynamic gesture consists of three gestures, that is, they are compatible with various application scenarios. In addition, in the embodiment of the present application, triggering the gesture perception process means that when the first valid gesture and the second valid gesture are of the same category, other gestures in the target dynamic gesture except the starting gesture are detected. If other gestures are detected, the preset operation corresponding to the target dynamic gesture is executed.

[0029] In combination with the first aspect, in some implementations of the first aspect, the first mode is further used to capture images at a first frame rate, and the second mode is further used to capture images at a second frame rate, and the first frame rate is lower than the second frame rate.

[0030] Based on the above solution, the camera can dynamically adjust the frame rate of image capture to adapt to different detection requirements. For example, in the first mode, the camera captures images at a lower frame rate to detect whether there is a first valid gesture within the camera range, and then captures images at a higher frame rate to identify whether there is a second valid gesture in the image and the specific type of the second valid gesture. In this way, the power consumption when capturing images in the first mode can be further reduced.

[0031] In combination with the first aspect, in some implementations of the first aspect, controlling the camera to operate in the first mode includes: in response to an operation of turning on the gesture perception function, controlling the camera to operate in the first mode.

[0032] The above scheme defines the starting conditions for the camera to operate in the first mode, that is, after the gesture sensing function is turned on, the camera is controlled to operate in the first mode. In other words, when the gesture sensing function is not turned on, the camera can be turned off to reduce power consumption.

[0033] In combination with the first aspect, in certain implementations of the first aspect, before controlling the camera to operate in the first mode, the method also includes: detecting that the state of the electronic device satisfies a first condition, the first condition including at least one of the following: the screen state of the electronic device is in a bright screen state; the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0034] This solution further limits the conditions under which the device's camera must be activated. Besides enabling gesture sensing, a first condition is added to prevent the camera from continuously capturing the first image when not necessary, thus reducing device power consumption. "Unnecessary" can be understood as any scenario where the camera cannot capture gestures.

[0035] In combination with the first aspect, in some implementations of the first aspect, the electronic device is a foldable device, and the foldable device includes a first screen and a second screen located on the front and back sides of the electronic device respectively in the unfolded state, wherein the first screen is a folding screen, and the camera includes a first physical camera and a second physical camera, and the first physical camera and the second physical camera are logical front cameras corresponding to the first screen and the second screen, respectively; controlling the camera to operate in a first mode includes: when it is detected that the first screen is in a bright screen state and the electronic device is in an unfolded state or a hovering state, controlling the first physical camera to operate in the first mode; or, when it is detected that the second screen is in a bright screen state and the electronic device is in a folded state, controlling the second physical camera to operate in the first mode.

[0036] This solution can be applied to foldable devices. If the device is in a folded or unfolded state and the corresponding screen (inner screen or outer screen) is in a bright screen state, the camera on the bright screen can be controlled to turn on to detect whether there is a gesture within the camera range, thereby realizing gesture perception for foldable devices.

[0037] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: when the first physical camera is operating in the first mode, upon detecting that the electronic device switches from an unfolded state or a hovering state to a folded state, controlling the second physical camera to operate in the first mode and turning off the first physical camera; when the second physical camera is operating in the first mode, upon detecting that the electronic device switches from a folded state to an unfolded state or a hovering state, controlling the first physical camera to operate in the first mode and turning off the second physical camera.

[0038] In this solution, when other conditions remain unchanged, if the user changes the physical state of the device screen, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the first image can be continuously captured by switching the camera so that the device screen can still realize the functions of intelligently sensing air gestures and controlling the device in the new physical state.

[0039] In a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the electronic device implements the steps of the method described in any one of the above-mentioned first aspects.

[0040] In a third aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0041] In a fourth aspect, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes any one of the methods in the first aspect.

[0042] In a fifth aspect, a chip system is provided, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the method described in any one of the first aspects above.

[0043] The chip system may be a single chip or a chip module composed of multiple chips.

[0044] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a gesture perception method 100 provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a gesture perception method 200 provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a gesture perception method 300 provided in an embodiment of the present application;

[0048] Figure 4 A flowchart of a specific implementation of the gesture perception method provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of an interface provided in an embodiment of the present application;

[0050] Figure 6 A schematic structural diagram of a foldable device provided in an embodiment of the present application;

[0051] Figures 7 to 9 A schematic diagram of a preset dynamic gesture provided in an embodiment of the present application;

[0052] Figure 10 A schematic diagram of an interface for an air screenshot scenario applied in an embodiment of the present application is shown;

[0053] Figure 11 A schematic diagram of an interface for an airborne audio and video control scenario applied in an embodiment of the present application is shown;

[0054] Figure 12 A schematic diagram of an interface for zooming out a picture in an airborne manner according to an embodiment of the present application is shown;

[0055] Figure 13 A schematic diagram of an interface for an air-swiping upward screen scenario applied in an embodiment of the present application is shown;

[0056] Figure 14 A schematic diagram of an interface for a screen sliding downwards in the air, as applied in an embodiment of the present application, is shown;

[0057] Figure 15 A schematic diagram of a dynamic frame adjustment provided in an embodiment of the present application;

[0058] Figure 16 A schematic structural diagram of an electronic device 500 provided in an embodiment of the present application;

[0059] Figure 17 A schematic structural diagram of another electronic device provided in an embodiment of the present application;

[0060] Figure 18 Schematic diagram of the structure of a system on chip (SoC) involved in an embodiment of the present application;

[0061] Figure 19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0062] Figure 20 A flowchart of a gesture perception method provided in an embodiment of the present application;

[0063] Figure 21 A schematic diagram of a hand node provided in an embodiment of the present application;

[0064] Figure 22 A flowchart of a gesture perception method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0066] In order to facilitate understanding of the solutions provided by the embodiments of the present application, some of the terms involved in the present application are first explained.

[0067] Frame rate refers to the number of images a camera captures or transmits in one second, typically expressed in fps (frames per second). In an embodiment of the present application, the camera captures images at a first frame rate in a first mode and at a second frame rate in a second mode, where the first frame rate is less than the second frame rate. In some embodiments, the camera captures images at a third frame rate in a third mode, where the third frame rate is greater than the second frame rate.

[0068] Resolution, also known as image resolution, refers to the amount of information stored in an image, which is the number of pixels per inch in the image. The units of resolution include: dpi (dots per inch), ppi (pixels per inch), etc.

[0069] A lightweight neural network is a lighter model that has performance comparable to heavier models, thereby achieving a hardware-friendly neural network. The weight here usually refers to the scale or number of parameters of the model. Commonly used lightweight neural network technologies include: distillation, pruning, quantization, weight sharing, low-rank decomposition, lightweight attention module, dynamic network architecture / training method, lighter network architecture design, etc., which are not limited in the embodiments of this application.

[0070] Background subtraction is a widely used method in motion target detection technology. Its basic concept is similar to the inter-frame difference method, which uses the difference operation of different images to extract the target area. Unlike the inter-frame difference method, background subtraction does not subtract the current frame image from the adjacent frame image, but subtracts the current frame image from a continuously updated background model to extract the moving target in the difference image. In an embodiment of the present application, the moving target can be the user's hand. Based on the background subtraction operation, the changes in the user's gestures in consecutive image frames are detected to determine the gesture category.

[0071] Compared to traditional contact-based human-computer interaction, handheld gestures allow users to operate with their bare hands, providing a contactless human-computer interaction method that eliminates the inconvenience of gesture interaction. Handheld gestures include pointing, waving, clenching a fist, and rotating the palm. Users can naturally express their interaction intentions through finger, wrist, and arm movements, offering a wide interaction space, greater flexibility, and a better interactive experience.

[0072] In some scenarios, after opening the camera app, users can use air gestures, such as spreading their palms, to control the camera to take photos or record videos. However, as user needs continue to increase, the scenarios for using air gestures are becoming more and more numerous. For example, in one possible scenario, the user is wearing gloves or has dirty fingers and does not want to touch the phone screen, but still needs to control the phone, such as watching short videos while eating crayfish. In another possible scenario, the user is driving a vehicle and is slightly away from the phone screen, so they need to access other applications, such as when they drive near the office and need to clock in for work.

[0073] To meet users' needs for remote control in various scenarios, devices must continuously recognize gestures, which places high demands on the device's camera and processor. Furthermore, the success rate of gesture recognition in related technologies needs to be improved. It's common for users to perform gestures but the electronic device cannot accurately recognize them, resulting in a poor user experience.

[0074] Therefore, a gesture sensing solution is urgently needed to meet the needs of gesture sensing functions while reducing device power consumption. Figures 1 to 3 The gesture perception method provided in the embodiments of the present application is introduced.

[0075] It should be noted that the gesture perception method provided in the embodiment of the present application is applied to an electronic device including a camera, which can capture gestures made by a user through the camera and perform corresponding preset operations according to the gestures made by the user.

[0076] In the embodiments of the present application, unless otherwise specified, the camera of an electronic device refers to the logical front camera of the electronic device. The logical front camera here is a concept relative to the physical camera. A physical camera refers to a physical camera installed on an electronic device or external to an electronic device, while a logical camera is a conceptual camera defined for different usage scenarios. When a user uses an application with image acquisition or video acquisition functions on an electronic device, the shooting direction of the physical camera is different when capturing images or videos. The logical camera can be divided into a logical front camera (for example, a camera with a shooting direction facing outward from the display screen currently used by the user in the electronic device, i.e., a front shooting angle of view) and a logical rear camera (for example, a camera with a shooting direction facing inward from the display screen currently used by the user in the electronic device, i.e., a rear shooting angle of view). It can be understood that in other embodiments, the logical camera may also include other types of logical cameras, such as a logical depth camera for capturing three-dimensional images determined according to the function of the camera, which is not limited here.

[0077] The electronic devices described in the embodiments of this application may also be referred to as terminal devices, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic devices may be mobile phones with display screens, smart televisions, wearable devices, tablet computers (Pads), computers with wireless transceiver capabilities, and other devices. This application does not limit the specific types of electronic devices. For convenience, the figures of this application illustrate the electronic device as a mobile phone, but this does not constitute a limitation on the embodiments of this application.

[0078] Figure 1 Schematic diagram of a gesture perception method 100 provided in an embodiment of the present application. Figure 1 As shown, the electronic device continuously captures images through a camera and performs image detection on the captured images. The image detection here refers to gesture category detection, that is, detecting whether there is a valid gesture in the image, and the specific category of the valid gesture when there is a valid gesture. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. Each preset dynamic gesture consists of one or more continuous gestures, and the starting gesture refers to the first gesture of these multiple continuous gestures.

[0079] If a valid gesture is detected in two consecutive frames of images, the detection results of the two frames are used to determine whether to execute the subsequent gesture perception process. Figure 1In the example shown, the electronic device captures a first image and a second image through a camera. If the detection result shows that the category of the valid gesture in the first image is the same as the category of the valid gesture in the second image, for example, both of the valid gestures are the starting hand postures of the target dynamic gesture in the preset dynamic gestures, then it is determined to execute the gesture perception process, that is, continue to detect other gestures in the target dynamic gesture. After determining that the complete target dynamic gesture is detected, the electronic device can execute the preset operation corresponding to the target dynamic gesture, such as swiping up or down the interface, adjusting the screen brightness, adjusting the volume, adjusting the vibration intensity, calling a specific application, etc.

[0080] In the above solution, the electronic device can collect and recognize the user's gestures in real time by continuously performing image acquisition and image detection operations. In this process, when the user makes the starting gesture of the target dynamic gesture, the electronic device determines that the starting gesture is detected through a double-frame confirmation mechanism, that is, the starting gesture is detected in both frames before continuing to perform the subsequent gesture perception process. This can reduce redundant detection for certain scenarios, such as the scenario where the user accidentally passes his hand across the camera, thereby reducing power consumption. At the same time, in this solution, only two frames of images ( Figure 1 The first image and the second image in the image are used to trigger the subsequent gesture perception process. Since the time interval between collecting two frames of images is usually short (such as 200ms), even if the user's gesture operation is relatively fast, it can usually be captured, which improves the efficiency and success rate of gesture detection.

[0081] It should be noted that in Figure 1 In the corresponding solution, in order to more accurately identify the gestures of the user's operation, the electronic device needs to capture high-resolution images (greater than or equal to a preset value) through the camera, such as capturing images with a size of 320×240, so as to accurately detect the specific categories of valid gestures in the first image and the second image. In order to meet the needs of users for airborne operations in different scenarios, the electronic device needs to continuously perform gesture recognition, and in order to continuously perform gesture recognition, it is necessary to continuously capture higher-resolution images, which will cause the electronic device to generate higher power consumption.

[0082] In view of this, Figure 2 FIG. 2 shows a schematic diagram of another gesture perception method 200 provided in an embodiment of the present application. Figure 2As shown, the electronic device controls the camera to operate in a first mode, continuously captures images in the first mode, and performs image detection on the captured images. Image detection here refers to valid gesture detection, that is, detecting whether there is a valid gesture in the image, wherein the valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. The first mode is a low-resolution mode for capturing low-resolution images. For example, the first mode is used to capture images of a first resolution, where the first resolution is less than or equal to a preset value. By capturing images in the first mode, the power consumption of the electronic device can be reduced.

[0083] When the electronic device detects a valid gesture in a low-resolution image captured in a first mode, the electronic device controls the camera to switch from the first mode to a second mode, and then determines whether to trigger a gesture sensing process based on two high-resolution images captured in the second mode. The second mode is a high-resolution mode for capturing high-resolution images, for example, the second mode is for capturing images at a second resolution greater than a preset value.

[0084] For example, in Figure 2 In the example shown, the electronic device obtains a frame of first image captured by the camera in the first mode, and when a valid gesture is detected from the first image, controls the camera to switch from the first mode to the second mode, and obtains a frame of second image and a frame of third image captured by the camera in the second mode.

[0085] The electronic device performs image detection on the second image and the third image. The detection here refers to gesture category recognition, that is, detecting whether there is a valid gesture in the second image and the third image, and the specific category of the valid gesture when there is a valid gesture. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. If the detection result indicates that the category of the valid gesture in the second image is the same as the category of the valid gesture in the third image, for example, both of the two valid gestures are the starting gestures of the target dynamic gesture in the preset dynamic gestures, then it is determined to execute the gesture perception process, that is, continue to detect other gestures in the target dynamic gesture. After determining that the complete target dynamic gesture is detected, the electronic device can execute the preset operation corresponding to the target dynamic gesture, such as swiping up or down the interface, adjusting the screen brightness, adjusting the volume, adjusting the vibration intensity, calling a specific application, etc.

[0086] In the above solution, the camera is set to two modes: a first mode and a second mode, representing a low-resolution mode and a high-resolution mode, respectively. In general scenarios, the electronic device controls the camera to continuously capture low-resolution (i.e., first-resolution) images in the first mode and performs detection on the captured images. During this process, if a valid gesture is detected, the camera switches to the second mode. Because the resolution of images captured in the first mode is relatively low, it is difficult to accurately detect the category of valid gestures in the images. Therefore, by switching the camera to the second mode, a high-resolution (i.e., second-resolution) image captured by the camera in the second mode can be obtained. In this solution, the electronic device still uses a dual-frame confirmation mechanism to determine whether the starting gesture has been detected. That is, the starting gesture of the target dynamic gesture must be detected in both high-resolution image frames before continuing with the subsequent process. Since users typically use the air gesture function only occasionally, continuously capturing high-resolution images would result in a waste of resources. Therefore, this solution significantly reduces device power consumption by capturing low-resolution images in general scenarios and only capturing high-resolution images when a valid gesture is detected. Compared to a solution that continuously captures high-resolution images, this solution can significantly reduce device power consumption.

[0087] From the above description, we can see that Figure 2 In the scheme shown, three frames of images (first image, second image, third image) are required to trigger the subsequent gesture perception process. If the user performs the gesture quickly, gesture recognition may not be completed. For example, it takes 300ms for an electronic device to capture three frames of images through a camera, but when the user makes a gesture, the speed is very fast and the dwell time of the initial gesture is less than 300ms. In this case, the initial gesture recognition will fail, and therefore the subsequent gesture perception process will not be triggered, resulting in the user performing the gesture, but the electronic device cannot perform the corresponding operation, affecting the user experience.

[0088] In view of this, Figure 3 FIG. 3 is a schematic diagram showing another gesture perception method 300 provided in an embodiment of the present application. Figure 3 As shown, the electronic device controls the camera to operate in the first mode, continuously captures images in the first mode, and performs image detection on the captured images. The image detection here refers to effective gesture recognition, that is, detecting whether there is a valid gesture in the image, wherein the effective gesture refers to any one of the starting gestures of one or more preset dynamic gestures.

[0089] When the electronic device detects a valid gesture in the low-resolution image acquired through the first mode, the electronic device controls the camera to switch from the first mode to the second mode. Figure 2The corresponding scheme is similar. The first mode is a low-resolution mode for capturing low-resolution images (e.g., a first resolution that is less than or equal to a preset value), and the second mode is a high-resolution mode for capturing high-resolution images (e.g., a second resolution that is greater than a preset value). The electronic device then determines whether to trigger the gesture perception process based on the high-resolution image captured in the second mode and the low-resolution image captured in the first mode.

[0090] For example, in Figure 3 In the example shown, the electronic device obtains a frame of first image captured by the camera in the first mode, and when a valid gesture is detected from the first image, controls the camera to switch from the first mode to the second mode, and obtains a frame of second image captured by the camera in the second mode.

[0091] The electronic device performs image detection on the second image. Image detection here refers to gesture category detection, that is, detecting whether there is a valid gesture in the second image, and the specific category of the valid gesture when there is a valid gesture. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. If the detection result indicates that there is a valid gesture in the second image, the first image is interpolated to obtain a high-resolution image, which is recorded as the third image. The resolution of the third image is the same as that of the second image, or the resolution of the third image is greater than a preset value. Then, image detection is performed on the third image. Image detection here refers to gesture category detection, that is, detecting whether there is a valid gesture in the third image, and the specific category of the valid gesture when there is a valid gesture.

[0092] If the detection result indicates that the category of the valid gesture in the second image is the same as the category of the valid gesture in the third image, for example, both valid gestures are the starting gestures of a target dynamic gesture in the preset dynamic gestures, then the gesture perception process is determined to be executed, that is, other gestures in the target dynamic gesture are detected. After determining that the complete target dynamic gesture has been detected, the electronic device can perform the preset operation corresponding to the target dynamic gesture, such as swiping up or down the interface, adjusting the screen brightness, adjusting the volume, adjusting the vibration intensity, calling a specific application, etc.

[0093] In the above solution, two modes are set for the camera, namely the first mode and the second mode, which represent a low-resolution mode and a high-resolution mode respectively. In general scenarios, the electronic device controls the camera to continuously capture low-resolution (i.e., first-resolution) images through the first mode, and performs image detection on the captured images. In this process, if a valid gesture is detected, the camera is switched to the second mode. Because the resolution of the image captured by the first mode is relatively low, it is difficult to accurately detect the category of the valid gesture in the image. Therefore, by switching the camera to the second mode, a high-resolution (i.e., second-resolution) image captured by the camera through the second mode can be obtained. Since users usually only use the air gesture function for a few times, if high-resolution images are always captured, it will cause a waste of resources. Therefore, this solution captures low-resolution images in general scenarios, and only captures high-resolution images when a valid gesture is detected. Compared with the solution of constantly capturing high-resolution images, the power consumption of the device can be greatly reduced.

[0094] In this solution, the electronic device still uses a dual-frame confirmation mechanism to determine whether the starting gesture has been detected, that is, the starting gesture of the target dynamic gesture is detected in both frames of high-resolution images before continuing to execute the subsequent process. Compared with method 200, method 300 only needs to collect two frames of images to trigger the subsequent gesture perception process, because in method 300, the first image of the first resolution (i.e., low resolution) is interpolated to obtain the third image (high-resolution image), so only one frame of high-resolution image (i.e., second image) needs to be collected to obtain two frames of high-resolution images. This solution can not only maintain the gesture perception function of the electronic device at a lower power consumption cost, but also improve the efficiency of gesture perception. Even if the user's gesture operation is relatively fast, a high detection success rate can be guaranteed, that is, a balance between power consumption and effect is achieved.

[0095] In order to further understand the gesture perception method provided by the embodiment of the present application, Figure 4 The method 400 in the embodiment of the present invention exemplifies the specific implementation process of the gesture perception method. It can be understood that the method 400 is a Figure 3 Further explanation of method 300 is provided in FIG.

[0096] S401, controlling the camera to operate in a first mode.

[0097] Exemplarily, the electronic device controls a camera to operate in a first mode, wherein the first mode is used to capture images at a first resolution. For example, the first resolution is lower than a preset value, i.e., the first mode is a low-resolution mode. The electronic device can capture images in the first mode through the camera at a lower power consumption cost.

[0098] Therefore, the electronic device controlling the camera to operate in the first mode means that the electronic device controls the camera to continuously capture images with the first resolution.

[0099] This application does not limit the conditions for triggering the electronic device to operate in the first mode. As an example, in response to an operation to enable the gesture sensing function, the electronic device controls the camera to operate in the first mode. In other words, after the gesture sensing function is enabled, the electronic device operates in the first mode.

[0100] It can be understood that the camera running in the first mode may mean that the camera runs in the background in the first mode. For example, the electronic device currently opens a third-party reference, and after the user turns on the gesture perception function, the electronic device controls the camera to run in the first mode.

[0101] The gesture perception function described in the embodiments of this application refers to the function of the electronic device to perceive and recognize the gestures operated by the user, and to perform preset operations according to the recognition results. The gesture perception function can also be called the air gesture function, gesture recognition function, air operation function, etc., which is not limited in this application. For example, the electronic device is pre-configured with a variety of preset dynamic gestures, each of which consists of one or more gestures (for a description of the preset dynamic gestures, please refer to the subsequent Figures 8 to 10 The corresponding description will not be repeated here), and each preset dynamic gesture corresponds to one or more preset operations (such as sliding up or down the interface, adjusting the screen brightness, adjusting the volume, adjusting the vibration intensity, calling a specific application, etc.). The user can choose to turn the gesture perception function on or off. After the gesture perception function is turned on, the electronic device performs gesture detection in real time. When the target dynamic gesture in the preset dynamic gesture is detected, the electronic device executes the preset operation corresponding to the target dynamic gesture. For the specific detection process, please refer to the description of S402 to S413 in this embodiment, which will not be repeated here.

[0102] It is understandable that the operation of turning on the gesture perception function can be a click operation of the user in the system setting interface, or a voice operation, etc., which is not limited in the embodiment of the present application.

[0103] For example, Figure 5 This is a schematic diagram of an interface provided in an embodiment of the present application. Figure 5 As shown in (a) and (b) of FIG, a user can choose to turn on or off the gesture sensing function in the settings interface of the system application. In response to the operation of turning on the gesture sensing function, the electronic device controls the camera to operate in the first mode and continuously captures images of the first resolution in the first mode through the camera.

[0104] In some embodiments, the user can also choose to perceive one or more gestures in the settings interface. Each gesture has a preset corresponding operation, for example, gesture A is used to slide down the interface / flip down the page, gesture B is used to slide up the interface / flip up the page, gesture C is used for screenshots, and gesture D is used to pause / resume playback.

[0105] For example, Figure 5 As shown in (c) and (d), the user can choose to turn on the perception of gesture A. If the electronic device recognizes gesture A, the operation corresponding to gesture A is performed: sliding down the interface / flipping down the page; the user can also choose to turn on the perception of gesture E. If the electronic device recognizes gesture E, the operation corresponding to gesture E is performed: calling the preset application.

[0106] In some embodiments, the user can customize the application corresponding to the gesture E, such as Figure 5 As shown in (d), (e), and (f), the user selects gesture E on the gesture E setting interface to call up application A. For example, gesture E is five fingers spread apart, and application A is a clock-in app for an office application. When the user drives near the office, they can use air gesture E to quickly launch the clock-in app for the office application and complete their work clock-in, realizing gesture perception and improving the user experience.

[0107] In some embodiments, the condition for the electronic device to control the camera to operate in the first mode also includes a first condition, and the first condition includes at least one of the following: the screen state of the electronic device is the bright screen state; the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0108] In this embodiment, if the electronic device has the gesture sensing function enabled and the first condition described above is met, the electronic device's camera is triggered to continuously capture the first image. By adding the first condition, the electronic device's camera is prevented from continuously capturing the first image when not necessary, further reducing device power consumption.

[0109] Based on the aforementioned embodiments, various triggering methods for controlling the camera to operate in the first mode (or controlling the camera to continuously capture images with the first resolution in the first mode) are described in detail below.

[0110] In a possible implementation, if the electronic device has enabled the gesture sensing function, the electronic device is triggered to control the camera to operate in the first mode.

[0111] In one possible implementation, if the electronic device has a gesture sensing function enabled, the screen status of the electronic device is detected. If the screen status of the electronic device is in the bright screen state, the electronic device is triggered to control the camera to operate in the first mode. The interfaces displayed in the bright screen state of the electronic device include, for example, the lock screen interface, the home interface, and the third-party application interface.

[0112] In a possible implementation, if the electronic device has a gesture sensing function enabled, it detects whether the electronic device is unlocked. If the electronic device is unlocked, it triggers the electronic device to control the camera to operate in the first mode.

[0113] In one possible implementation, if the electronic device has enabled the gesture sensing function, it detects whether the light signal emitted by the proximity light sensor of the electronic device is blocked. If it is determined that the light signal emitted by the proximity light sensor is not blocked, it triggers the electronic device to control the camera to operate in the first mode.

[0114] As an example, if the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the proximity light sensor does not receive the reflected signal, it can be determined that the light signal emitted by the proximity light sensor is not blocked.

[0115] It is understandable that if the user makes or receives a call through the receiver, or the screen of the electronic device is turned upside down, or the electronic device is in a handbag or pocket, the light signal emitted by the proximity light sensor of the electronic device will be blocked, and the image captured by the front camera will not be able to detect gestures. At this time, the camera can be stopped from continuously capturing images to reduce device power consumption.

[0116] In one possible implementation, if the electronic device has the gesture sensing function enabled, the electronic device detects whether the detection data of the ambient light sensor of the electronic device is greater than a third threshold. If the detection data of the ambient light sensor is determined to be greater than the third threshold, the electronic device is triggered to control the camera to operate in the first mode. The detection data primarily refers to the ambient light brightness. It should be understood that if the detection data of the ambient light sensor of the electronic device is greater than the third threshold, it indicates that the electronic device is not in a dark environment, such as if the electronic device is in a pocket or it is currently night time.

[0117] In one possible implementation, if the electronic device has a gesture sensing function enabled, the screen orientation of the electronic device is detected. If the screen of the electronic device is facing a preset direction, the electronic device is triggered to control the camera to operate in a first mode. In this implementation, the preset direction includes the direction in which a user is holding the electronic device to browse interface content, the direction in which the electronic device is placed on a desktop to browse interface content, etc. This direction can be determined by detecting posture data of the electronic device, where the posture data includes pitch angle, yaw angle, and roll angle.

[0118] In one possible implementation, if the electronic device has turned on the gesture perception function, it detects whether the electronic device is running a preset application. If it is determined that the electronic device is running the preset application, the electronic device is triggered to control the camera to run in the first mode. For example, the preset application is a short video application. If the electronic device detects that the user is watching a short video, the electronic device camera is triggered to continuously capture the first image to detect whether the user is using the preset air gesture to meet the user's need to browse short videos through the air. For another example, the preset application is a navigation application. If the electronic device detects that the user is using the driving navigation of the navigation application, the electronic device camera is triggered to continuously capture the first image to detect whether the user is using the preset air gesture to meet the user's need to listen to traffic reports through the air. The embodiment of the present application does not limit the preset application. In addition, it should be noted that the electronic device running the preset application includes displaying the interface of the preset application on the current electronic device screen, or the preset application runs in the background.

[0119] In one possible implementation, if the electronic device has turned on the gesture perception function and is determined to meet at least two of the following conditions: the screen state of the electronic device is on, the electronic device is unlocked, the light signal emitted by the proximity light sensor of the electronic device is not blocked, and the electronic device runs a preset application, then the electronic device is triggered to control the camera to operate in the first mode.

[0120] In another embodiment, if the electronic device has a gesture sensing function enabled and detects that the electronic device is in motion, the electronic device's camera may be triggered to capture an image at a first resolution. Conversely, if the electronic device has a gesture sensing function enabled and detects that the electronic device is stationary, the camera may be controlled to shut down to reduce device power consumption. The mobile state of the electronic device includes, for example, the following scenarios: a user using the electronic device for navigation while driving a vehicle, or a user using the electronic device for navigation while riding a bicycle.

[0121] In another embodiment, if the electronic device has a gesture sensing function enabled and detects that the electronic device is stationary, the gesture camera may be triggered to capture an image at a first resolution to sense the user's potential need for remote control at any time. The stationary state of the electronic device includes, for example, placing the electronic device on a desktop.

[0122] In one possible implementation, the electronic device in the embodiment of the present application may be a foldable device, such as a mobile phone with a foldable screen. The foldable device has multiple screens and multiple cameras, and which camera is controlled to operate in the first mode can be determined based on the specific physical form.

[0123] The following is combined with Figure 6 A detailed description of gesture sensing methods for foldable devices is provided.

[0124] For example, Figure 6 This is a schematic diagram of the structure of a foldable device provided in an embodiment of the present application. Figure 6 As shown, the screen of the foldable device includes a first screen and a second screen. The first screen is a foldable screen that folds inward. Therefore, when the electronic device is in the folded state, the first screen is the inner screen of the electronic device, and the second screen is the outer screen of the electronic device. The virtual axis along which the folding screens lie is the folding edge, along which the first screen can fold inward. The angle α in the figure represents the folding angle of the first screen.

[0125] The foldable device includes three physical states: unfolded state, hovering state and folded state. The unfolded state refers to the state where the first screen is unfolded to the maximum angle, such as Figure 6 The folded state refers to the state after the first screen is folded, as shown in (a) and (b) in FIG. Figure 6 As shown in (d) in . The hover state refers to a state between the expanded state and the collapsed state, such as Figure 6 As shown in (c) in .

[0126] The first screen includes a first physical camera. When the foldable device is in the unfolded state, the first physical camera can serve as the logical front-facing camera of the foldable device. The second screen includes a second physical camera. When the foldable device is in the folded state or the hovering state, the second physical camera can serve as the logical front-facing camera of the foldable device. The back panel includes a third physical camera. When the foldable device is in the unfolded state or the hovering state, the third physical camera can serve as the logical rear-facing camera of the foldable device.

[0127] As an example, a foldable device includes a first screen and a second screen located on the front and back sides of the electronic device respectively in an unfolded state, wherein the first screen is a folding screen, and the camera includes a first physical camera and a second physical camera, which are logical front cameras corresponding to the first screen and the second screen respectively.

[0128] As an example, if it is detected that the second screen of the electronic device is in a bright screen state and the electronic device is in a folded state, the second physical camera is controlled to operate in the first mode. As an example, if it is detected that the second screen of the electronic device is in a bright screen state and the electronic device is in an unfolded state or a hovering state (referring to a state of the electronic device between the folded state and the unfolded state), the second physical camera is controlled to operate in the first mode. Based on these two examples, in some embodiments, before controlling the second physical camera to operate in the first mode, it also includes: detecting that the state of the electronic device satisfies at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the second screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the second screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0129] As an example, if it is detected that the first screen of the electronic device is in a bright screen state, and the electronic device is in an unfolded state or a hovering state, the first physical camera is controlled to operate in the first mode. Before controlling the first physical camera to operate in the first mode, it also includes: detecting that the state of the electronic device satisfies at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device (on the first screen) and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the first screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0130] Based on the above examples, before controlling the first physical camera to operate in the first mode, or controlling the second physical camera to operate in the first mode, it also includes: detecting that the state of the electronic device meets the second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the first screen or the second screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0131] As an example, when the second physical camera is operating in the first mode, if it is detected that the electronic device switches from a folded state to an unfolded state or a hovering state, the electronic device controls the first physical camera to operate in the first mode and turns off the second physical camera.

[0132] As an example, when the first physical camera is operating in the first mode, if it is detected that the electronic device switches from an unfolded state or a hovering state to a folded state, the electronic device controls the first physical camera to turn off and controls the second physical camera to operate in the first mode; or, the electronic device controls the first physical camera to turn off.

[0133] It should be noted that other forms of foldable devices can refer to the gesture perception method. Figure 6 The implementation principle and technical effect of the foldable device shown are similar, and the embodiments of the present application do not impose any restrictions on the structural style of the foldable device.

[0134] Optionally, the first mode is further configured to capture images at a first frame rate, wherein the first frame rate is less than or equal to a preset frame rate. This solution can further reduce power consumption of the electronic device when the camera captures images in the first mode.

[0135] S402: Acquire a first image frame captured by a camera in a first mode.

[0136] S403: Detect the first image to obtain a first detection result.

[0137] Exemplarily, after the electronic device controls the camera to operate in the first mode, the camera continuously captures images of the first resolution in the first mode, and performs valid gesture detection on the captured images, that is, detects whether there is a valid gesture in the image.

[0138] The present application embodiment is described using a first image captured by an electronic device through a camera as an example, where the first image is a frame. After acquiring the first image, the electronic device detects the first image to obtain a first detection result, which indicates whether a first valid gesture exists in the first image. The first valid gesture is any one of the starting gestures of one or more preset dynamic gestures.

[0139] In an embodiment of the present application, the preset dynamic gestures are dynamic gestures configured by default or customized by the user, and each preset dynamic gesture corresponds to one or more preset operations, such as swiping up or down the interface, adjusting the screen brightness, adjusting the volume, adjusting the vibration intensity, calling up a specific application, etc.

[0140] The target dynamic gesture in the embodiment of the present application specifically refers to the preset dynamic gesture detected by the electronic device, that is, the target dynamic gesture is one of the preset dynamic gestures, so the description of the preset dynamic gesture in the embodiment of the present application also applies to the target dynamic gesture.

[0141] Each preset dynamic gesture consists of one gesture or multiple continuous gestures. The starting gesture of the preset dynamic gesture refers to the first gesture among the one or more gestures, the ending gesture of the preset dynamic gesture is the last gesture among the one or more gestures, and the intermediate gesture of the preset dynamic gesture refers to the gesture between the starting gesture and the ending gesture (if any).

[0142] For example, Figures 7 to 9 The following is a schematic diagram of a preset dynamic gesture provided in an embodiment of the present application. The preset dynamic gesture and its application scenarios are described in detail with reference to the accompanying drawings.

[0143] In one possible implementation, the preset dynamic gesture includes only a first gesture, i.e., the number of gestures in the preset dynamic gesture is one. In this case, the starting gesture and the ending gesture of the preset dynamic gesture both refer to the first gesture, and the preset dynamic gesture has no intermediate gestures. An example is provided below to illustrate this.

[0144] For example, in Figure 7 In the example shown in (a), the gesture action corresponding to the first gesture is five fingers spreading out. When the electronic device recognizes the five fingers spreading out gesture, it can perform the preset operation corresponding to the gesture. For example, if the current screen state of the electronic device is the screen-on state and the interface is any interface, after the electronic device recognizes the five fingers spreading out gesture, the current interface is captured. For another example, if the current interface of the electronic device is the selfie interface, after the electronic device recognizes the user making the five fingers spreading out action, it can control the camera to capture an image, or start / pause video recording. For another example, if the current screen state of the electronic device is the screen-off state, after the electronic device recognizes the five fingers spreading out gesture, it turns on the screen to check the time / message, or opens the office application and performs the work clocking-in task. For another example, if the current interface of the electronic device is the navigation interface, after recognizing the five fingers spreading out gesture, it can automatically broadcast the real-time traffic conditions.

[0145] For example, Figure 7 The gesture shown in (b) is extending the index finger. For example, if the electronic device is currently viewing a short video, upon recognizing the gesture, the electronic device may like or save the short video. For another example, if the electronic device is currently viewing a selfie, upon recognizing the gesture, the electronic device may start a 3-second countdown to capture the video.

[0146] In a possible implementation, the preset dynamic gesture includes a first gesture and a second gesture, the first gesture is a starting gesture of the gesture change, and the second gesture is a terminating gesture of the gesture change.

[0147] For example, in Figure 8In the example shown in (a), the gesture action corresponding to the first gesture is spreading the palm, and the gesture corresponding to the second gesture is making a fist. That is, the preset dynamic gesture in this example includes a dynamic gesture process of spreading the palm and changing into a fist with five fingers, that is, the starting gesture is the palm facing the front camera of the electronic device, and the ending gesture is making a fist. For example, if the screen state of the electronic device is on and the interface is any interface, after the electronic device recognizes two consecutive gestures as the first gesture and the second gesture, it executes the preset operation corresponding to the preset dynamic gesture. As an example, the preset operation is, for example, a screenshot operation. Figure 10 The schematic diagram of the interface of the air screenshot scene of the embodiment of the present application is shown. In this scene, when the electronic device detects the user's gesture process from opening the palm to grasping the five fingers into a fist (such as Figure 10 (a) to Figure 10 (b) in the figure), then take a screenshot on the current interface and obtain a screenshot (as shown in Figure 10 (as shown in (c) in the figure).

[0148] For example, in Figure 8 In the example shown in (b), the gesture action corresponding to the first gesture is spreading the palm, and the gesture corresponding to the second gesture is also spreading the palm, but the distance between the first gesture and the electronic device is greater than the distance between the second gesture and the electronic device, that is, the preset dynamic gesture in this example is the action of pressing the palm forward facing the electronic device. The preset operation corresponding to the gesture action can be pause or continue playing. For example, if the current interface of the electronic device is a video / music playback interface and is currently in a playback state, the electronic device will pause the video / music after recognizing the gesture; if the current interface of the electronic device is a video / music playback interface and is currently in a paused state, the electronic device will continue to play the video / music after recognizing the gesture. Figure 11 The interface diagram of the airborne audio and video control scenario of the embodiment of the present application is shown. In this scenario, the display interface of the electronic device is currently playing a video screen. When the electronic device detects the user's operation of spreading the palm and pressing forward (ie, toward the electronic device) (such as Figure 11 (a) and Figure 11 (b) in the figure), the currently playing video is paused (as shown in Figure 11 (as shown in (c) in the figure).

[0149] For example, in Figure 8In the example shown in (c), the first gesture corresponds to a two-finger spread (opening the thumb and index finger, with the remaining fingers clenched into a fist); the second gesture corresponds to a two-finger pinch (pinch the thumb and index finger, with the remaining fingers clenched into a fist). In other words, the preset dynamic gesture in this example is a two-finger spread to pinch movement. For example, if the current electronic device's display interface displays an image, the electronic device will zoom out after recognizing the two-finger pinch gesture. Figure 12 The following is a schematic diagram of the interface of the air zooming picture scene used in the embodiment of the present application. In this scene, the display interface of the electronic device is currently displaying an image of a puppy. When the electronic device detects the user's gesture process from opening two fingers to pinching two fingers (such as Figure 12 (a) and Figure 12 (b) in the figure), the puppy image currently displayed on the interface is zoomed out (as shown in Figure 12 (as shown in (c) in the figure).

[0150] For example, in Figure 8 In the example shown in (d), the first gesture corresponds to a pinching motion (pinch together, i.e., pinching the thumb and index finger together while the remaining fingers form a fist); the second gesture corresponds to an opening motion (pinch apart, i.e., opening the thumb and index finger together while the remaining fingers form a fist). In other words, the preset dynamic gesture in this example is a two-finger pinch-to-open motion. For example, if the current electronic device's display interface displays an image, the electronic device may zoom in on the image after recognizing the opening motion.

[0151] In a possible implementation, the target dynamic gesture includes a first gesture, a second gesture, and a third gesture, wherein the first gesture is a starting gesture of a gesture change, the second gesture is an intermediate gesture of the gesture change, and the third gesture is an ending gesture of the gesture change.

[0152] For example, Figure 9 The gesture shown in (a) is an upward swipe action, in which the first gesture is to face the back of the hand toward the front camera of the electronic device, the second gesture is to flatten the palm, and the third gesture is to face the palm toward the front camera of the electronic device, that is, the gesture is from bending downward to flattening the palm and then flipping the palm to face the camera. For example, if the electronic device currently displays a data stream interface, the electronic device will scroll down a fixed length after recognizing the gesture. For another example, if the electronic device currently displays a page, the electronic device will turn down a page after recognizing the gesture. For another example, if the electronic device currently plays a video, the electronic device will turn up the volume or brightness after recognizing the gesture. For another example, if the current interface of the electronic device is a short video interface, the electronic device will play the next short video after recognizing the gesture. Figure 13The schematic diagram of the interface of the upward sliding screen scenario applied in the embodiment of the present application is shown. In this scenario, the display interface of the electronic device is currently displaying an article. When the electronic device detects the user's gesture process from bending down to flattening the palm and then turning the palm to face the camera (such as Figure 13 (a) Figure 13 As shown in (b) and Figure 13 ), then slide the screen display upwards (as shown in (c) Figure 13 (as shown in (d) in the figure).

[0153] For example, Figure 9 The gesture shown in (b) is a sliding gesture, in which the first gesture is to face the palm of the hand toward the front camera of the electronic device, the second gesture is to flatten the palm of the hand, and the third gesture is to face the back of the hand toward the front camera of the electronic device, that is, the gesture is from the palm of the hand facing the front camera of the electronic device to the palm of the hand clasped perpendicular to the screen of the electronic device, and then to the palm of the hand bent with the back of the hand facing the front camera. For example, if the electronic device currently displays a data stream interface, the electronic device will scroll up a fixed length after recognizing the gesture. For another example, if the electronic device currently displays a page, the electronic device will turn up a page after recognizing the gesture. For another example, if the electronic device currently plays a video, the electronic device will lower the volume or brightness after recognizing the gesture. For another example, if the current interface of the electronic device is a short video interface, the electronic device will play the previous short video after recognizing the gesture.

[0154] Figure 14 The schematic diagram of the interface of the screen sliding downward in the air in the embodiment of the present application is shown. In this scenario, the display interface of the electronic device is currently playing a short video. When the electronic device detects that the user has made a gesture from the palm facing the front camera of the electronic device to the palm facing down perpendicular to the screen of the electronic device (such as Figure 14 (a) and Figure 14 (b) in the figure), and then to the gesture process of bending the palm of the hand and facing the front camera (as shown in Figure 14 (c) in the figure), then switch to the previous short video to play (as shown in Figure 14 (as shown in (d) in the figure).

[0155] Based on the above embodiments, it can be seen that in different application scenarios, users can use preset air gestures to achieve diversified control of electronic devices. Application scenarios include, for example, the electronic device is in a black screen state, the user is watching a video or short video, the user is browsing the web or photo album, the user is driving and navigating, and the user is taking pictures or recording videos.

[0156] It should be noted that the embodiments of the present application do not limit the application scenarios of the gesture perception method. In addition to the application scenarios described in the above embodiments, it can also be applied to any other scenarios where electronic devices can be controlled through air gesture operations.

[0157] It should be noted that the present application does not limit the specific implementation method of detecting the first image to obtain the first detection result. In one possible example, the electronic device inputs the first image into the first model and outputs the first detection result, wherein the first model is a two-classification model, that is, the first detection result only displays two results, namely, the first image includes the first valid gesture, or the first image does not include the first valid gesture, and does not indicate the specific category of the first valid gesture. Compared with the multi-classification model, the power consumption caused by the binary classification model is lower, and since the electronic device needs to continuously capture images of the first resolution through the camera in the first mode and perform effective gesture detection on these first resolution images, the frequency of detection is very high. The present application detects images of the first resolution through a binary classification model, which can effectively reduce the power consumption of the electronic device. Exemplarily, the first model can be a neural network model.

[0158] S404: Determine whether a first valid gesture is detected.

[0159] Exemplarily, after acquiring the first image for detection to obtain a first detection result, it is determined whether a first valid gesture is detected from the first image according to the first detection result.

[0160] If a first valid gesture is detected from the first image, optionally, in S405, the electronic device caches the first image. Since the first image is an image of the first resolution (i.e., a low-resolution image), the cache space occupied by caching the first image is relatively small, which can save storage space.

[0161] It is understandable that in some application scenarios, the first image may be a frame of image shared by multiple services, that is, the first image is an image of the second resolution, rather than an image of the first resolution collected in the first mode. In order to make the solution provided in the embodiment of the present application applicable to this scenario, the first image can be downsampled so that the resolution of the first image is changed from the second resolution to the first resolution, so that the first model can be used to detect the downsampled first image. In this implementation method, the high-resolution first image can be directly cached without performing subsequent interpolation operations (for the interpolation operation, please refer to the subsequent description).

[0162] S406: Control the camera to switch from the first mode to the second mode.

[0163] For example, when a first valid gesture is detected from a first image, the electronic device controls the camera to switch from a first mode to a second mode. The second mode is used to capture images at a second resolution, where the first resolution is lower than the second resolution. For example, the first resolution can be set to 120×160, and the second resolution can be set to 320×240.

[0164] That is to say, the second resolution is high resolution, and the second mode is high resolution mode. When the electronic device detects the first valid gesture from the first image, it means that the user may be performing an air gesture within the camera's field of view to use the gesture recognition function. However, since the resolution of the first image (i.e., the first resolution) is relatively low (lower than the preset value), it is difficult to detect the specific category of the gesture performed by the user. Therefore, the electronic device controls the camera to switch to high resolution mode (i.e., the second mode) to capture a high-resolution image. Through the high-resolution image, the electronic device can more accurately detect the category of the gesture performed by the user and improve the success rate of the detection.

[0165] Optionally, the second mode is also used to capture images at a second frame rate, and the first frame rate is lower than the second frame rate. For example, the first frame rate can be set to 2fps, and the second frame rate can be set to 5fps. Through this solution, the frame rate of the camera when capturing high-resolution images can be increased, that is, the speed at which the camera captures high-resolution images per unit time can be increased. Therefore, when the user performs gestures quickly, the electronic device can also capture the user's gestures as much as possible, thereby improving the success rate of detection. In this solution, the switching of the camera between different working modes is closely related to the switching of the image frame rate. The following describes a solution for adjusting the frame rate of the camera capturing images.

[0166] For example, Figure 15 This is a schematic diagram of a dynamic frame adjustment provided by an embodiment of the present application. Figure 15 As shown in (a), in a possible frame adjustment scheme, the frame rate of the camera's image acquisition / transmission is adjusted by adjusting the clock frequency. Specifically, data transmission is based on the change of the clock, and data is usually acquired / transmitted at the rising or falling edge of the clock. If the frame rate of the camera's image acquisition needs to be doubled, the clock frequency can be doubled, and the corresponding data transmission speed will also be doubled; if the frame rate of the camera's image acquisition needs to be halved, the clock frequency can be halved, and the corresponding data transmission speed will also be halved. Figure 15 As shown in (b), one possible frame adjustment scheme adjusts the frame rate of the camera's image capture / transmission by adjusting the ratio of the active area to the blanking area. Specifically, the frame rate can be adjusted by adjusting the blanking area length. Increasing the blanking area length can reduce the frame rate, while decreasing it can increase it.

[0167] It should be noted that since the first frame adjustment scheme mentioned above requires shutting down the clock and powering it on again to modify the clock frequency, dynamic adjustment between frames cannot be achieved. Therefore, the embodiment of the present application adopts the second frame adjustment scheme mentioned above to achieve the purpose of dynamic frame adjustment.

[0168] S407: Acquire a second image frame captured by the camera in the second mode.

[0169] S408: Detect the second image to obtain a second detection result.

[0170] Exemplarily, after the electronic device controls the camera to switch from the first mode to the second mode, the camera captures images in the second mode and performs gesture category detection on the captured images, that is, detecting whether there is a valid gesture in the image, and the category of the valid gesture if there is a valid gesture.

[0171] The embodiments of the present application are described using a second image captured by an electronic device through a camera as an example, where the second image is a single frame. After acquiring the second image, the electronic device detects the second image to obtain a second detection result, which indicates whether a second valid gesture exists in the second image and, if so, the category of the second valid gesture. The second valid gesture is any one of the starting gestures of one or more preset dynamic gestures.

[0172] by Figure 8 and Figure 9 The dynamic gesture shown is an example of a preset dynamic gesture: there are 6 preset dynamic gestures, each of which corresponds to a starting gesture, for example, Figure 8 The starting gestures of the preset dynamic gestures shown in (a) and (b) are both gestures in which the palm is spread out and the palm faces the electronic device; Figure 8 The starting gesture of the preset dynamic gesture shown in (c) is a gesture with two fingers open and the remaining fingers clenched into a fist; Figure 8 The starting gesture of the preset dynamic gesture shown in (c) is a gesture of pinching two fingers together and making a fist with the remaining fingers; Figure 9 The preset dynamic gesture shown in (a) is a gesture in which the palm is spread out and the back of the hand faces the electronic device; Figure 9 The preset dynamic gesture shown in (b) is a gesture in which the palm is spread out and the palm faces the electronic device.

[0173] From this we can see that Figure 8 (a) Figure 8 (b) and Figure 9 The starting gestures of the preset dynamic gestures shown in (b) are the same, that is, the categories of their starting gestures are consistent. Figure 8 and Figure 9Among the 6 preset dynamic gestures shown, one includes 4 starting gestures.

[0174] Detecting the second image refers to detecting whether the second image includes any of the starting gestures of the preset dynamic gestures, and if so, the category of the starting gesture. Figure 8 and Figure 9 As an example, the preset dynamic gesture shown in FIG. 1 is used to detect whether the second image includes Figure 8 and Figure 9 Any of the four starting gestures shown, and if there is a starting gesture, the starting gesture is Figure 8 and Figure 9 Which of the four starting gestures shown is the starting gesture. That is, the category of the starting gesture in this application is used to indicate which starting gesture the starting gesture is. In specific implementation, a number can be set for each category of starting gesture, combined with Figure 8 and Figure 9 Take the four starting gestures shown as an example: Figure 8 (a) Figure 8 (b) and Figure 9 The starting gesture shown in (b) is recorded as category #1 (indicated by "00"), Figure 8 The starting gesture shown in (c) is recorded as category #2 (indicated by "01"), Figure 8 The starting gesture shown in (d) is recorded as category #3 (indicated by "10"), Figure 9 The starting gesture shown in (a) is recorded as category #4 (indicated by "11"). After the second image is detected, the category of the second valid gesture in the second image can be represented by a number. That is to say, the second detection result obtained by detecting the second image can be directly represented by the category number of the detected second valid gesture.

[0175] It should be noted that the present application does not limit the specific implementation method of detecting the second image to obtain the second detection result. In one possible example, the electronic device inputs the second image into the second model and outputs the second detection result, wherein the second model is a multi-classification model, that is, the second detection result can display multiple results, namely, the second image does not include the second valid gesture, or the category of the second valid gesture in the second image is category #1, or the category of the second valid gesture in the second image is category #2, or the category of the second valid gesture in the second image is category #3... Compared with the two-classification model, the multi-classification model can output the specific category of the detected valid gesture, so that the electronic device can determine which starting gesture the user performed based on the detection result, so as to execute the subsequent gesture perception process. Exemplarily, the second model can be a neural network model.

[0176] S409: Determine whether a second valid gesture is detected.

[0177] Exemplarily, after acquiring the second image for detection to obtain the second detection result, it is determined whether a second valid gesture is detected from the second image according to the first detection result.

[0178] If the electronic device detects a second valid gesture from the second image, steps S410 to S413 are triggered.

[0179] If the electronic device fails to detect the second valid gesture from the second image, in one possible implementation, the electronic device discards the cached first image and restarts gesture detection. For example, the electronic device controls the camera to switch to the first mode again and re-executes the solution provided in this application from step S401 (this example is recorded as Solution A); for another example, the electronic device continues to control the camera in the second mode and controls the camera to continuously capture images in the second mode. At this time, the solution executed by the electronic device is the same as that in the first mode. Figure 1 The method 100 described above is similar and will not be described in detail here (this example is referred to as Solution B). In another possible implementation, the electronic device retains the cached first image and returns to step S407 to reacquire a frame of the second image, and then triggers the execution of S408 and S409. If a second valid gesture is detected in the reacquired second image, the subsequent steps S410 to S413 are triggered. If the second valid gesture is still not detected, the electronic device discards the cached first image and restarts gesture detection (as described in Solution A or Solution B above).

[0180] In the above scheme, an invalid frame is allowed to exist between two valid frames. That is, in one implementation, if the electronic device obtains two consecutive frames of images through the camera, the first frame of the two frames is an image of the first resolution, and the second frame is an image of the second resolution, and valid gestures are detected in both frames, then the subsequent steps S410 to S413 are triggered. In another implementation, if the electronic device obtains three consecutive frames of images through the camera, the first frame of the three frames is an image of the first resolution, the second and third frames are images of the second resolution, and valid gestures are detected in the first and third frames, but no valid gesture is detected in the second frame, in this case, the subsequent steps S410 to S413 are still triggered, that is, the three consecutive frames are compatible with the situation where the sequence is <valid low resolution, invalid high resolution, valid high resolution>.

[0181] In summary, in the embodiment of the present application, the first image and the second image used to trigger the subsequent steps S410 to S413 are two adjacent frames of images, or, in the embodiment of the present application, there is one frame of image between the first image and the second image used to trigger the subsequent steps S410 to S413.

[0182] S410: Perform interpolation processing on the first image to obtain a third image.

[0183] Exemplarily, the electronic device interpolates the first image to obtain a third image, wherein the resolution of the third image is the same as that of the second image (i.e., the second resolution), or the resolution of the third image is greater than a preset value. In other words, the electronic device upward interpolates the low-resolution image (the first image at the first resolution) in which the valid gesture is detected to obtain a high-resolution image.

[0184] In one possible implementation, step S410 is triggered by step S409, that is, when the electronic device detects the second valid gesture from the second image, it obtains the first image from the cache, and then interpolates the first image to obtain the third image. This implementation is conducive to reducing power consumption. Specifically, after the electronic device detects the first valid gesture from the first image, it will cache the first image and control the camera to switch to the second mode to capture the second image. Since the first valid gesture detected in the first image may not be the gesture made by the user when performing an air gesture operation, for example, the first valid gesture may be a gesture made by the user when he accidentally swipes his hand across the camera, in this case, the second valid gesture is likely not to be detected in the second image. Therefore, in this implementation, S410 is triggered only when it is determined that the second valid gesture has been detected from the second image, which can reduce unnecessary interpolation operations, save resources, and reduce power consumption.

[0185] In another possible implementation, step S410 is triggered by step S404, i.e., when the electronic device detects the first valid gesture from the first image, it directly interpolates the first image. Because in some scenarios, the starting gestures of the preset dynamic gestures are relatively special gestures, the probability of user misoperation is relatively small. Therefore, after detecting the first valid gesture from the first image, interpolating the first image in advance can improve the efficiency of gesture perception and further increase the success rate of gesture perception when the user performs gestures quickly.

[0186] It is understandable that, since the third image is obtained by interpolating the first image, and the premise for executing step S410 is that the first valid gesture is detected in the first image, the third image also includes the first valid gesture.

[0187] S411: Detect the third image to obtain a third detection result.

[0188] Exemplarily, after performing interpolation processing on the first image to obtain a third image, the electronic device detects the third image to obtain a third detection result, where the third detection result is used to indicate the category of the first valid gesture.

[0189] It is understandable that due to the relatively low resolution of the first image, it is difficult to accurately identify the specific category of the first valid gesture in the first image. Therefore, in the embodiment of the present application, by interpolating the first image to obtain a high-resolution image (i.e., the third image), and detecting the high-resolution image, the specific category of the first valid gesture can be determined more accurately, thereby improving the accuracy of gesture perception.

[0190] It should be noted that this application does not limit the specific implementation method of detecting the third image to obtain the third detection result. In one possible example, the electronic device inputs the third image into the second model and outputs the third detection result. That is, the electronic device can use the same model to detect the second image and the third image. The second model is a multi-classification model. Compared with the binary classification model, the multi-classification model can output the specific category of the detected valid gesture, so that the electronic device can determine which starting gesture the user performed based on the detection result, so as to execute the subsequent gesture perception process.

[0191] S412: Determine whether the category of the first valid gesture is the same as the category of the second valid gesture.

[0192] For example, after the electronic device detects the second image and the third image respectively, it determines whether to execute the gesture perception process based on the detection results (i.e., the second detection result and the third detection result). The gesture perception process is used to detect a target dynamic gesture and perform a preset operation corresponding to the target dynamic gesture, where the target dynamic gesture is one of multiple preset dynamic gestures. For an introduction to the gesture perception process, please refer to the description of step S413, which will not be repeated here.

[0193] When the first valid gesture and the second valid gesture are of the same category, that is, the first valid gesture and the second valid gesture are the same gesture, and the gesture is the starting gesture of the target dynamic gesture in the preset dynamic gesture, in this case, the electronic device triggers the subsequent gesture perception process.

[0194] Therefore, in the solution provided in the embodiment of the present application, a dual-frame confirmation mechanism is used to determine whether the starting gesture of the target dynamic gesture has been detected, that is, the starting gesture of the target dynamic gesture is detected in both frames of high-resolution images (the second image and the third image) before the subsequent process is continued.

[0195] In this solution, only two frames of images need to be captured to trigger the subsequent gesture sensing process. Because method 400 interpolates the first image at the first resolution (i.e., low resolution) to obtain the third image (i.e., high-resolution image), only one more frame of high-resolution image (i.e., second image) needs to be captured to obtain two frames of high-resolution images. This solution not only maintains the gesture sensing function of the electronic device at a low power cost, but also improves the efficiency of gesture sensing. Even if the user's gesture operation is relatively fast, a high detection success rate can be guaranteed, thus achieving a balance between power consumption and effectiveness.

[0196] Optionally, when the category of the second valid gesture indicated by the first detection result is the same as the category of the first valid gesture indicated by the second detection result, the electronic device may further display an indication icon, wherein the indication icon is used to prompt the user to continue inputting the target dynamic gesture (i.e., inputting other gestures in the target dynamic gesture except the starting gesture), or the indication icon is used to indicate to the user that the electronic device has detected the starting gesture of the target dynamic gesture.

[0197] This indicator icon can prompt the user to continue performing other gestures after the electronic device confirms the detection of the starting gesture of the target dynamic gesture, thereby improving the user experience. For example, after the user observes the indicator icon, the user knows that the electronic device has detected the starting gesture of the target dynamic gesture and is ready to detect other gestures in the target dynamic gesture. At this time, the user can continue to perform other gestures to trigger the preset operation corresponding to the target dynamic gesture.

[0198] This application does not limit the specific style of the indication image, which can be represented by text or icons. This application does not limit it. As an example, the style of the indication icon corresponds to the category of the target dynamic gesture. Figures 10 to 14 In the application scenario of the gesture perception function shown in FIG, several indicator icons are given as examples. Figure 10 For example: Figure 10 As shown in (a), after the electronic device confirms that the starting gesture is detected, it displays an indicator icon S1001 to prompt the user to continue to perform other gestures. The user continues to perform the gestures according to the indicator icon S1001. Figure 10 As can be seen from the figure, the gesture style in the indicator icon S1001 is palm-up, which is consistent with the gesture type performed by the user, making it easy to observe whether the electronic device accurately recognizes the starting gesture performed by the user, so that if the electronic device recognizes it incorrectly, the user can perform the air gesture again.

[0199] In one implementation, for gestures of the same category, the styles of the starting icons are consistent, such as Figure 11 The indicator icon S1101 in (a) and Figure 14 As shown in the indicator icon S1401 in (a), their styles are similar to Figure 10 The indicator icon S1001 in (a) is consistent with that in (b). Figure 12 The style of the indicator icon S1201 shown in (a) is consistent with the two-finger spread gesture operated by the user. Figure 13 The style of the indicator icon S1301 shown in (a) is consistent with the style of the user's palm facing downward, and will not be described in detail here.

[0200] S413, executing the gesture perception process.

[0201] Exemplarily, if the second detection result indicates that the second image includes a second valid gesture, and the category of the second valid gesture is the same as the category of the first valid gesture indicated by the third detection result, the gesture perception process is performed. In this case, the first valid gesture and the second valid gesture are of the same category and are the starting gestures of the target dynamic gesture.

[0202] The electronic device performing gesture perception function described in the embodiment of the present application means that after the electronic device detects the starting gesture of the target dynamic gesture in two frames of images, it detects other gestures in the target dynamic gesture except the starting gesture (if there are other gestures). If other gestures are detected, the preset operation corresponding to the target dynamic gesture is executed.

[0203] In one example, the target dynamic gesture includes a first gesture, so the first gesture is both the starting gesture and the ending gesture of the target dynamic gesture. That is, in this example, the target dynamic gesture consists of one gesture, such as Figure 7 In this example, when the electronic device determines that the first valid gesture and the second valid gesture are of the same category, it determines the starting gesture (i.e., the first gesture) of the detected target dynamic gesture. In this case, the electronic device directly performs the preset operation corresponding to the target dynamic gesture.

[0204] In another example, the target dynamic gesture includes a first gesture and a second gesture, the first gesture is the starting gesture of the target dynamic gesture, and the second gesture is the ending gesture of the target dynamic gesture. That is, in this example, the target dynamic gesture consists of two gestures, for example, Figure 8Any of the four preset dynamic gestures in . In this example, when the electronic device determines that the categories of the first valid gesture and the second valid gesture are the same, it obtains a frame of the fourth image captured by the camera in the second mode, and detects the fourth image to obtain a fifth detection result. The fifth detection result is used to indicate whether there is a fourth valid gesture in the fourth image, and the category of the fourth valid gesture when the fourth valid gesture exists. The fourth valid gesture is any one of the termination gestures of one or more preset dynamic gestures. When the fifth detection result indicates that the fourth valid gesture is the second gesture, the electronic device controls the execution of the preset operation corresponding to the target dynamic gesture.

[0205] In another example, the target dynamic gesture includes a third gesture, a fourth gesture, and a fifth gesture, wherein the third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the middle gesture of the target dynamic gesture, and the fifth gesture is the ending gesture of the target dynamic gesture. That is, in this example, the target dynamic gesture consists of three gestures, for example, Figure 9 In this example, if the electronic device determines that the first valid gesture and the second valid gesture are of the same category, it obtains a fifth image and a sixth image captured by the camera in the second mode, and performs detection on the fifth and sixth images to obtain a sixth detection result and a seventh detection result. The sixth detection result indicates whether the fifth valid gesture exists in the fifth image and, if so, its category. The fifth valid gesture is any of the intermediate gestures of one or more preset dynamic gestures. The seventh detection result indicates whether the sixth valid gesture exists in the sixth image and, if so, its category. The sixth valid gesture is any of the terminating gestures of one or more preset dynamic gestures. If the sixth detection result indicates that the fifth valid gesture is the fourth gesture and the seventh detection result indicates that the sixth valid gesture is the fifth gesture, the electronic device controls the execution of a preset operation corresponding to the target dynamic gesture. It will be understood that the fifth and sixth images are not necessarily adjacent frames, as the camera may capture multiple images including the fifth valid gesture and multiple images including the sixth valid gesture. Therefore, the fifth and sixth images may be separated by x frames, where x can be a preset value.

[0206] It is understandable that the embodiment of the present application is described by taking the first detection result as an example of indicating whether the first image includes the first valid gesture, but in another possible implementation, the first detection result may also only indicate whether the first image includes a gesture. That is, in another implementation, after a gesture is detected from the first image (regardless of whether it is a valid gesture), the camera is triggered to switch to the second mode, and after the starting gesture of the target dynamic gesture is detected, the camera is triggered to switch to the third mode, and the third mode is used to capture an image of a third resolution, which is greater than the second resolution. For example, the electronic device controls the camera to capture a first image in the first mode, detect whether there is a gesture within the camera range, capture a second image in the second mode, detect whether the gesture is the starting gesture of the target dynamic gesture, and capture a third image in the third mode to detect gesture changes, that is, detect whether the gesture changes include the intermediate gesture and the end gesture of the target dynamic gesture. By adjusting the working mode of the camera step by step, the resolution and / or frame rate of the captured image is improved, while minimizing the power consumption of the device, the accuracy of gesture recognition is improved.

[0207] S414: Determine whether the number of detection frames is less than a threshold.

[0208] Optionally, when the category of the first valid gesture is different from the category of the second valid gesture, the electronic device may return to step S407, i.e., reacquire the high-resolution (i.e., second resolution) image captured by the camera in the second mode, and detect the captured high-resolution image to determine whether a valid gesture is detected. If the number of detections exceeds a preset threshold value (e.g., n) and no valid gesture is detected, the camera is controlled to switch from the second mode back to the first mode to save power consumption. For example, when the category of the first valid gesture is different from the category of the second valid gesture, the electronic device acquires n consecutive frames of images captured by the camera in the second mode, and detects the n frames of images in sequence to obtain n fourth detection results, the n fourth detection results corresponding to the n frames of images, each fourth detection result being used to indicate whether the corresponding image has a third valid gesture, and the category of the third valid gesture when the third valid gesture exists, and n is an integer greater than or equal to 1. When the n fourth detection results indicate that the third valid gesture does not exist in any of the n frames of images, the camera is controlled to switch from the second mode back to the first mode.

[0209] In summary, an embodiment of the present application provides a gesture perception method, in which an electronic device controls a camera to capture images in a first mode to detect whether there is a valid gesture within the camera range. Since the first mode is a low-resolution mode, the electronic device can continue to capture images with lower power consumption. After a valid gesture is detected in the first image (a frame of image captured in the first mode), the electronic device switches the shooting mode and controls the camera to capture a frame of the second image in the second mode to detect whether there is a valid gesture within the camera range and the category of the valid gesture. On the other hand, the electronic device interpolates the first image to obtain a high-resolution third image, and detects the category of the valid gesture in the third image. Finally, based on the detection results corresponding to the second image and the third image, it is determined whether to execute the gesture detection process.

[0210] This solution automatically switches between high-resolution and low-resolution camera modes during different detection phases to meet the image quality requirements of each detection stage and reduce device power consumption. Furthermore, this solution can determine whether to trigger the gesture perception process based on just two frames of image data, ensuring a high detection success rate even when the user's gestures are fast, thus achieving a balance between power consumption and performance.

[0211] Compared to the above solution, the embodiment of the present application also provides another solution, in which the electronic device controls the camera to continuously capture images of the second resolution in the second mode. That is, the electronic device continuously captures high-resolution (i.e., second resolution) images and detects the captured images to determine whether to trigger the subsequent gesture perception process. The following is an exemplary description:

[0212] The electronic device controls the camera in the second mode to capture a seventh image, and performs a downsampling operation on the seventh image to obtain an eighth image. The resolution of the eighth image is the first resolution, or the resolution of the eighth image is less than a preset value. In other words, the resolution of the seventh image is reduced by downsampling (e.g., from the second resolution to the first resolution), and the purpose of reducing the resolution is to use the first model to effectively detect gestures in the image (please refer to the subsequent description for details).

[0213] The electronic device then detects the eighth image and obtains an eighth detection result. The eighth detection result indicates whether the eighth image contains an eighth valid gesture, where the eighth valid gesture is any one of the starting gestures of multiple preset dynamic gestures. For example, the electronic device inputs the eighth image into a first model and outputs an eighth detection result, where the first model is a binary classification model. For an explanation of the first model, refer to the above embodiment and are not further described here. Furthermore, if the eighth detection result indicates that the eighth image contains the eighth valid gesture, the electronic device captures another ninth image frame using the camera in the second mode and detects the ninth image to obtain a ninth detection result. The ninth detection result indicates whether the ninth valid gesture is present in the ninth image and, if present, the category of the ninth valid gesture. For example, the electronic device inputs the ninth image into a second model and outputs a ninth detection result, where the second model is a multi-classification model. For an explanation of the second model, refer to the above embodiment and are not further described here.

[0214] If the ninth detection result indicates that the ninth image includes a ninth valid gesture, the electronic device detects the eighth image to obtain a tenth detection result, where the tenth detection result is used to indicate the category of the eighth valid gesture. If the category of the eighth valid gesture is the same as the category of the ninth valid gesture, the electronic device determines to trigger the gesture perception process.

[0215] Therefore, in the above solution, high-resolution images are continuously captured, and during the first-stage detection process (detecting the presence of valid gestures), the high-resolution images can be downsampled so that the first model can be used to detect the downsampled images. Because the first model is a binary classification model, it consumes less power than a multi-classification model. Because in gesture perception scenarios, the majority of the time, the first-stage detection is performed, using a binary classification model can effectively reduce power consumption. After detecting a valid gesture in the image obtained by downsampling the seventh image, the electronic device can use the second model to detect the gesture category of the seventh image and the next frame (i.e., the ninth image), and determine whether to trigger the subsequent gesture perception process based on the detection results. In other words, this solution does not require a mode switching process and only requires two frames of images to determine whether to trigger the subsequent gesture perception process. It also reduces the power consumption of the first-stage detection process, achieving a balance between detection effectiveness and power consumption.

[0216] Corresponding to the gesture perception method described in the above embodiment, the present application also provides a hardware structure diagram of an electronic device for implementing the gesture perception method, such as Figure 16 As shown in the electronic device 500.

[0217] Electronic device 500 includes a processor 510, a camera 520, a display screen 530, and a memory 540. Optionally, it also includes a sensor module 550, which may include, for example, a distance sensor 551. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Electronic device 500 includes 1 to N cameras and 1 to M display screens, where N and M are both positive integers greater than 1.

[0218] It is understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. Figure 16 The modules or units shown, the electronic device 500 may also include one or more of the following: a mobile communication module, a wireless communication module, an antenna, an audio module, a speaker, a microphone, a headphone interface, a button, a motor, an indicator, a subscriber identification module (SIM) card interface, an external memory interface, a charging management module, a battery, a power management module, etc. The sensor module 550 may also include one or more of the following sensors: a pressure sensor, a gyroscope sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, etc.

[0219] The processor 510 may include one or more processing units, for example: the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a DSP, a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0220] The different processing units may be independent devices or integrated into one or more processors. For example, the processor 510 is used to execute the gesture perception method in the embodiment of the present application.

[0221] The memory 540 may refer to an internal memory, also known as "memory," and may be used to store computer executable program code, which includes instructions. The processor 510 calls the executable program code stored in the memory 540 to enable the electronic device to perform the gesture perception method described in the embodiments of this application. The memory 540 may include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), and the like.

[0222] As a possible implementation, in an embodiment of the present application, the memory of the electronic device adopts tightly coupled memory (TCM) and double data rate (DDR) memory.

[0223] Among them, DDR memory is a double data rate synchronous dynamic random access memory with a very high data transmission rate. Its full name is double data rate synchronous dynamic random access memory (DDR SDRAM).

[0224] The TCM is included in the memory address map and can be accessed as fast memory. It provides low-latency memory to the processor without the unpredictability inherent in cache memory. The TCM can be used to store critical routines, such as interrupt handlers or real-time tasks that require extreme avoidance of cache uncertainty. Furthermore, the TCM can be used to store temporary register data, data types with local properties that are not suitable for cache memory, and important data structures such as the interrupt stack.

[0225] It should be noted that from the perspective of storage capacity, TCM memory capacity is smaller, while DDR memory capacity is larger. From the perspective of data transmission rate, the data transmission rate of DDR memory is greater than that of TCM memory. From the perspective of power consumption, the power consumption of DDR memory is greater than that of TCM memory. Electronic devices generally use DDR memory, so the power consumption is relatively high; in some scenarios, the embodiments of the present application use TCM memory for data caching to reduce power consumption.

[0226] For example, in an embodiment of the present application, in some scenarios of early pre-perception services (effective gesture recognition scenarios), both the dynamic memory and the static memory in the electronic device use TCM memory to store continuous multiple frames of images captured by the camera in the first mode. These images are images of QQVGA image specifications, where the QQVGA image specification is a smaller image size. It can be understood that the image resolution of the QVGA image specification is lower and the image occupies less memory.

[0227] For example, in some scenarios involving later-stage recognition of specific services (gesture category recognition scenarios), the electronic device's dynamic and static memories utilize a combination of DDR and TCM memory to store multiple consecutive frames of images captured by the camera in the second mode. These images are multiple consecutive frames of images in the QVGA image format, which is a larger image size than the QQVGA image format. It is understood that the QVGA image format has a relatively higher resolution than the QQVGA image format, and the image occupies a larger memory space.

[0228] It is understandable that QQVGA images have the smallest resolution and occupy the smallest memory.

[0229] In the embodiment of the present application, the camera can automatically switch between two modes. For convenience, the process of the camera capturing images of a first resolution in the first mode and performing effective gesture recognition on these first-resolution images is referred to as the first stage, and the process of the camera capturing images of a second resolution in the second mode and performing gesture category recognition on these second-resolution images is referred to as the second stage.

[0230] In the first stage, on the one hand, the camera captures smaller-sized images of QQVGA specifications and stores them in the TCM memory, thus occupying very little memory space; on the other hand, the processor analyzes multiple consecutive frames of images to determine whether the user has triggered a certain service. For example, if a valid gesture is detected in the image through image analysis, the processor can determine that the user has triggered the air gesture service, and then trigger subsequent gesture perception services.

[0231] In the second stage, on the one hand, the camera captures large-size images of QVGA specifications and stores them in DDR memory and TCM memory; on the other hand, the processor analyzes the captured large-size images and identifies the specific content of the triggered business. If the image analysis detects that the palm changes from an open state to a grasping state, the processor can recognize the grasping gesture and use it to trigger a screenshot.

[0232] It can be seen that in the scenario where the camera is in the normally-on state, although the camera will continue to collect a large number of images, through the solution of this application, in the first stage, by acquiring continuous multi-frame images of smaller size and storing them in low-power TCM memory, image analysis is performed on the continuous multi-frame images of smaller size, which can greatly reduce power consumption.

[0233] Display screen 530 is used to display images, videos, etc. Display screen 594 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In the embodiment of the present application, a user can control the content displayed on display screen 530 through gesture sensing, such as zooming in and out of the image displayed on the display screen, taking a screenshot of the current image on the display screen, or sliding the content displayed on the display screen upward.

[0234] The electronic device 500 also includes various sensors that can convert various physical signals into electrical signals. For example, the distance sensor 551 is used to measure the distance. The electronic device 500 can measure the distance by infrared or laser. For example, in a shooting scene, the electronic device 500 can use the distance sensor 551 to measure the distance to achieve fast focus. In the solution provided in the embodiment of the present application, the distance sensor 551 can be used to measure the distance of the gesture operated by the user. For example, in Figure 11 In the described scenario, the user can control the start and pause of the video screen by spreading the palm and pressing forward, where Figure 11 The distance between the starting gesture shown in (a) and the electronic device is greater than Figure 11 The distance between the end gesture shown in (b) and the electronic device. In order to accurately recognize the target dynamic gesture, in addition to collecting the gesture image through the camera and recognizing the collected image, it is also necessary to use the distance sensor 551 to measure the change in the distance between the gesture and the electronic device.

[0235] Electronic device 500 implements display functionality through a GPU, display screen 530, and an application processor. The GPU is a microprocessor for image processing that connects display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 510 may include one or more GPUs that execute program instructions to generate or modify display information.

[0236] The electronic device 500 can implement a shooting function through an ISP, a camera 520, a video codec, a GPU, a display screen 530, and an application processor.

[0237] The camera 520 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 520, where N is a positive integer greater than 1.

[0238] The camera 520 in the embodiment of the present application can switch between at least two modes: a first mode and a second mode. The first mode is used to capture images at a first resolution, and the second mode is used to capture images at a second resolution. The processor 510 can control the camera 520 to switch between the two modes based on preset logic and control the camera to capture images in the two modes.

[0239] In an embodiment of the present application, the camera 520 can realize functions such as air gesture control of the screen through the real-time online (always-on camera, AON) function, wherein the implementation of the AON function depends on the ISP. The ISP can run various algorithm programs to process image signals in real time. The camera module is connected to the ISP through the camera serial interface (camera serial interface, CSI), and the ISP is connected to the AP. After the ISP receives the image captured by the camera module, the ISP performs image signal processing on the image captured by the camera module. The AP can control the working mode of the ISP through the built-in integrated circuit (inter-integrated circuit, I2C), obtain the working status of the ISP, etc.

[0240] The above is a detailed description of the embodiments of the present application using electronic device 500 as an example. It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on electronic device 500. Electronic device 500 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration.

[0241] The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0242] In the embodiment of the present application, in order to realize the gesture sensing function of the electronic device, it is necessary to make hardware and software improvements to the camera and processor of the electronic device. The hardware improvements of the electronic device are first described below.

[0243] For example, Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 17 As shown, the electronic device may include an improved camera 601 and an improved processor 602. The improved camera 601 refers to adding a control circuit and a working circuit corresponding to the newly added shooting mode in the existing camera module to achieve a low-power configuration. For example, the shooting mode of the existing camera module is mode 1, and the camera module includes a working circuit corresponding to mode 1. If mode 2 is added to the shooting mode, correspondingly, switching between mode 1 and mode 2 is also involved. For this, the improved camera module includes not only the working circuit corresponding to mode 1, but also the newly added working circuit corresponding to mode 2, and the control circuit corresponding to the switching between the two modes. It should be understood that more than two shooting modes can be set according to actual application requirements, and this embodiment of the present application does not impose any restrictions on this.

[0244] In one possible implementation, the improved camera 601 includes two operating modes: a first mode and a second mode. The resolution of images captured by the camera 601 in the first mode is lower than the resolution of images captured in the second mode. Optionally, the frame rate of images captured by the camera 601 in the first mode is lower than the frame rate of images captured in the second mode. The camera 601 can switch between these two modes. In this example, the first frame rate is lower than the second frame rate, and the first resolution is lower than the second resolution.

[0245] Exemplarily, in a gesture perception scenario, if the conditions for starting the camera are met (such as the gesture perception function is turned on), the camera 601 operates in a first mode and continuously captures images of a first resolution at a first frame rate (i.e., the camera is in permanent scanning) to detect whether there is a first valid gesture within the range of the camera 601; if a first valid gesture is detected in a frame of the first image captured, the camera 601 switches from the first mode to the second mode and captures a second image of a second resolution at a second frame rate to detect whether the second image includes a second valid gesture and the category of the second valid gesture when a second valid gesture exists.

[0246] Based on the above example, the camera 601 can dynamically adjust the resolution (and optionally the frame rate) of the captured image to adapt to different detection requirements. For example, the camera 601 captures images at a lower resolution (and frame rate) to detect whether there is a first valid gesture within the range of the camera 601, and the camera 601 captures images at a higher resolution (and frame rate) to identify whether there is a second valid gesture in the image and the specific category of the second valid gesture. The low resolution (i.e., the first resolution) of the camera 601 can be 120×160, and the high resolution (i.e., the second resolution) can be 240×320, or 480×640. Exemplarily, the low frame rate (i.e., the first frame rate) of the camera 601 can be 1fps, and the high frame rate (i.e., the second frame rate) can be 30fps.

[0247] It should be noted that the embodiments of this application do not impose specific restrictions on the resolution range and frame rate range of the images captured by the camera. In other words, there are no restrictions on the minimum and maximum frame rates of the images captured by the camera, nor are there any restrictions on the minimum and maximum resolutions of the images captured by the camera. In actual applications, the frame rate range and resolution range can be reasonably set according to needs.

[0248] It should be noted that the camera 601 can be a front camera or a rear camera of the electronic device, and this embodiment does not limit this.

[0249] The improved processor 602 may be a SoC. In an embodiment of the present application, when the electronic device triggers the camera 601 to perform a resident scan to acquire an image of the current scene, the camera 601 may send the image to the SoC for image analysis to detect whether there is a valid gesture in the image and the category of the valid gesture.

[0250] In order to achieve low power consumption, SoC can support low power real-time online ISP (always on ISP, AONISP), refer to the attached Figure 18 Camera 601 transmits the low-resolution image to the AON ISP. The AON ISP converts the image format without performing any image processing. The converted image is then stored in on-chip static random-access memory (SRAM). The SoC can also support an extremely low-power core, allowing computing, algorithm execution, and image storage to operate in low-power mode. Furthermore, the SoC can also support a low-power embedded neural network processor (eNPU).

[0251] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method and system embodiment section and will not be repeated here.

[0252] The SoC involved in the embodiments of the present application is described in detail below.

[0253] For example, Figure 18 This is a schematic diagram of the structure of a SoC provided in an embodiment of the present application. Figure 18 As shown, the SoC includes a first processing module, a second processing module, and a third processing module. The first processing module includes an I2C bus interface, an AON ISP, an on-chip SRAM, a digital signal processor DSP, and an eNPU; the second processing module includes an image signal processing ISP, a neural network processor NPU, and a central processing unit CPU; and the third processing module includes a GPU and an NPU. In the SoC, the power consumption of the first processing module is lower than that of the second processing module. Specifically, the power consumption of the eNPU in the first processing module is lower than that of the NPU in the second processing module, and the power consumption of the AON ISP in the first processing module is lower than that of the ISP in the second processing module.

[0254] As an example, the first processing module can be used to process the first image of the first resolution mentioned above captured by the camera 901. For example, in the first mode, the camera 601 captures the first image of the first resolution, the AON ISP obtains the first image of the first resolution through the I2C bus interface, and after processing by the AON ISP, the eNPU detects the processed first image of the first resolution, for example, detecting whether there is a first valid gesture in the first image. The on-chip SRAM in the first processing module can be used to store the processed first image of the first resolution. For example, after detecting the first valid gesture from the first image, the first image is stored in the on-chip SRAM. The DSP can be used to notify the eNPU to perform image detection, receive the detection results reported by the eNPU, and report the detection results to the upper-layer application. The first processing module adopts a low-power configuration to reduce the power consumption of the electronic device.

[0255] As an example, the second processing module can be used to process the second image of the second resolution captured by the camera 601. For example, in the second mode, the camera 601 captures the second image of the second resolution, and after IPS processing, the NPU recognizes the processed second image of the second resolution, for example, identifying whether there is a second valid gesture in the second image and the category of the second valid gesture. Before sending the data (such as image data) to the memory, the second processing module can perform security processing (such as encryption processing) and store the security processed data in a security buffer of the memory.

[0256] As an example, the third processing module can be used to obtain a first image from the on-chip SRAM, and interpolate the first image through the GPU to obtain a third image of the second resolution, and then the NPU recognizes the interpolated third image of the second resolution, for example, identifying the category of the first valid gesture in the third image.

[0257] It should be noted that the embodiments of the present application do not limit the format of image data transmitted between the first processing module or the second processing module and the camera, or between the first processing module and the third processing module. For example, the image data may be camera serial interface (CSI) or mobile industry processor interface (MIPI) data.

[0258] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device. Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The layered architecture divides the software system of an electronic device into several layers, each with clear roles and divisions of labor. Layers communicate with each other through software interfaces.

[0259] Reference Figure 19 The electronic device of the embodiment of the present application includes an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, a sensor hub and a hardware layer.

[0260] The application layer may include a series of applications, and the application layer runs the applications by calling the application programming interface (API) provided by the application framework layer.

[0261] In an embodiment of the present application, the application layer may include a gesture perception application and a perception module. The gesture perception application is connected to the perception module, and the gesture perception application is registered in the perception module. The perception module performs status management and data transmission. For example, when the perception module learns from the first processing module in the sensorhub that there is a valid gesture within the camera range, the perception module notifies the second processing module in the HAL so that the second processing module obtains an image of the second resolution and identifies the category of the second valid gesture. Finally, the perception module reports the recognition result to the gesture perception application.

[0262] In some embodiments, the application layer also includes other applications ( Figure 19 Not shown), such as the gaze-on-screen application and the gaze-on-always-on-display (AOD) application. One possible scenario is that multiple applications correspond to the same algorithm. For example, the gaze-on-screen application and the gaze-on-AOD application correspond to the gaze detection algorithm, and the perception module can be used to uniformly schedule and manage the gaze detection algorithm. Another possible scenario is that different applications correspond to different algorithms. For example, the gesture perception application corresponds to the gesture recognition algorithm (including two algorithms for identifying whether there is a valid gesture and identifying the category of valid gestures), and the gaze-on-screen application corresponds to the gaze detection algorithm. Both algorithms involve obtaining image data from the underlying camera, and the perception module can be used to schedule and manage the priorities among multiple algorithms. There is a possibility that the gesture recognition algorithm and the gaze detection algorithm have the same priority, and the perception module can notify the underlying camera to report the image data to the gesture perception application and the gaze-on-screen application at the same time. In this embodiment, the algorithm for identifying whether there is a valid gesture is deployed in the first processing module, and the algorithm for identifying the category of valid gestures is deployed in the second processing module and the third processing module. The algorithms shown in this embodiment are only examples.

[0263] In some embodiments, if the electronic device includes a foldable screen, the application layer further includes a fourth processing module, which is configured to obtain the physical state of the foldable screen and the state of the internal and external screen cameras (on or off) reported by the first processing module. Furthermore, the fourth processing module is further configured to notify the second processing module of the state of the internal and external screen cameras.

[0264] In some embodiments, the application may also include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which may be system applications or third-party applications. The embodiments of the present application do not limit this.

[0265] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 19 As shown, the application framework layer may include a camera service (CameraService), which is used to schedule and manage the priorities of all applications that need to use the camera.

[0266] In some embodiments, the application framework layer may also include, for example, a window manager, a content provider, a resource manager, a notification manager, a view system, etc., which is not limited in the embodiments of the present application.

[0267] The hardware abstraction layer may include an AO service, a second processing module, and a third processing module. The AO service may be used to control the on or off of the algorithm for identifying gesture categories in the second processing module, as well as the on or off of the algorithm for identifying the presence or absence of gestures in the third processing module, and the on or off of the algorithm for identifying the presence or absence of gestures in the first processing module, as well as upper and lower layer data transmission. The second processing module may be used to process images with higher resolution and / or higher frame rate, such as the second image mentioned above, to detect the category of valid gestures in the second image. The second processing module is also used for camera mode switching. For example, the second processing module receives instructions from the perception module or the first processing module to control the camera to switch from the first mode to the second mode.

[0268] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to enable it to operate. In an embodiment of the present application, the kernel layer may include a camera driver, which is used to drive the camera in the electronic device to operate in a first mode or a second mode to capture images at different frame rates and / or resolutions.

[0269] In addition, the core layer may also include display drivers, audio drivers, sensor drivers, motor drivers, etc., which are not limited in the embodiments of the present application. Among them, the sensor driver can drive, for example, a proximity light sensor to emit a light signal to detect whether the user is currently holding the electronic device close to the ear to make a call, etc. The sensor driver can also drive, for example, a gyroscope sensor to detect the posture data of the electronic device; the sensor driver can also drive, for example, an ambient light sensor to detect the ambient light brightness to detect whether the electronic device is in a dark environment, such as a mobile phone in a pocket.

[0270] The sensor hub is used to centrally control sensors, reducing CPU load. The sensor hub is equivalent to a microprogrammed control unit (MCU), which can run programs that drive multiple sensors. This means the sensor hub can support multiple sensors. It can be placed as a standalone chip between the CPU and various sensors, or integrated into the CPU's application processor (AP).

[0271] In an embodiment of the present application, the sensorhub may include a first processing module, which may be used to process images with lower resolution and / or lower frame rate, such as the first image mentioned above, to detect whether a first valid gesture exists in the first image. Compared to the second processing module, the first processing module is a low-power processing module, and the first processing module runs permanently or in a low-power form. As an example, when the gesture sensing function is turned on, the first processing module is also used to obtain data reported by various sensors, and determine various states of the electronic device based on the various sensor data, such as screen state, unlock state, usage state, etc. If the device state meets the first condition, the first processing module may send a first shooting instruction to the camera, instructing the camera to scan the image permanently in a low-power shooting mode (such as the first mode) to detect whether a first valid gesture exists within the camera range. As an example, for a foldable device, the first processing module may determine whether to send the first shooting instruction to the camera (the camera on the inner screen or the outer screen) by detecting whether the physical state of the folding screen, the screen state, and the device state meet the second condition. As an example, when the first processing module detects that the physical state of the screen of the foldable screen mobile phone has changed, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the first processing module can control the camera of the foldable screen mobile phone (such as the camera on the external screen of the mobile phone, and / or the camera on the internal screen of the mobile phone) to be turned on or off.

[0272] The hardware layer may include, for example, cameras, various sensors, and AON ISP.

[0273] It is understandable that Figure 19 The layers in the illustrated hierarchical structure and the modules or components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in this application. Figure 19 The modules included in each layer shown are modules involved in the embodiments of the present application, and the modules included in each layer do not constitute a limitation on the structure of the electronic device and the level of module deployment (illustrative example). Figure 19 The modules shown in can be deployed individually, or several modules can be deployed together. Figure 19 The division of modules in is an example. In some embodiments, Figure 19 The module names shown in are examples.

[0274] Based on the structure of the electronic device shown above, the gesture perception method provided by the embodiment of the present application is described below in conjunction with specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0275] Figure 20 This is a flow chart of a gesture perception method provided in an embodiment of the present application. Figure 20 As shown, the gesture perception method provided by this embodiment includes:

[0276] S701: The target application registers a gesture perception function in the perception module.

[0277] In this embodiment, the target application is a gesture-sensing application at the application layer. In some embodiments, the target application obtains information about gesture-sensing functions from a server (or cloud). After obtaining the information about gesture-sensing functions, the target application may register the gesture-sensing functions with a sensing module so that the sensing module can run resident and execute gesture-sensing-related tasks. The gesture-sensing function information includes code information that implements the gesture-sensing functions, and may also include information such as preset operations corresponding to different air gesture operations.

[0278] S702: The perception module determines whether to start the gesture perception function.

[0279] If the perception module determines that the gesture perception function is activated, S703 is executed.

[0280] For example, referring to Figure 5 In (a) and (b), if the user chooses to turn on the gesture perception function in the settings interface of the system application, the system application can send a notification to the perception module to inform the perception module that the gesture perception function has been turned on in the mobile phone.

[0281] S703. The perception module sends a first instruction to the first processing module.

[0282] In one implementation, when the gesture sensing function is enabled, the sensing module sends a first indication to the first processing module, instructing the first processing module to detect whether a first valid, recognizable gesture is within range of the electronic device's camera. The sensing module's transmission of the first indication may control the activation of the first processing module, including powering on the module, issuing a working scenario, and preparing resources.

[0283] In another implementation, when the gesture sensing function is enabled and a first condition is satisfied, the sensing module sends a first indication to the first processing module. In this embodiment, the first condition includes at least one of the following:

[0284] The screen of the electronic device is in a bright state;

[0285] The electronic device is unlocked;

[0286] The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal;

[0287] Detection data of the ambient light sensor of the electronic device is greater than a third threshold;

[0288] The screen of the electronic device faces a preset direction;

[0289] The electronic device runs the target application.

[0290] In this embodiment, if the electronic device has the gesture sensing function enabled and the first condition is met, the camera of the electronic device is triggered to continuously capture the first image. By adding the first condition, the electronic device is prevented from continuously capturing the first image when not necessary, further reducing the power consumption of the device.

[0291] S704: The first processing module sends a first shooting instruction to the camera in response to the first instruction, where the first shooting instruction is used to instruct the camera to work in the first mode.

[0292] The first processing module sends a first shooting instruction to control the camera startup, including camera power-on, mode switching, image output resolution and frame rate settings, etc. The first mode is a low-resolution shooting mode, in which the camera continuously captures images at a lower resolution (optionally also including a lower frame rate).

[0293] In some embodiments, if the electronic device is a foldable device, the foldable device includes a first screen and a second screen, the first screen is correspondingly provided with a first camera, and the second screen is correspondingly provided with a second camera.

[0294] In one possible scenario, the detection of the first condition described in S703 can be replaced by: the first processing module detects that the inner screen of the electronic device is in a bright screen state, and the electronic device is in an unfolded state. Correspondingly, S704 can be: the first processing module sends a first shooting instruction to the first camera. In some embodiments, before the first processing module sends the first shooting instruction to the first camera, it also includes: detecting that the state of the electronic device meets the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the inner screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the inner screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0295] In one possible scenario, the detection of the first condition described in S703 can be replaced by: the first processing module detects that the external screen of the electronic device is in a bright screen state, and the electronic device is in a folded state. Correspondingly, S704 can be: the first processing module sends a first shooting instruction to the second camera. In some embodiments, before the first processing module sends the first shooting instruction to the second camera, it also includes: detecting that the state of the electronic device meets the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the external screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the external screen of the electronic device is facing a preset direction; the electronic device runs a preset application.

[0296] S705: The camera sends the first image to the first processing module.

[0297] In response to the first shooting instruction, the camera captures a first image with a first resolution in a first mode (at a first frame rate). For example, the first frame rate can be set to 5 fps, and the first resolution can be set to 200×180 or 640×480.

[0298] S706: The first processing module identifies whether the first image contains a first valid gesture.

[0299] If the first processing module recognizes the first valid gesture in the first image, step S707 is executed. Otherwise, the first processing module continues to acquire the image of the first resolution from the camera and recognizes whether the acquired image contains the first valid gesture, unless the camera is controlled to be turned off.

[0300] In some embodiments, the first processing module is pre-installed with a first model, which is an effective gesture recognition model. The first model is a binary classification model. The first processing module detects the first image through the first model and determines whether there is a first effective gesture in the first image based on the recognition result output by the first model.

[0301] In some embodiments, the first model can be trained using a lightweight neural network model to identify whether the first valid gesture exists in the image. The first model can be deployed on the eNPU of the first processing module and has good real-time performance.

[0302] As an example, the training process of the first model may include:

[0303] Step a: construct a training set and a test set for the first model. The training set or the test set includes sample images and the annotation results corresponding to the sample images (i.e., the presence or absence of gestures in the sample images). The sample images in the training set and the test set are different.

[0304] Step b: training the first model based on the initial first model and the training set. Specifically, the sample images of the training set are used as inputs of the initial first model, and the annotation results corresponding to the sample images of the training set are used as outputs of the initial first model to train the first model.

[0305] Step c: Based on the first model trained in step b and the test set, the prediction results of the first model are verified, and when the model loss function converges, the training of the first model is stopped.

[0306] S707. When a first valid gesture is recognized from the first image, the first processing module sends a first message to the perception module. The first message is used to notify the perception module that the first valid gesture exists within the camera range of the electronic device.

[0307] S708. The perception module sends a second instruction to the second processing module.

[0308] The second indication instructs the second processing module to identify whether a second valid gesture exists within the camera range and, if so, the type of the second valid gesture. The first processing module sends the second indication to control the activation of the second processing module, including powering on the module, issuing the work scenario, and preparing resources.

[0309] S709: The second processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to work in the second mode.

[0310] The second processing module sends a second shooting instruction to control the camera to switch modes, that is, switch from the first mode to the second mode. The camera continues to capture images with a higher resolution in the second mode, that is, the second image.

[0311] S710: The camera sends a second image to a second processing module.

[0312] In response to the second shooting instruction, the camera captures a second image of a second resolution at a second frame rate in a second mode and sends the second image to the second processing module. In this embodiment, the second image is a frame of image.

[0313] S711 . The second processing module identifies whether a second valid gesture exists in the second image and the category of the second valid gesture.

[0314] In some embodiments, the second processing module is preset with a second model, and the second processing module identifies whether a second valid gesture exists in the second image and the category of the second valid gesture through the second model.

[0315] As an example, the second model can be trained using a lightweight neural network model based on deep learning methods and is used to identify the categories of valid gestures in images. As an example, the second model can include Model #1, Model #2, and Model #3, where Model #1 is used to identify the overall frame and orientation of the hand, Model #2 is used to identify 3D hand nodes, and Model #3 is used to classify the identified hand nodes to determine the specific category of the gesture.

[0316] Model #1 can be a model that can recognize a single frame image, mainly providing an accurately cropped palm image for the second model. In addition, it can recognize a variety of different palm sizes, recognize hand occlusion, and accurately locate the hand by recognizing the arm, torso, or personal features. Model #2 can be a landmark model that can recognize 21 hand nodes and the position occlusion between these nodes. For example, Figure 21 A schematic diagram of the hand nodes is shown. Model #2 can identify the coordinates of the 21 hand nodes. Model #3 can infer the movements of each finger, such as straightening or bending, based on the hand node data extracted by Model #2. These movements are matched with preset gestures to predict the gesture category.

[0317] As an example, the second model can be deployed on the NPU of the second processing module and has good real-time performance.

[0318] In some embodiments, the second processing module identifies whether a second valid gesture exists in the second image and the category of the second valid gesture, which may include: identifying a hand area from the second image to remove non-hand areas in the second image; performing gesture analysis on the identified hand area, such as inputting the image of the identified hand area into the above-mentioned second model to determine the category of the gesture in the image.

[0319] In some embodiments, the second processing module may use background subtraction to separate the foreground (ie, the hand area) and the background (ie, the non-hand area) in the second image.

[0320] Exemplarily, when there is no second valid gesture in the second image, the second processing module sends a fourth message (not shown in the figure) to the perception module, and the fourth message is used to notify the perception module that there is no second valid gesture in the second image.

[0321] S712. When a second valid gesture exists in the second image, the second processing module sends a second message to the perception module. The second message is used to notify the perception module that a second valid gesture exists in the second image and the category of the second valid gesture.

[0322] S713. The perception module sends a third instruction to the third processing module, where the third instruction is used to instruct the third processing module to perform interpolation processing on the first image and identify the category of the first valid gesture based on the high-resolution (second resolution) image obtained by interpolation.

[0323] It is understandable that S713 can be triggered and executed by step S712 or by step S707, and this application does not limit this.

[0324] S714: The third processing module performs interpolation processing on the first image to obtain a third image.

[0325] Exemplarily, in response to the third instruction from the perception module, the third processing module extracts the first image from the memory (such as the On-chip SRAM in the first processing module), and performs interpolation processing on the first image to obtain a third image of the second resolution.

[0326] S715: The third processing module detects the third image to identify the category of the first valid gesture.

[0327] In some embodiments, the third processing module is preset with a second model, and the third processing module identifies the category of the first valid gesture in the third image through the second model.

[0328] S716. After detecting the category of the first valid gesture, the third processing module sends a third message to the perception module. The third message is used to notify the perception module of the category of the first valid gesture.

[0329] S717: The perception module determines whether to trigger the gesture perception process.

[0330] Exemplarily, the perception module determines whether the category of the first valid gesture and the category of the second valid gesture are the same. If the categories of the first valid gesture and the second valid gesture are the same and both are the starting gestures of the target dynamic gesture, the gesture perception process is triggered; if they are different, the gesture perception process is not triggered.

[0331] An example of the execution process of the gesture perception process is as follows: in the case of determining that the gesture perception process is triggered, the perception module can send an instruction to the second processing module, and the second processing module obtains one or more frames of images from the camera based on the instruction of the perception module, and performs gesture category recognition on the acquired images, and notifies the perception module of the recognition result. The perception module obtains the recognition result sent by the second processing module. If other gestures other than the starting gesture are detected in the target valid gesture, an indication message is sent to the target application to indicate that the electronic device recognizes the target dynamic gesture. Optionally, the indication message may include an identifier of the target dynamic gesture. The target application stores the correspondence between the target dynamic gesture and the preset operation (for details, please refer to the examples in the above embodiment). In response to the indication message from the perception module, the target application executes the preset operation corresponding to the target dynamic gesture.

[0332] In the above embodiment, if the electronic device has registered the gesture perception function and has turned on the gesture perception function, the perception module can send a first shooting instruction to the camera through the first processing module (low-power processing module) to enable the camera to capture the first image at a lower frame rate and / or resolution. If the first processing module recognizes that there is a first valid gesture in the first image, it can notify the second processing module so that the second processing module sends a second shooting instruction to the camera to enable the camera to capture the second image at a higher frame rate and / or resolution. The second processing module recognizes the category of the second valid gesture in the second image. On the other hand, the third processing module interpolates the first image into a high-resolution third image, and identifies the category of the first valid gesture based on the third image. If the category of the first valid gesture is the same as the category of the second valid gesture, the gesture perception process is triggered.

[0333] The embodiment of the present application also provides a gesture perception method, which is applied to an electronic device with a flexible screen. The following is an example of a folding screen mobile phone. The folding screen mobile phone has a first screen and a second screen. The first screen is a folding screen that can be folded inward. The first screen and the second screen are respectively provided with a first physical camera and a second physical camera for collecting image data. The gesture perception method of this embodiment involves the processing logic of the underlying module of the device when the folding screen state changes. Figure 22 Provide explanation.

[0334] For example, Figure 22 This is a flow chart of a gesture perception method provided in an embodiment of the present application. Figure 22 As shown, the gesture perception method of this embodiment may include the following steps:

[0335] S801. The first processing module obtains sensor data to determine the change in the physical state of the folding screen.

[0336] In this embodiment, sensor data includes, for example, magnetic sensors and Hall sensors, and is used to determine whether the physical state of the foldable screen has changed. The change in the physical state of the foldable screen includes a change from a folded state to an unfolded state, or vice versa.

[0337] S802a. The first processing module controls the first physical camera to be turned on or off based on the change in the physical state of the folding screen.

[0338] S802b. The first processing module controls the second physical camera to be turned on or off based on the change in the physical state of the folding screen.

[0339] In a possible implementation, if the physical state of the folding screen changes from a folded state to an unfolded state, the first processing module may control the first physical camera to be turned on.

[0340] In one possible implementation, the second physical camera is turned on. If the physical state of the folding screen changes from a folded state to an unfolded state, the first processing module can control the second physical camera to be turned off and at the same time control the first physical camera to be turned on.

[0341] In one possible implementation, the first physical camera is turned on. If the physical state of the folding screen changes from an unfolded state to a folded state, the first processing module can control the first physical camera to be turned off and the second physical camera to be turned on at the same time.

[0342] S802c. The first processing module reports the physical state of the folding screen and the state of the internal and external screen cameras to the third processing module.

[0343] It should be noted that this embodiment does not impose any limitation on the execution order of S802a to S802c.

[0344] S803: The fourth processing module sends a notification to the second processing module, notifying the status of the internal and external screen cameras.

[0345] This embodiment shows the interaction process between various modules inside a foldable screen mobile phone when the physical state of the screen changes. Through the above interaction, precise control of the low-power continuous scanning function of the camera is achieved, so that when the user uses the foldable screen mobile phone, the mobile phone can recognize the user's air gesture operations, and then realize the control of the mobile phone in different usage scenarios, such as audio and video playback control, picture display control, etc., to enhance the user experience of the phone.

[0346] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0347] An embodiment of the present application provides a computer program product. When the computer program product runs on a device, the device can implement the steps in the above-mentioned method embodiments when the computer program product is executed.

[0348] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, mobile hard disk, magnetic disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0349] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0350] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0351] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0352] In addition, it should be noted that the various numerical numbers used in this application (such as the terms "first," "second," "third," "fourth," and other terminology (if any) in the specification, claims, and drawings) are merely for descriptive purposes and are not intended to limit the scope of this application. The order of execution of each process does not necessarily imply a specific order of execution; the execution order of each process should be determined by its function and inherent logic.

[0353] The terms "including" and "having" and any variations thereof mean "including but not limited to," unless specifically stated otherwise. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0354] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0355] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terms and / or descriptions between different embodiments are consistent and can be referenced from each other. The technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The specific operating methods in the method embodiments of this application can also be applied to the device embodiments or system embodiments.

Claims

1. A gesture perception method, applied to an electronic device, wherein the electronic device includes a camera, characterized in that: The method comprises: Controlling the camera to operate in a first mode, where the first mode is used to capture images with a first resolution; Obtaining a first image frame captured by the camera in the first mode, and detecting the first image to obtain a first detection result, where the first detection result is used to indicate whether a first valid gesture exists in the first image, where the first valid gesture is any one of the starting gestures of one or more preset dynamic gestures; When the first valid gesture is detected from the first image, controlling the camera to switch from the first mode to a second mode, where the second mode is used to capture images with a second resolution, the first resolution being lower than the second resolution; Acquiring a second image frame captured by the camera in the second mode, and detecting the second image to obtain a second detection result, where the second detection result is used to indicate whether a second valid gesture exists in the second image and, if so, the category of the second valid gesture, where the second valid gesture is any one of the starting gestures of the one or more preset dynamic gestures; interpolating the first image to obtain a third image, and detecting the third image to obtain a third detection result, where the third detection result is used to indicate a category of the first valid gesture, and a resolution of the third image is the same as a resolution of the second image or a resolution of the third image is greater than a preset value; Based on the second detection result and the third detection result, determine whether to execute a gesture perception process, wherein the gesture perception process is used to detect a target dynamic gesture and execute a preset operation corresponding to the target dynamic gesture, and the target dynamic gesture is one of the multiple preset dynamic gestures.

2. The method according to claim 1, characterized in that The interpolating the first image to obtain the third image includes: After determining that the second valid gesture exists in the second image according to the second detection result, interpolating the first image to obtain a third image; or After determining, according to the first detection result, that the first image includes a first valid gesture, the first image is directly interpolated to obtain the third image.

3. The method according to claim 1 or 2, characterized in that The detecting the first image to obtain a first detection result includes: Inputting the first image into a first model and outputting a first detection result, wherein the first model is a binary classification model; The detecting the second image to obtain a second detection result includes: Inputting the second image into a second model and outputting the second detection result, wherein the second model is a multi-classification model; The detecting the third image to obtain a third detection result includes: The third image is input into the second model and the third detection result is output.

4. The method according to any one of claims 1 to 3, characterized in that In a case where the category of the second valid gesture is different from the category of the first valid gesture, the method further includes: Obtaining n consecutive frames of images captured by the camera in the second mode, and sequentially detecting the n frames of images to obtain n fourth detection results, the n fourth detection results corresponding one-to-one to the n frames of images, each fourth detection result being used to indicate whether a third valid gesture exists in the corresponding image, and, if so, the category of the third valid gesture, where n is an integer greater than or equal to 1; When the n fourth detection results indicate that the third valid gesture does not exist in the n frames of image, the camera is controlled to switch from the second mode back to the first mode.

5. The method according to any one of claims 1 to 4, characterized in that The first image and the second image are two adjacent frames of images, or there is one frame of image between the first image and the second image.

6. The method according to any one of claims 1 to 5, characterized in that The determining whether to execute the gesture perception process according to the second detection result and the third detection result includes: When the second detection result indicates that the second image includes the second valid gesture, and the category of the second valid gesture is the same as the category of the first valid gesture indicated by the third detection result, the gesture perception process is executed, and the first valid gesture and the second valid gesture are the starting gestures of the target dynamic gesture.

7. The method according to claim 6, characterized in that In a case where the category of the second valid gesture indicated by the first detection result is the same as the category of the first valid gesture indicated by the second detection result, the method further includes: An indication icon is displayed, where the indication icon is used to prompt the user to continue inputting the target dynamic gesture.

8. The method according to claim 6 or 7, characterized in that The target dynamic gesture includes a first gesture and a second gesture, the first gesture is a starting gesture of the target dynamic gesture, and the second gesture is a terminating gesture of the target dynamic gesture; The execution of the gesture perception process includes: obtaining a fourth image frame captured by the camera in the second mode, and performing detection on the fourth image to obtain a fifth detection result, wherein the fifth detection result is used to indicate whether a fourth valid gesture exists in the fourth image and, if so, a category of the fourth valid gesture, wherein the fourth valid gesture is any one of the termination gestures of the one or more preset dynamic gestures; If the fifth detection result indicates that the fourth valid gesture is the second gesture, control is performed to execute a preset operation corresponding to the target dynamic gesture.

9. The method according to claim 6 or 7, characterized in that The target dynamic gesture includes a third gesture, a fourth gesture and a fifth gesture, the third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the middle gesture of the target dynamic gesture, and the fifth gesture is the ending gesture of the target dynamic gesture; The execution of the gesture perception process includes: Acquire a fifth image and a sixth image captured by the camera in the second mode, and perform detection on the fifth image and the sixth image respectively to obtain a sixth detection result and a seventh detection result, the sixth detection result being used to indicate whether a fifth valid gesture exists in the fifth image and, if so, the category of the fifth valid gesture, the fifth valid gesture being any one of the intermediate gestures of the one or more preset dynamic gestures; the seventh detection result being used to indicate whether a sixth valid gesture exists in the sixth image and, if so, the category of the sixth valid gesture, the sixth valid gesture being any one of the terminating gestures of the one or more preset dynamic gestures; When the sixth detection result indicates that the fifth valid gesture is the fourth gesture and the seventh detection result indicates that the sixth valid gesture is the fifth gesture, control is performed to execute a preset operation corresponding to the target dynamic gesture.

10. The method according to any one of claims 1 to 9, characterized in that The first mode is further used to capture images at a first frame rate, and the second mode is further used to capture images at a second frame rate, wherein the first frame rate is lower than the second frame rate.

11. The method according to any one of claims 1 to 10, characterized in that The controlling the camera to operate in the first mode includes: In response to an operation of turning on the gesture sensing function, the camera is controlled to operate in the first mode.

12. The method according to any one of claims 1 to 11, characterized in that Before controlling the camera to operate in the first mode, the method further includes: It is detected that the state of the electronic device meets a first condition, where the first condition includes at least one of the following: The screen state of the electronic device is a bright screen state; The electronic device is unlocked; The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; Detection data of the ambient light sensor of the electronic device is greater than a third threshold; The screen of the electronic device faces a preset direction; The electronic device runs a preset application.

13. The method according to any one of claims 1 to 12, characterized in that The electronic device is a foldable device, comprising a first screen and a second screen located on the front and back of the electronic device, respectively, in an unfolded state, wherein the first screen is a foldable screen, and the camera comprises a first physical camera and a second physical camera, wherein the first physical camera and the second physical camera are logical front cameras corresponding to the first screen and the second screen, respectively; The controlling the camera to operate in the first mode includes: When it is detected that the first screen is in the bright screen state and the electronic device is in the unfolded state or the hovering state, controlling the first physical camera to operate in the first mode; or, When it is detected that the second screen is in the bright screen state and the electronic device is in the folded state, the second physical camera is controlled to operate in the first mode.

14. The method according to claim 13, characterized in that The method further comprises: When the first physical camera operates in the first mode, upon detecting that the electronic device switches from the unfolded state or the hovering state to the folded state, controlling the second physical camera to operate in the first mode and turning off the first physical camera; When the second physical camera operates in the first mode, upon detecting that the electronic device switches from a folded state to an unfolded state or a hovering state, the first physical camera is controlled to operate in the first mode and the second physical camera is turned off.

15. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

16. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 8.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises instructions, which, when executed on an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 8.

Images