Gesture perception method and electronic device

Abstract

This application provides a gesture sensing method and electronic device, relating to the fields of terminal technology, artificial intelligence, intelligent sensing, and intelligent control. In this method, the electronic device controls a camera to acquire a first image at a first resolution. If a valid gesture is detected in the first image, the camera switches from a first mode to a second mode, and acquires a second image at a second resolution through this second mode. Further, gesture detection is performed on the second image and a third image obtained by up-interpolating the first image, and the gesture detection results determine whether to trigger subsequent gesture sensing processes to detect target dynamic gestures. This solution enables the electronic device to implement gesture sensing functionality with lower power consumption, and when implementing this function, it can respond to user gestures more quickly, improving the efficiency and success rate of gesture sensing.

Description

Technical Field

[0001] This application belongs to the field of electronic devices, and in particular relates to a gesture sensing method and an electronic device. Background Technology

[0002] Traditional contact-based human-computer interaction refers to users interacting with computers using touch keyboards, mice, handwriting tablets, touchscreens, or physical buttons. With the development of human-computer interaction technology for smart terminals, users can control smart terminals through air gestures, i.e., contactless human-computer interaction.

[0003] In related technologies, users can open the camera app and control the camera to take photos using air gestures. This function requires continuous gesture recognition by the device. Furthermore, users also have air gesture needs in scenarios where touching the device screen is inconvenient, such as while driving or when their fingers are dirty.

[0004] However, in some scenarios, electronic devices sometimes fail to recognize user gestures, affecting the user experience. Improving the success rate of gesture recognition is a pressing issue that needs to be addressed. Summary of the Invention

[0005] This application provides a gesture sensing method and an electronic device, which enables the electronic device to implement gesture sensing function with low power consumption and to respond to user gestures more quickly when implementing the function.

[0006] A first aspect provides a gesture perception method applied to an electronic device, the electronic device including a camera, the method comprising: controlling the camera to operate in a first mode, the first mode being used to acquire images at a first resolution; acquiring a first image frame acquired by the camera in the first mode, and detecting the first image to obtain a first detection result, the first detection result being used to indicate whether a first valid gesture exists in the first image, the first valid gesture being any one of the starting gestures of one or more preset dynamic gestures; if a first valid gesture is detected from the first image, controlling the camera to switch from the first mode to a second mode, the second mode being used to acquire images at a second resolution, the first resolution being lower than the second resolution; acquiring a second image frame acquired by the camera in the second mode, and detecting the second image... A second detection result is obtained by performing detection. The second detection result is used to indicate whether a second valid gesture exists in the second image, and if so, the category of the second valid gesture. The second valid gesture is any one of a plurality of preset dynamic gestures as the starting gesture. The first image is interpolated to obtain a third image, and the third image is detected to obtain a third detection result. The third detection result is used to indicate the category of the first valid gesture. The resolution of the third image is the same as the resolution of the second image or the resolution of the third image is greater than a preset value. Based on the second and third detection results, it is determined whether to execute the gesture perception process. The gesture perception process is used to detect the target dynamic gesture and execute the preset operation corresponding to the target dynamic gesture. The target dynamic gesture is one of a plurality of preset dynamic gestures.

[0007] Based on the above scheme, the electronic device controls the camera to acquire images in a first mode to detect whether a valid gesture exists within the camera's range. Since the first mode is a low-resolution mode, the electronic device can continuously acquire images with low power consumption. After a valid gesture is detected in the first image (a frame acquired in the first mode), the electronic device switches the shooting mode and controls the camera to acquire a second image in a second mode to detect whether a valid gesture exists within the camera's range and the type of valid gesture. Simultaneously, the electronic device interpolates the first image to obtain a high-resolution third image and detects the type of valid gesture in the third image. Finally, based on the detection results corresponding to the second and third images, it determines whether to execute the gesture detection process. This scheme, by setting both high-resolution and low-resolution modes for the camera, automatically switches the camera mode at different detection stages to meet the image quality requirements of different detection stages, thus reducing device power consumption. Furthermore, this scheme can determine whether to trigger the gesture perception process with only two frames, ensuring a high detection success rate even if the user's gesture operation is relatively fast, achieving a balance between power consumption and effectiveness.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, interpolating the first image to obtain the third image includes: after determining that a second valid gesture exists in the second image based on the second detection result, interpolating the first image to obtain the third image.

[0009] Based on the above scheme, when the electronic device detects a second valid gesture from the second image, it interpolates the first image to obtain a third image. As an example, the electronic device can retrieve the first image from a cache. This implementation helps reduce power consumption. Specifically, after detecting the first valid gesture from the first image, the electronic device caches the first image and controls the camera to switch to the second mode to capture the second image. Since the first valid gesture detected in the first image may not be the gesture made by the user during an air gesture operation—for example, it might be the gesture made by the user unintentionally waving their hand across the camera—the second valid gesture may not be detected in the second image in such cases. Therefore, in this implementation, this step is only triggered when it is determined that a second valid gesture has been detected in the second image, which reduces unnecessary interpolation operations, saves resources, and lowers power consumption.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, interpolating the first image to obtain the third image includes: after determining that the first image includes the first valid gesture based on the first detection result, directly interpolating the first image to obtain the third image.

[0011] Based on the above scheme, when the electronic device detects a first valid gesture from the first image, it directly performs interpolation on the first image. This is because in some scenarios, the initial gesture for a preset dynamic gesture is a relatively specific gesture, and the probability of user error is low. Therefore, performing interpolation on the first image before detecting the first valid gesture can improve the efficiency of gesture perception and further increase the success rate of gesture perception when the user performs rapid gestures.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, detecting the first image to obtain a first detection result includes: inputting the first image into a first model and outputting the first detection result, wherein the first model is a binary classification model; detecting the second image to obtain a second detection result includes: inputting the second image into a second model and outputting the second detection result, wherein the second model is a multi-classification model; detecting the third image to obtain a third detection result includes: inputting the third image into the second model and outputting the third detection result.

[0013] Based on the above scheme, the electronic device performs valid gesture detection on the first image using a binary classification model. That is, the first detection result only displays two outcomes: the first image includes the first valid gesture, or the first image does not include the first valid gesture, without indicating the specific category of the first valid gesture. Compared to multi-classification models, binary classification models result in lower power consumption. Since the electronic device needs to continuously acquire images at a first resolution using a camera in the first mode and perform valid gesture detection on these images, the detection frequency is very high. This application effectively reduces the power consumption of the electronic device by using a binary classification model to detect images at the first resolution.

[0014] Based on the above scheme, the electronic device uses a multi-classification model to detect valid gestures in the second and third images. Taking the second detection result as an example: the second detection result can display multiple results, such as the second image not containing the second valid gesture, or the category of the second valid gesture in the second image being category #1, or the category of the second valid gesture in the second image being category #2, or the category of the second valid gesture in the second image being category #3, and so on. Compared with the binary classification model, the multi-classification model can output the specific category of the detected valid gesture, so that the electronic device can determine which initial gesture the user performed based on the detection result, in order to execute the subsequent gesture perception process.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, when the category of the second valid gesture is different from the category of the first valid gesture, the method further includes: acquiring n consecutive frames of images captured by the camera in the second mode, and sequentially detecting the n frames to obtain n fourth detection results, wherein the n fourth detection results correspond one-to-one with the n frames, and each fourth detection result is used to indicate whether the corresponding image has a third valid gesture, and the category of the third valid gesture when it exists, where n is an integer greater than or equal to 1; when the n fourth detection results indicate that there is no third valid gesture in the n frames, controlling the camera to switch from the second mode back to the first mode.

[0016] In the above scheme, when the categories of the first valid gesture and the second valid gesture are different, the electronic device re-acquires the high-resolution (i.e., second resolution) image captured by the camera in the second mode, and performs detection on the acquired high-resolution image to determine whether a valid gesture has been detected. If no valid gesture is detected after the number of detections exceeds a preset threshold (e.g., n), the camera is controlled to switch from the second mode back to the first mode to save power consumption.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the first image and the second image are two adjacent frames, or the first image and the second image are separated by one frame.

[0018] In the above scheme, one invalid frame is allowed between two valid frames. That is, in one implementation, if the electronic device acquires two consecutive frames via the camera, where the first frame is at a first resolution and the second frame is at a second resolution, and a valid gesture is detected in both frames, then the subsequent detection process is triggered. In another implementation, if the electronic device acquires three consecutive frames via the camera, where the first frame is at a first resolution and the second and third frames are at a second resolution, and a valid gesture is detected in the first and third frames, but not in the second frame, the subsequent detection process is still triggered. This approach is compatible with cases where the three consecutive frames are in the order of <valid low resolution, invalid high resolution, valid high resolution>. This method allows for compatibility with more scenarios and improves the detection success rate.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, when a first valid gesture is detected from the first image, the method further includes: caching the first image; after determining that the first detection result indicates that a second valid gesture exists in the second image, interpolating the first image to obtain a third image, including: after determining that the first detection result indicates that a second valid gesture exists in the second image, retrieving the first image from the cache, and then interpolating the first image to obtain the third image.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, if no first valid gesture is detected from the first image, the method further includes: discarding the first image.

[0021] In the above scheme, only the image where the first valid gesture is detected can be cached, instead of caching all images at the first resolution. This is because if an image does not contain a detected first valid gesture, subsequent detection processes will not use that image, so discarding it directly saves storage space. Furthermore, by caching the first image where the first valid gesture is detected, the gesture detection process can directly retrieve the first image from the cache.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, determining whether to execute a gesture perception process based on the first detection result and the second detection result includes: executing a gesture perception process when the second detection result indicates that the second image includes a second valid gesture, and the category of the second valid gesture is the same as the category of the first valid gesture indicated by the third detection result, wherein the first valid gesture and the second valid gesture are the starting gestures of the target dynamic gesture.

[0023] In the above scheme, the electronic device determines that the initial gesture has been detected through a two-frame confirmation mechanism. That is, the initial gesture is detected in both frames of the image before the subsequent gesture perception process is executed. This can reduce unnecessary detection in certain scenarios, such as when the user accidentally swipes his hand across the camera, thereby reducing power consumption.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, where the category of the second valid gesture indicated by the first detection result is the same as the category of the first valid gesture indicated by the second detection result, the method further includes: displaying an indicator icon, the indicator icon being used to prompt the user to continue inputting the target dynamic gesture.

[0025] This indicator icon prompts the user to perform other gestures after the electronic device has confirmed the initial gesture of the target dynamic gesture, thus improving the user experience. For example, after observing the indicator icon, the user knows that the electronic device has detected the initial gesture of the target dynamic gesture and is ready to detect other gestures within the target dynamic gesture. At this point, the user can continue to perform other gestures to trigger the preset operation corresponding to the target dynamic gesture.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, the target dynamic gesture includes a first gesture and a second gesture, wherein the first gesture is the starting gesture of the target dynamic gesture and the second gesture is the ending gesture of the target dynamic gesture; the gesture perception process includes: acquiring a fourth image frame captured by the camera in the second mode, and detecting the fourth image to obtain a fifth detection result, wherein the fifth detection result is used to indicate whether a fourth valid gesture exists in the fourth image, and the category of the fourth valid gesture when it exists, wherein the fourth valid gesture is any one of the ending gestures of one or more preset dynamic gestures; and when the fifth detection result indicates that the fourth valid gesture is the second gesture, controlling the execution of a preset operation corresponding to the target dynamic gesture.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the target dynamic gesture includes a third gesture, a fourth gesture, and a fifth gesture. The third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the intermediate gesture of the target dynamic gesture, and the fifth gesture is the ending gesture of the target dynamic gesture. The gesture perception process includes: acquiring a fifth image and a sixth image captured by the camera in the second mode, and detecting the fifth image and the sixth image respectively to obtain a sixth detection result and a seventh detection result. The sixth detection result is used to indicate whether there is a fifth valid gesture in the fifth image, and the category of the fifth valid gesture when it exists. The fifth valid gesture is any one of the intermediate gestures of one or more preset dynamic gestures. The seventh detection result is used to indicate whether there is a sixth valid gesture in the sixth image, and the category of the sixth valid gesture when it exists. The sixth valid gesture is any one of the ending gestures of one or more preset dynamic gestures. When the sixth detection result indicates that the fifth valid gesture is the fourth gesture, and the seventh detection result indicates that the sixth valid gesture is the fifth gesture, the preset operation corresponding to the target dynamic gesture is controlled to be executed.

[0028] The two solutions described above are compatible with both cases where the target dynamic gesture consists of two gestures and cases where it consists of three gestures, thus being compatible with various application scenarios. Furthermore, in this embodiment, triggering the gesture perception process refers to detecting other gestures in the target dynamic gesture besides the initial gesture when the first and second valid gestures are of the same category. If other gestures are detected, the preset operation corresponding to the target dynamic gesture is executed.

[0029] In conjunction with the first aspect, in some implementations of the first aspect, the first mode is also used to acquire images at a first frame rate, and the second mode is also used to acquire images at a second frame rate, wherein the first frame rate is lower than the second frame rate.

[0030] Based on the above scheme, the camera can dynamically adjust the frame rate of image acquisition to adapt to different detection needs. For example, in the first mode, the camera acquires images at a lower frame rate to detect whether there is a first valid gesture within the camera's range, and acquires images at a higher frame rate to identify whether there is a second valid gesture in the image and the specific category of the second valid gesture. In this way, the power consumption when acquiring images in the first mode can be further reduced.

[0031] In conjunction with the first aspect, in some implementations of the first aspect, controlling the camera to operate in a first mode includes: controlling the camera to operate in the first mode in response to an operation that enables gesture sensing functionality.

[0032] The above scheme limits the conditions for the camera to operate in the first mode. That is, after the gesture perception function is turned on, the camera is controlled to operate in the first mode. In other words, when the gesture perception function is not turned on, the camera can be turned off to reduce power consumption.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, before controlling the camera to operate in the first mode, the method further includes: detecting that the state of the electronic device meets a first condition, the first condition including at least one of the following: the screen state of the electronic device is on; the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor of the electronic device and the reflected light signal is greater than a first threshold, and / or, the signal strength of the reflected signal is less than a second threshold, and / or, the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0034] This solution further restricts the conditions under which the device can activate its camera. In addition to the user enabling the gesture recognition function, a first condition is added to prevent the camera from continuously capturing images when unnecessary, thereby reducing device power consumption. "Unnecessary" can be understood as any scenario where the camera is unlikely to capture a gesture.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, the electronic device is a foldable device, which includes a first screen and a second screen located on the front and back of the electronic device respectively in the unfolded state, wherein the first screen is a foldable screen, and the camera includes a first physical camera and a second physical camera, which are logical front-facing cameras corresponding to the first screen and the second screen respectively; controlling the camera to operate in a first mode includes: controlling the first physical camera to operate in the first mode when it is detected that the first screen is in a lit state and the electronic device is in an unfolded state or a hovered state; or, controlling the second physical camera to operate in the first mode when it is detected that the second screen is in a lit state and the electronic device is in a folded state.

[0036] This solution can be applied to foldable devices. If a device is in a folded or unfolded state and a corresponding screen (inner or outer screen) is on, the camera on the on screen can be turned on to detect whether there is a gesture within the camera's range, thus achieving gesture perception for foldable devices.

[0037] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: when the first physical camera is running in the first mode, if the electronic device is detected to switch from an unfolded state or a hovered state to a folded state, controlling the second physical camera to run in the first mode and turning off the first physical camera; when the second physical camera is running in the first mode, if the electronic device is detected to switch from a folded state to an unfolded state or a hovered state, controlling the first physical camera to run in the first mode and turning off the second physical camera.

[0038] In this solution, if the user changes the physical state of the device screen, such as from a folded state to an unfolded state or from an unfolded state to a folded state, the first image can be continuously captured by switching cameras, so that the device screen can still realize the function of intelligently sensing air gestures and controlling the device in the new physical state.

[0039] In a second aspect, an electronic device is provided, including a memory and a processor, the memory storing a computer program executable on the processor, wherein when the processor executes the computer program, the electronic device performs the steps of the method as described in any of the first aspects above.

[0040] Thirdly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the method as described in any of the first aspects above.

[0041] Fourthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects.

[0042] Fifthly, a chip system is provided, the chip system including a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the method described in any of the first aspects above.

[0043] The chip system can be a single chip or a chip module composed of multiple chips.

[0044] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0045] Figure 1 A schematic diagram of a gesture perception method 100 provided in an embodiment of this application;

[0046] Figure 2 A schematic diagram of a gesture perception method 200 provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of a gesture perception method 300 provided in an embodiment of this application;

[0048] Figure 4 A flowchart illustrating the specific implementation of the gesture perception method provided in this application embodiment;

[0049] Figure 5 A schematic diagram of an interface provided for an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of a foldable device provided in an embodiment of this application;

[0051] Figures 7 to 9 A schematic diagram of a preset dynamic gesture provided in an embodiment of this application;

[0052] Figure 10 A schematic diagram of the interface for a remote screenshot scenario applied in an embodiment of this application is shown;

[0053] Figure 11 A schematic diagram of the interface for a remote audio / video control scenario applied in an embodiment of this application is shown;

[0054] Figure 12 A schematic diagram of the interface for an image shrinking scenario applied in an embodiment of this application is shown;

[0055] Figure 13 A schematic diagram of the interface for an air-to-slide-up screen scenario applied in an embodiment of this application is shown;

[0056] Figure 14 A schematic diagram of the interface for a scenario where the screen is swiped downwards in the air, as described in an embodiment of this application, is shown.

[0057] Figure 15 A schematic diagram of dynamic frame adjustment provided in an embodiment of this application;

[0058] Figure 16 This is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of this application;

[0059] Figure 17 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0060] Figure 18 This is a schematic diagram of the structure of a system-on-chip (SoC) according to an embodiment of this application;

[0061] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0062] Figure 20 A flowchart illustrating a gesture perception method provided in an embodiment of this application;

[0063] Figure 21 A schematic diagram of a hand node provided in an embodiment of this application;

[0064] Figure 22 This is a flowchart illustrating a gesture perception method provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0066] To facilitate understanding of the solutions provided in the embodiments of this application, some terms used in this application will first be explained.

[0067] Frame rate refers to the number of images a camera captures or transmits per second, usually expressed in fps (frames per second). In this embodiment, the camera captures images at a first frame rate in a first mode and at a second frame rate in a second mode, where the first frame rate is less than the second frame rate. In some embodiments, the camera captures images at a third frame rate in a third mode, where the third frame rate is greater than the second frame rate.

[0068] Resolution, or image resolution, refers to the amount of information stored in an image. It is the number of pixels per inch of an image. Units of resolution include dpi (dots per inch) and ppi (pixels per inch).

[0069] A lightweight neural network is a relatively lightweight model that achieves performance comparable to heavier models, thus enabling hardware-friendly neural networks. Here, "lightweight" typically refers to the model's size or number of parameters. Common techniques for lightweight neural networks include distillation, pruning, quantization, weight sharing, low-rank decomposition, lightweight attention modules, dynamic network architectures / training methods, and lighter network architecture designs, etc., which are not limited in the embodiments of this application.

[0070] Background subtraction is a widely used method in moving target detection. Its basic idea is similar to inter-frame difference, both using difference operations between different images to extract the target region. However, unlike inter-frame difference, background subtraction does not subtract the current frame image from adjacent frames; instead, it subtracts the current frame image from a continuously updated background model, extracting the moving target from the difference image. In this embodiment, the moving target can be a user's hand. Based on background subtraction, changes in the user's gestures in consecutive image frames are detected to determine the gesture category.

[0071] Compared to traditional contact-based human-computer interaction, air gestures allow users to operate without their hands, providing a contactless human-computer interaction method that does not cause any inconvenience to users' gesture interaction. Air gestures include pointing, waving, clenching a fist, and rotating the palm, etc. Users can naturally use finger, wrist, and arm movements to express their interaction intentions in the air, which has the characteristics of a wide interaction space, greater flexibility, and a better interactive experience.

[0072] In some scenarios, users can use air gestures, such as spreading their palm, to control the camera to take photos or record videos after opening the camera app. However, as user needs continue to increase, the scenarios for using air gestures are becoming more and more numerous. For example, in one possible scenario, a user might be wearing gloves or have dirty fingers and not want to touch the phone screen, but still need to control the phone, such as a user watching short videos while eating crayfish. In another possible scenario, a user might be driving and, being some distance from the phone screen, need to access other apps, such as a user driving to their workplace and needing to clock in.

[0073] To meet users' needs for air gesture control in different scenarios, devices need to continuously recognize gestures, which places high demands on the device's camera and processor. Furthermore, the success rate of gesture recognition in related technologies still needs improvement; frequently, users perform gestures, but the electronic device fails to accurately recognize them, resulting in a poor user experience.

[0074] Therefore, a gesture sensing solution is urgently needed to reduce device power consumption while fulfilling gesture sensing functionality. The following section will combine... Figures 1 to 3 This application introduces a gesture perception method provided in its embodiments.

[0075] It should be noted that the gesture perception method provided in this application embodiment is applied to an electronic device including a camera. The electronic device can capture the gestures made by the user through the camera and perform corresponding preset operations based on the gestures made by the user.

[0076] In this application, unless otherwise specified, the camera of an electronic device refers to the logical front-facing camera. Here, "logical front-facing camera" is a concept relative to a physical camera. A physical camera refers to a physical camera installed on or connected to an electronic device, while a logical camera is a conceptual camera defined for different usage scenarios. Depending on the image or video capture direction of the physical camera when the user uses an application with image or video capture capabilities on the electronic device, logical cameras can be divided into logical front-facing cameras (e.g., cameras shooting outwards from the user's current display screen, i.e., front-facing shooting angle) and logical rear-facing cameras (e.g., cameras shooting inwards from the user's current display screen, i.e., rear-facing shooting angle). It is understood that in other embodiments, the logical camera may also include other types of logical cameras, such as a logical depth camera used for capturing three-dimensional images, determined according to the camera's function; this is not limited here.

[0077] The electronic devices described in the embodiments of this application may also be referred to as terminal equipment (UE), user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic devices can be mobile phones with displays, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, etc. This application does not limit the specific type of electronic device. For convenience, the accompanying drawings of this application use a mobile phone as an example for illustration, but this does not constitute a limitation on the embodiments of this application.

[0078] Figure 1 This is a schematic diagram of a gesture perception method 100 provided in an embodiment of this application. Figure 1 As shown, the electronic device continuously acquires images through a camera and performs image detection on the acquired images. Here, image detection refers to gesture category detection, that is, detecting whether there is a valid gesture in the image, and the specific category of the valid gesture when there is a valid gesture. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. Each preset dynamic gesture consists of one or more consecutive gestures, and the starting gesture refers to the first gesture among these multiple consecutive gestures.

[0079] If a valid gesture is detected in two consecutive image frames, the decision on whether to proceed with the subsequent gesture perception process is based on the detection results of these two frames. For example, in... Figure 1In the example shown, the electronic device captures a first image and a second image using a camera. If the detection results show that the category of a valid gesture in the first image is the same as the category of a valid gesture in the second image—for example, if both valid gestures are the starting hand gestures of a target dynamic gesture in a preset dynamic gesture—then the gesture perception process is executed, i.e., the detection of other gestures in the target dynamic gesture continues. After determining that a complete target dynamic gesture has been detected, the electronic device can perform a preset operation corresponding to that target dynamic gesture, such as swiping up or down on the interface, adjusting screen brightness, adjusting volume, adjusting vibration intensity, or launching a specific application.

[0080] In the above scheme, the electronic device can continuously capture and recognize user gestures in real time by continuously performing image acquisition and image detection operations. During this process, when the user makes the initial gesture of the target dynamic gesture, the electronic device uses a two-frame confirmation mechanism to determine that the initial gesture has been detected. That is, the initial gesture is detected in both frames of the image before proceeding to the subsequent gesture perception process. This reduces unnecessary detection in certain scenarios, such as when the user unintentionally swipes their hand across the camera, thereby reducing power consumption. Furthermore, this scheme only requires two frames of images (…). Figure 1 The first and second images in the image are used to trigger the subsequent gesture perception process. Since the time interval between acquiring two frames is usually short (e.g., 200ms), even if the user's gesture operation is fast, it can usually be captured, which improves the efficiency and success rate of gesture detection.

[0081] It should be noted that, in Figure 1 In the corresponding solution, to more accurately recognize user gestures, the electronic device needs to capture high-resolution images (greater than or equal to a preset value) via a camera, such as images of 320×240 pixels, to accurately detect the specific category of valid gestures in the first and second images. To meet the user's need for air gesture operation in different scenarios, the electronic device needs to continuously perform gesture recognition. However, continuous gesture recognition requires continuously capturing high-resolution images, which leads to high power consumption in the electronic device.

[0082] In view of this, Figure 2 A schematic diagram of another gesture sensing method 200 provided in an embodiment of this application is shown. Figure 2As shown, the electronic device controls the camera to operate in a first mode, continuously acquiring images and performing image detection on the acquired images. This image detection refers to valid gesture detection, that is, detecting whether a valid gesture exists in the image. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. This first mode is a low-resolution mode used to acquire low-resolution images; for example, it is used to acquire images at a first resolution, which is less than or equal to a preset value. Acquiring images in the first mode reduces the power consumption of the electronic device.

[0083] When the electronic device detects a valid gesture in a low-resolution image acquired through the first mode, it controls the camera to switch from the first mode to the second mode. Then, it determines whether to trigger the gesture recognition process based on two high-resolution images acquired in the second mode. This second mode is a high-resolution mode used to acquire high-resolution images; for example, it is used to acquire images with a second resolution greater than a preset value.

[0084] For example, in Figure 2 In the example shown, the electronic device acquires a first image frame captured by the camera in the first mode. If a valid gesture is detected from the first image, the electronic device controls the camera to switch from the first mode to the second mode and acquires a second image frame and a third image frame captured by the camera in the second mode.

[0085] The electronic device performs image detection on the second and third images. This detection refers to gesture category recognition, specifically detecting whether a valid gesture exists in the second and third images, and if so, the specific category of that valid gesture. The valid gesture refers to any one of the starting hand gestures of one or more preset dynamic gestures. If the detection result indicates that the category of the valid gesture in the second image is the same as the category of the valid gesture in the third image—for example, both valid gestures are the starting hand gestures of a target dynamic gesture in the preset dynamic gestures—then the gesture perception process is executed, meaning that other gestures in the target dynamic gesture are detected. After confirming that a complete target dynamic gesture has been detected, the electronic device can execute the preset operation corresponding to that target dynamic gesture, such as swiping up or down on the interface, adjusting screen brightness, adjusting volume, adjusting vibration intensity, or launching a specific application.

[0086] In the above scheme, the camera is configured with two modes: a first mode and a second mode, representing low-resolution and high-resolution modes, respectively. In general scenarios, the electronic device controls the camera to continuously acquire low-resolution (i.e., first resolution) images in the first mode and performs detection on the acquired images. If a valid gesture is detected during this process, the camera switches to the second mode. Because the images acquired in the first mode have relatively low resolution, it is difficult to accurately detect the type of valid gesture in the image. Therefore, by switching the camera to the second mode, high-resolution (i.e., second resolution) images acquired by the camera in the second mode can be obtained. In this scheme, the electronic device still uses a two-frame confirmation mechanism to determine if a starting gesture has been detected; that is, the starting gesture of the target dynamic gesture is detected in both high-resolution images before proceeding with subsequent processes. Since users typically only use the air gesture function a few times, continuously acquiring high-resolution images would waste resources. Therefore, this scheme, by acquiring low-resolution images in general scenarios and only acquiring high-resolution images when a valid gesture is detected, significantly reduces the device's power consumption compared to a scheme that continuously acquires high-resolution images.

[0087] As can be seen from the above description, in Figure 2 The illustrated scheme requires three image frames (first image, second image, and third image) to trigger the subsequent gesture recognition process. If the user's gesture is too fast, gesture recognition may fail. For example, it takes 300ms for the electronic device to capture three images through the camera, but if the user makes a gesture very quickly, with the initial gesture's duration less than 300ms, the initial gesture recognition will fail. Therefore, the subsequent gesture recognition process cannot be triggered, resulting in the user making a gesture, but the electronic device being unable to perform the corresponding operation, thus impacting the user experience.

[0088] In view of this, Figure 3 A schematic diagram of another gesture sensing method 300 provided in an embodiment of this application is shown. Figure 3 As shown, the electronic device controls the camera to operate in the first mode and continuously acquires images in the first mode, and performs image detection on the acquired images. Here, image detection refers to valid gesture recognition, that is, detecting whether there is a valid gesture in the image. The valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures.

[0089] When the electronic device detects a valid gesture in a low-resolution image acquired through the first mode, the electronic device controls the camera to switch from the first mode to the second mode. Figure 2The corresponding scheme is similar. The first mode is a low-resolution mode, used to acquire low-resolution images (such as the first resolution, which is less than or equal to a preset value). The second mode is a high-resolution mode, used to acquire high-resolution images (such as the second resolution, which is greater than a preset value). Then, the electronic device determines whether to trigger the gesture perception process based on the high-resolution image acquired in the second mode and the low-resolution image acquired in the first mode.

[0090] For example, in Figure 3 In the example shown, the electronic device acquires a first image frame captured by the camera in the first mode. If a valid gesture is detected from the first image, the electronic device controls the camera to switch from the first mode to the second mode and acquires a second image frame captured by the camera in the second mode.

[0091] The electronic device performs image detection on the second image, specifically gesture category detection. This involves detecting whether a valid gesture exists in the second image and, if so, the specific category of that valid gesture. A valid gesture refers to any one of the starting gestures of one or more preset dynamic gestures. If the detection result indicates the presence of a valid gesture in the second image, the first image is interpolated to obtain a high-resolution image, denoted as the third image. The resolution of the third image is the same as that of the second image, or the resolution of the third image is greater than a preset value. Then, image detection is performed on the third image, again specifically gesture category detection.

[0092] If the detection results indicate that the category of the valid gesture in the second image is the same as the category of the valid gesture in the third image—for example, if both valid gestures are the starting hand gestures of the target dynamic gesture in the preset dynamic gestures—then the gesture perception process is executed, that is, the detection of other gestures in the target dynamic gesture is continued. After determining that a complete target dynamic gesture has been detected, the electronic device can perform the preset operation corresponding to the target dynamic gesture, such as swiping up or down on the interface, adjusting screen brightness, adjusting volume, adjusting vibration intensity, or launching a specific application.

[0093] In the above scheme, the camera is configured with two modes: a first mode and a second mode, representing low-resolution and high-resolution modes, respectively. In general scenarios, the electronic device controls the camera to continuously acquire low-resolution (i.e., first resolution) images in the first mode and performs image detection on the acquired images. During this process, if a valid gesture is detected, the camera switches to the second mode. Because the images acquired in the first mode have relatively low resolution, it is difficult to accurately detect the type of valid gesture in the image. Therefore, by switching the camera to the second mode, a high-resolution (i.e., second resolution) image acquired by the camera in the second mode can be obtained. Since users typically only use the air gesture function occasionally, continuously acquiring high-resolution images would be a waste of resources. Therefore, this scheme significantly reduces the device's power consumption by acquiring low-resolution images in general scenarios and only acquiring high-resolution images when a valid gesture is detected, compared to a scheme that continuously acquires high-resolution images.

[0094] In this scheme, the electronic device still uses a two-frame confirmation mechanism to determine if a starting gesture has been detected. That is, the starting gesture of the target dynamic gesture is detected in both high-resolution images before proceeding to the next step. Compared to method 200, method 300 only requires acquiring two images to trigger the subsequent gesture perception process. This is because method 300 interpolates the first image (i.e., the low-resolution image) to obtain the third image (the high-resolution image), thus requiring only one more high-resolution image (the second image) to obtain two high-resolution images. This scheme not only maintains the gesture perception function of the electronic device with lower power consumption but also improves the efficiency of gesture perception. Even if the user's gesture operation is relatively fast, it can guarantee a high detection success rate, thus achieving a balance between power consumption and performance.

[0095] To further understand the gesture perception method provided in the embodiments of this application, the following is combined with... Figure 4 Method 400 in the document exemplifies the specific implementation process of this gesture perception method. It can be understood that Method 400 is a... Figure 3 Further explanation of method 300 in the text.

[0096] S401 controls the camera to operate in the first mode.

[0097] For example, the electronic device controls the camera to operate in a first mode, wherein the first mode is used to acquire images at a first resolution. As an example, the first resolution is lower than a preset value, that is, the first mode is a low-resolution mode, and the electronic device can acquire images through the camera in the first mode at a lower power consumption cost.

[0098] Therefore, when an electronic device controls a camera to operate in the first mode, it means that the electronic device controls the camera to continuously acquire images at a first resolution.

[0099] This application does not limit the conditions under which the electronic device is triggered to operate in the first mode. As an example, the electronic device controls the camera to operate in the first mode in response to the operation of activating the gesture sensing function. That is, the electronic device operates in the first mode after the gesture sensing function is activated.

[0100] It is understandable that "camera running in first mode" could mean that the camera is running in the background in first mode. For example, if the electronic device is currently running a third-party application and the user enables the gesture recognition function, the electronic device controls the camera to run in first mode.

[0101] The gesture sensing function described in this application refers to the function of an electronic device to sense and recognize user gestures and execute preset operations based on the recognition results. This gesture sensing function can also be called a gesture recognition function, a gesture operation function, etc., and this application does not limit it. For example, the electronic device may be pre-configured with a variety of preset dynamic gestures, each of which consists of one or more gestures (a description of the preset dynamic gestures can be found later). Figures 8-10 The corresponding descriptions are not elaborated here, and each preset dynamic gesture corresponds to one or more preset operations (such as swiping up or down the interface, adjusting screen brightness, adjusting volume, adjusting vibration intensity, or launching a specific application). Users can choose to enable or disable this gesture sensing function. When the gesture sensing function is enabled, the electronic device performs gesture detection in real time. When a target dynamic gesture is detected among the preset dynamic gestures, the electronic device executes the preset operation corresponding to that target dynamic gesture. The specific detection process can be found in the descriptions of sections S402 to S413 of this embodiment, which are not elaborated here.

[0102] It is understood that enabling the gesture recognition function can be a user's click operation in the system settings interface or a voice operation, etc., and this application embodiment does not limit this.

[0103] For example, Figure 5 This is a schematic diagram of an interface provided for an embodiment of this application. For example... Figure 5 As shown in (a) and (b), the user can choose to enable or disable the gesture sensing function in the system application's settings interface. In response to enabling the gesture sensing function, the electronic device controls the camera to operate in a first mode and continuously acquires images at a first resolution through the camera in the first mode.

[0104] In some embodiments, users can also select to sense one or more gestures in the settings interface. Each gesture has a preset corresponding operation. For example, gesture A is used to swipe down on the interface / page down, gesture B is used to swipe up on the interface / page up, gesture C is used to take a screenshot, and gesture D is used to pause / resume playback.

[0105] For example, such as Figure 5 As shown in (c) and (d), the user can choose to enable sensing gesture A. If the electronic device recognizes gesture A, it will perform the operation corresponding to gesture A: swiping down the interface / scrolling down the page. The user can also choose to enable sensing gesture E. If the electronic device recognizes gesture E, it will perform the operation corresponding to gesture E: calling the preset application.

[0106] In some embodiments, users can customize the application corresponding to gesture E, such as... Figure 5 As shown in (d), (e), and (f), the user selects gesture E in the gesture E settings interface to access application A. For example, gesture E is five fingers spread, and application A is an office application's attendance tracking mini-program. When the user arrives near the office by car, they can quickly launch the office application's attendance tracking mini-program using the air gesture E to complete their attendance tracking, achieving gesture recognition and improving the user's mobile experience.

[0107] In some embodiments, the conditions for the electronic device to control the camera to operate in a first mode further include a first condition, which includes at least one of the following: the screen of the electronic device is in a bright state; the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor of the electronic device and the reflected light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the screen of the electronic device is facing a preset direction; and the electronic device is running a preset application.

[0108] In this embodiment, if the electronic device has enabled gesture sensing and meets the first condition described above, the camera of the electronic device is triggered to continuously capture the first image. By adding the first condition, the camera of the electronic device is prevented from continuously capturing the first image when unnecessary, thereby further reducing the device's power consumption.

[0109] Based on the foregoing embodiments, the following provides a detailed description of various triggering methods for controlling the camera to operate in the first mode (or controlling the camera to continuously acquire images of the first resolution in the first mode).

[0110] In one possible implementation, if the electronic device has the gesture sensing function enabled, it triggers the electronic device to control the camera to operate in the first mode.

[0111] In one possible implementation, if the electronic device has gesture sensing enabled, the screen status of the electronic device is detected. If the screen is on, the electronic device is triggered to control the camera to operate in a first mode. The interface displayed when the electronic device's screen is on includes, for example, the lock screen, the main screen, and third-party application interfaces.

[0112] In one possible implementation, if the electronic device has the gesture sensing function enabled, it detects whether the electronic device is unlocked. If the electronic device is unlocked, it triggers the electronic device to control the camera to operate in the first mode.

[0113] In one possible implementation, if the electronic device has the gesture sensing function enabled, it detects whether the light signal emitted by the proximity sensor of the electronic device is blocked. If it is determined that the light signal emitted by the proximity sensor is not blocked, it triggers the electronic device to control the camera to operate in the first mode.

[0114] As an example, if the time difference between the light signal emitted by the proximity sensor of an electronic device and the reflected light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the proximity sensor does not receive the reflected signal, it can be determined that the light signal emitted by the proximity sensor is not blocked.

[0115] Understandably, if a user makes or receives a call through the handset, or if the electronic device screen is upside down, or if the electronic device is in a handbag or pocket, the light signal emitted by the light sensor near the electronic device will be blocked, and the image captured by the front camera will not be able to detect the gesture. In this case, the camera can stop continuously capturing images to reduce device power consumption.

[0116] In one possible implementation, if the electronic device has gesture sensing enabled, it checks whether the detection data from the ambient light sensor is greater than a third threshold. If the ambient light sensor's detection data is determined to be greater than the third threshold, the electronic device is triggered to control the camera to operate in a first mode. Here, the detection data primarily refers to ambient light brightness. It should be understood that if the ambient light sensor's detection data is greater than the third threshold, it indicates that the electronic device is not in a dark environment, such as when the device is in a pocket or during nighttime.

[0117] In one possible implementation, if the electronic device has gesture sensing enabled, the orientation of the electronic device's screen is detected. If the screen is facing a preset direction, the electronic device is triggered to control the camera to operate in a first mode. In this implementation, the preset direction includes the direction in which the user holds the electronic device to browse the interface content, the direction in which the electronic device is placed on a table to browse the interface content, etc. This direction can be determined by detecting the attitude data of the electronic device, wherein the attitude data includes pitch angle, yaw angle, and roll angle.

[0118] In one possible implementation, if the electronic device has gesture sensing enabled, it detects whether the electronic device is running a preset application. If it is determined that the electronic device is running a preset application, it triggers the electronic device to control the camera to operate in a first mode. For example, if the preset application is a short video application, and the electronic device detects that the user is watching a short video, it triggers the camera to continuously capture a first image to detect whether the user is using a preset air gesture, thus satisfying the user's need to view short videos remotely. As another example, if the preset application is a navigation application, and the electronic device detects that the user is using driving navigation in a navigation application, it triggers the camera to continuously capture a first image to detect whether the user is using a preset air gesture, thus satisfying the user's need to listen to traffic updates remotely. This application does not limit the preset application. Furthermore, it should be noted that running a preset application includes displaying the interface of the preset application on the current screen of the electronic device, or the preset application running in the background.

[0119] In one possible implementation, if the electronic device has enabled the gesture sensing function and determines that at least two of the following conditions are met: the screen of the electronic device is on, the electronic device is unlocked, the light signal emitted by the proximity sensor of the electronic device is not blocked, and the electronic device is running a preset application, then the electronic device is triggered to control the camera to operate in the first mode.

[0120] In another embodiment, if the electronic device has gesture sensing enabled and is detected to be in motion, the camera of the electronic device can be triggered to capture an image at a first resolution. Conversely, if the electronic device has gesture sensing enabled and is detected to be stationary, the camera can be turned off to reduce device power consumption. The electronic device being in motion includes scenarios such as a user using navigation while driving a vehicle or while cycling.

[0121] In another embodiment, if the electronic device has gesture sensing enabled and the device is detected to be stationary, the gesture camera can be triggered to capture an image at a first resolution to detect potential user needs for air control. The electronic device being stationary includes, for example, the device being placed on a table.

[0122] In one possible implementation, the electronic device in this application embodiment can be a foldable device, such as a mobile phone with a foldable screen. The foldable device has multiple screens and multiple cameras, and can determine which camera to control to operate in a first mode based on its specific physical form.

[0123] The following is in conjunction with the appendix Figure 6 This paper provides a detailed explanation of the gesture sensing method for foldable devices.

[0124] For example, Figure 6 This is a schematic diagram of the structure of a foldable device provided in an embodiment of this application. Figure 6 As shown, the foldable device's screen includes a first screen and a second screen. The first screen is a foldable screen that can be folded inwards. Therefore, when the electronic device is in a folded state, the first screen is the inner screen of the electronic device, and the second screen is the outer screen. The virtual axis along which the foldable screen is located is the folding edge, and the first screen can be folded inwards along this folding edge. The included angle α in the figure represents the folding angle of the first screen.

[0125] The foldable device has three physical states: unfolded, hovered, and folded. The unfolded state refers to the state where the first screen is unfolded to its maximum angle, such as... Figure 6 As shown in (a) and (b) in the image. The folded state refers to the state after the first screen is folded, as shown in the image. Figure 6 As shown in (d) in the diagram. The hovering state refers to a state between the unfolded and folded states, such as... Figure 6 As shown in (c) in the figure.

[0126] The first screen includes a first physical camera, which serves as the logical front-facing camera of the foldable device when it is in the unfolded state. The second screen includes a second physical camera, which serves as the logical front-facing camera of the foldable device when it is in the folded or hovered state. The back panel includes a third physical camera, which serves as the logical rear-facing camera of the foldable device when it is in the unfolded or hovered state.

[0127] As an example, a foldable device includes a first screen and a second screen located on the front and back of the electronic device respectively in the unfolded state, wherein the first screen is a foldable screen, and the camera includes a first physical camera and a second physical camera, which are logical front-facing cameras corresponding to the first screen and the second screen respectively.

[0128] As an example, if the second screen of the electronic device is detected to be on and the electronic device is in a folded state, the second physical camera is controlled to operate in a first mode. As another example, if the second screen of the electronic device is detected to be on and the electronic device is in an unfolded or hovered state (referring to a state between a folded and unfolded state), the second physical camera is controlled to operate in the first mode. Based on these two examples, in some embodiments, before controlling the second physical camera to operate in the first mode, the following steps are further included: detecting that the state of the electronic device satisfies at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor (on the second screen) of the electronic device and the reflected light signal is greater than a first threshold, and / or, the signal strength of the reflected signal is less than a second threshold, and / or, the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the second screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0129] As an example, if the first screen of the electronic device is detected to be on, and the electronic device is in an unfolded or hovered state, the first physical camera is controlled to operate in a first mode. Before controlling the first physical camera to operate in the first mode, the method further includes: detecting that the state of the electronic device satisfies at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor of the electronic device (on the first screen) and the reflected light signal is greater than a first threshold, and / or, the signal strength of the reflected signal is less than a second threshold, and / or, the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the first screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0130] Based on the above examples, before controlling the first physical camera to operate in the first mode, or controlling the second physical camera to operate in the first mode, the method further includes: detecting that the state of the electronic device meets a second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor of the electronic device and the reflected light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the first screen or the second screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0131] As an example, when the second physical camera is running in the first mode, if the electronic device detects that it has switched from a folded state to an unfolded state or a hovering state, the electronic device controls the first physical camera to run in the first mode and turns off the second physical camera.

[0132] As an example, when the first physical camera is running in the first mode, if the electronic device detects that it has switched from an unfolded or hovered state to a folded state, the electronic device controls the first physical camera to turn off and controls the second physical camera to run in the first mode; or, the electronic device controls the first physical camera to turn off.

[0133] It should be noted that for other types of foldable devices implementing gesture sensing methods, please refer to [the relevant documentation / reference]. Figure 6 The foldable devices shown are implemented in a similar principle and have similar technical effects. This application does not impose any restrictions on the structural style of the foldable devices.

[0134] Optionally, the first mode is also used to acquire images at a first frame rate, where the first frame rate is less than or equal to a preset frame rate. This approach can further reduce the power consumption of the electronic device when the camera acquires images in the first mode.

[0135] S402, acquire a first frame of the first image captured by the camera in the first mode.

[0136] S403, perform detection on the first image to obtain the first detection result.

[0137] For example, after the electronic device controls the camera to operate in a first mode, it continuously acquires images of a first resolution through the camera in the first mode, and performs valid gesture detection on the acquired images, that is, detects whether there are valid gestures in the images.

[0138] This application embodiment uses a first image captured by an electronic device through a camera as an example for illustration, wherein the first image is a single frame. After acquiring the first image, the electronic device performs detection on the first image to obtain a first detection result. The first detection result is used to indicate whether a first valid gesture exists in the first image. The first valid gesture is any one of the starting gestures of one or more preset dynamic gestures.

[0139] In this embodiment, the preset dynamic gestures are either default configurations or user-defined dynamic gestures. Each preset dynamic gesture corresponds to one or more preset operations, such as swiping up or down the interface, adjusting screen brightness, adjusting volume, adjusting vibration intensity, or launching a specific application.

[0140] In this application embodiment, "target dynamic gesture" specifically refers to the preset dynamic gesture detected by the electronic device. That is, the target dynamic gesture is one of the preset dynamic gestures. Therefore, the description of the preset dynamic gesture in this application embodiment also applies to the target dynamic gesture.

[0141] Each preset dynamic gesture consists of one or more consecutive gestures. The starting gesture of a preset dynamic gesture is the first gesture among these one or more gestures, the ending gesture of a preset dynamic gesture is the last gesture among these one or more gestures, and the intermediate gesture of a preset dynamic gesture is the gesture between the starting gesture and the ending gesture (if any).

[0142] For example, Figures 7 to 9 This is a schematic diagram of a preset dynamic gesture provided in an embodiment of this application. The preset dynamic gesture and its application scenarios will be described in detail below with reference to the accompanying drawings.

[0143] In one possible implementation, the preset dynamic gesture includes only the first gesture, meaning the preset dynamic gesture contains only one gesture. In this case, the start and end gestures of the preset dynamic gesture both refer to the first gesture, and the preset dynamic gesture has no intermediate gestures. An example is provided below.

[0144] For example, in Figure 7 In the example shown in (a), the first gesture corresponds to the action of spreading five fingers. When the electronic device recognizes the gesture of spreading five fingers, it can execute a preset operation corresponding to that gesture. For example, if the current screen state of the electronic device is on and the interface is any, the electronic device will capture the current interface after recognizing the gesture of spreading five fingers. Another example is if the current interface of the electronic device is a selfie interface; after recognizing the user's gesture of spreading five fingers, the electronic device can control the camera to capture an image or start / pause video recording. Yet another example is if the current screen state of the electronic device is off; after recognizing the gesture of spreading five fingers, the electronic device will light up the screen to check the time / messages, or open an office application and perform a work attendance check-in task. Yet another example is if the current interface of the electronic device is a navigation interface; after recognizing the gesture of spreading five fingers, it can automatically broadcast real-time traffic conditions.

[0145] For example, Figure 7 The gesture shown in (b) is extending the index finger. For example, if the current screen of the electronic device is a short video screen, the electronic device will like or favorite the short video after recognizing the gesture. As another example, if the current screen of the electronic device is a selfie screen, the electronic device will start a 3-second countdown to record after recognizing the gesture.

[0146] In one possible implementation, the preset dynamic gesture includes a first gesture and a second gesture, where the first gesture is the starting gesture of the gesture change and the second gesture is the ending gesture of the gesture change.

[0147] For example, in Figure 8In the example shown in (a), the first gesture corresponds to an open palm, and the second gesture corresponds to a clenched fist. That is, the preset dynamic gesture in this example includes the dynamic gesture process of the open palm changing into a clenched fist; that is, the starting gesture is the palm facing the front-facing camera of the electronic device, and the ending gesture is a clenched fist. For example, if the screen of the electronic device is on and the interface is arbitrary, after the electronic device recognizes two consecutive gestures, the first gesture and the second gesture, it executes the preset operation corresponding to the preset dynamic gesture. As an example, this preset operation could be a screenshot. Figure 10 This illustration shows an interface diagram of an air screenshot scenario applied in an embodiment of this application. In this scenario, when the electronic device detects the user's gesture of changing from an open palm to a clenched fist (e.g., ...), ... Figure 10 (a) in Figure 10 If (b) is shown in the image, a screenshot operation will be performed on the current screen to obtain a screenshot image (such as...). Figure 10 (as shown in (c)).

[0148] For example, in Figure 8 In the example shown in (b), the first gesture corresponds to an open palm, and the second gesture also corresponds to an open palm. However, the distance between the first gesture and the electronic device is greater than the distance between the second gesture and the electronic device. In other words, the preset dynamic gesture in this example is a palm pressing forward on the electronic device. The preset operation corresponding to this gesture can be pause or resume playback. For example, if the current interface of the electronic device is a video / music playback interface and is currently in playback mode, the electronic device will pause playback of the video / music after recognizing the gesture; if the current interface of the electronic device is a video / music playback interface and is currently in pause mode, the electronic device will resume playback of the video / music after recognizing the gesture. Figure 11 The diagram illustrates an interface schematic of a hands-free audio / video control scenario applied in an embodiment of this application. In this scenario, the electronic device's display is currently playing a video. When the electronic device detects an operation performed by the user—spreading their palm and pressing forward (i.e., towards the electronic device)—[the user then presses the screen with their hand outstretched]. Figure 11 (a) and Figure 11 As shown in (b) above, the currently playing video will be paused (e.g., ...). Figure 11 (as shown in (c)).

[0149] For example, in Figure 8In the example shown in (c), the first gesture corresponds to the action of spreading two fingers, that is, spreading the thumb and index finger while making a fist with the other fingers; the second gesture corresponds to the action of pinching two fingers together, that is, pinching the thumb and index finger together while making a fist with the other fingers. In other words, the preset dynamic gesture in this example is the action of two fingers moving from spreading to pinching. For example, if the current display interface of the electronic device shows an image, the electronic device will shrink the image after recognizing the action of pinching two fingers together. Figure 12 This diagram illustrates an interface schematic of a scenario where an image is zoomed out remotely using an embodiment of this application. In this scenario, the electronic device's display is currently showing an image of a puppy. When the electronic device detects a user's gesture of changing from spreading two fingers to pinching two fingers together (e.g., ...), the image zooms out remotely. Figure 12 (a) and Figure 12 As shown in (b) in the image, the puppy image currently displayed on the screen will be scaled down (e.g., ...). Figure 12 (as shown in (c)).

[0150] For example, in Figure 8 In the example shown in (d), the first gesture corresponds to a pinching motion with two fingers together (thumb and index finger pinched together, the other fingers clenched into a fist); the second gesture corresponds to a spreading motion with two fingers apart (thumb and index finger spread apart, the other fingers clenched into a fist). In other words, the preset dynamic gesture in this example is the movement of two fingers moving from pinching to spreading. For example, if the current electronic device's display screen shows an image, the electronic device will enlarge the image after recognizing the spreading-finger gesture.

[0151] In one possible implementation, the target dynamic gesture includes a first gesture, a second gesture, and a third gesture, wherein the first gesture is the starting gesture of the gesture change, the second gesture is the intermediate gesture of the gesture change, and the third gesture is the ending gesture of the gesture change.

[0152] For example, Figure 9 The gesture shown in (a) is an upward swipe gesture. The first gesture involves the back of the hand facing the front-facing camera of the electronic device; the second gesture involves the palm being flat; and the third gesture involves the palm facing the front-facing camera. In other words, the gesture progresses from bending downwards to flattening the palm and then turning the palm towards the camera. For example, if the electronic device is currently displaying a data stream interface, recognizing this gesture will scroll down a fixed distance. Another example is if the electronic device is currently displaying a page, recognizing this gesture will turn down one page. Yet another example is if the electronic device is currently playing a video, recognizing this gesture will increase the volume or brightness. Still another example is if the electronic device is currently displaying a short video interface, recognizing this gesture will play the next short video. Figure 13This illustration shows a schematic diagram of an interface in a scenario where the user swipes the screen upwards in the air, as described in an embodiment of this application. In this scenario, the electronic device's display is currently showing an article. When the electronic device detects a user's gesture (e.g., bending the hand downwards, flattening the palm, and then flipping the palm towards the camera), the user swipes the screen upwards. Figure 13 (a) Figure 13 As shown in (b) and Figure 13 As shown in (c) in the figure, the content displayed on the screen will be slid upwards (as shown in the figure). Figure 13 (as shown in (d)).

[0153] For example, Figure 9 The gesture shown in (b) is a swipe-down motion. The first gesture involves the palm facing the front-facing camera of the electronic device; the second gesture involves the palm flat; and the third gesture involves the back of the hand facing the front-facing camera. That is, the gesture progresses from the palm facing the camera to the palm folding down perpendicular to the screen, and then the back of the hand facing the camera. For example, if the electronic device is currently displaying a data stream interface, recognizing this gesture will scroll up a fixed distance. Another example is if the electronic device is currently displaying a page, recognizing this gesture will scroll up one page. Yet another example is if the electronic device is currently playing a video, recognizing this gesture will lower the volume or brightness. Still another example is if the electronic device is currently displaying a short video interface, recognizing this gesture will play the previous short video.

[0154] Figure 14 This illustration shows a schematic diagram of an interface in an embodiment of this application, depicting a scenario where the electronic device's display is currently playing a short video. The electronic device detects a user's hand movement from facing the front-facing camera of the device to patting the screen downwards (e.g., the palm of the hand is perpendicular to the screen). Figure 14 (a) and Figure 14 (as shown in (b)) to the gesture of bending the back of the hand towards the front camera (as shown in (b)). Figure 14 If (c) is displayed, then switch to the previous short video for playback (e.g., ...). Figure 14 (as shown in (d)).

[0155] As can be seen from the above embodiments, users can achieve diverse control of electronic devices through preset air gestures in different application scenarios. These application scenarios include, for example, when the electronic device is in a blackout state, when the user is watching a video or short video, when the user is browsing a webpage or photo album, when the user is driving and navigating, when the user is taking pictures or recording videos, etc.

[0156] It should be noted that the application scenarios of the gesture perception method are not limited in the embodiments of this application. In addition to the application scenarios described in the above embodiments, it can also be applied to any other scenario in which electronic devices can be controlled by air gestures.

[0157] It should be noted that this application does not limit the specific implementation of detecting the first image to obtain the first detection result. In one possible example, the electronic device inputs the first image into a first model and outputs the first detection result, wherein the first model is a binary classification model, that is, the first detection result only displays two results: either the first image includes the first valid gesture, or the first image does not include the first valid gesture, without indicating the specific category of the first valid gesture. Compared with multi-classification models, binary classification models result in lower power consumption, while the electronic device needs to continuously acquire images of the first resolution through the camera in the first mode and perform valid gesture detection on these images of the first resolution, thus the detection frequency is very high. This application can effectively reduce the power consumption of the electronic device by detecting images of the first resolution using a binary classification model. Exemplarily, the first model can be a neural network model.

[0158] S404, determine whether the first valid gesture has been detected.

[0159] For example, after acquiring the first image and performing detection to obtain the first detection result, it is determined whether a first valid gesture was detected from the first image based on the first detection result.

[0160] If a first valid gesture is detected from the first image, optionally, in step S405, the electronic device caches the first image. Since the first image is an image with a first resolution (i.e., a low-resolution image), the cache space occupied when caching the first image is relatively small, which can save storage space.

[0161] It is understandable that in some application scenarios, the first image may be a single frame shared by multiple services; that is, the first image is a second-resolution image, rather than a first-resolution image captured in the first mode. To make the solution provided in this application applicable to such scenarios, the first image can be downsampled to change its resolution from the second resolution to the first resolution. This allows the first model to be used to detect the downsampled first image. In this implementation, the high-resolution first image can be directly cached without performing subsequent interpolation operations (see subsequent descriptions for information on interpolation operations).

[0162] S406 controls the camera to switch from the first mode to the second mode.

[0163] For example, upon detecting a first valid gesture from a first image, the electronic device controls the camera to switch from a first mode to a second mode. The second mode is used to capture images at a second resolution, where the first resolution is lower than the second resolution. For example, the first resolution can be set to 120×160, and the second resolution can be set to 320×240.

[0164] In other words, the second resolution is high resolution, and the second mode is high resolution mode. When the electronic device detects a first valid gesture from the first image, it means that the user may be making a gesture in the air within the camera's field of view to use the gesture recognition function. However, because the resolution of the first image (i.e., the first resolution) is relatively low (below the preset value), it is difficult to detect the specific category of the user's gesture. Therefore, the electronic device controls the camera to switch to high resolution mode (i.e., the second mode) to acquire a high resolution image. With the high resolution image, the electronic device can more accurately detect the category of the user's gesture, improving the success rate of detection.

[0165] Optionally, the second mode is also used to acquire images at a second frame rate, where the first frame rate is lower than the second frame rate. For example, the first frame rate can be set to 2fps, and the second frame rate can be set to 5fps. This scheme increases the frame rate of the camera when acquiring high-resolution images, that is, it increases the speed at which the camera acquires high-resolution images per unit time. Therefore, when a user makes rapid gestures, the electronic device can also capture the user's gestures as much as possible, improving the success rate of detection. In this scheme, the switching between different operating modes of the camera is closely related to the switching of the image frame rate. The following explains how to adjust the frame rate of the camera's image acquisition.

[0166] For example, Figure 15 This is a schematic diagram illustrating dynamic frame adjustment as provided in an embodiment of this application. Figure 15 As shown in (a), in one possible frame rate adjustment scheme, the frame rate of image capture / transmission by the camera is adjusted by regulating the clock frequency. Specifically, data transmission is based on clock changes, typically capturing / transmitting data on the rising or falling edge of the clock. If the frame rate of image capture by the camera needs to be doubled, the clock frequency can be doubled, and the corresponding data transmission speed will also double; if the frame rate of image capture by the camera needs to be halved, the clock frequency can be halved, and the corresponding data transmission speed will also be halved. Figure 15 As shown in (b), in one possible frame rate adjustment scheme, the frame rate of the camera capturing / transmitting images is adjusted by regulating the ratio of the valid area to the blanking area. Specifically, the frame rate can be adjusted by regulating the configuration of the blanking area length. Increasing the length of the blanking area can decrease the frame rate, and conversely, decreasing the length of the blanking area can increase the frame rate.

[0167] It should be noted that, since the first frame adjustment scheme described above requires turning off the clock and powering on again to modify the clock frequency, it cannot achieve dynamic adjustment between frames. Therefore, the second frame adjustment scheme described above is adopted in this application embodiment to achieve the purpose of dynamic frame adjustment.

[0168] S407, acquire a second image frame captured by the camera in the second mode.

[0169] S408, perform detection on the second image to obtain the second detection result.

[0170] For example, after the electronic device controls the camera to switch from the first mode to the second mode, it captures images through the camera in the second mode and performs gesture category detection on the captured images, that is, it detects whether there is a valid gesture in the image, and the category of the valid gesture when there is a valid gesture.

[0171] This application embodiment uses a second image acquired by an electronic device through a camera as an example for illustration, wherein the second image is a single frame image. After acquiring the second image, the electronic device performs detection on the second image to obtain a second detection result. The second detection result is used to indicate whether a second valid gesture exists in the second image, and if a second valid gesture exists, the type of the second valid gesture. The second valid gesture is any one of the starting gestures of one or more preset dynamic gestures.

[0172] by Figure 8 and Figure 9 The dynamic gestures shown are examples of preset dynamic gestures: there are a total of 6 preset dynamic gestures, each of which corresponds to a starting gesture, for example, Figure 8 The starting gestures of the preset dynamic gestures shown in (a) and (b) are both gestures with the palms outstretched and facing the electronic device. Figure 8 The starting gesture of the preset dynamic gesture shown in (c) is a gesture with two fingers spread and the other fingers clenched into a fist. Figure 8 The starting gesture of the preset dynamic gesture shown in (c) is a gesture with two fingers pinched together and the other fingers clenched into a fist. Figure 9 The preset dynamic gesture shown in (a) is a gesture with the palm open and the back of the hand facing the electronic device; Figure 9 The preset dynamic gesture shown in (b) is a gesture with the palm open and facing the electronic device.

[0173] Therefore, Figure 8 (a) Figure 8 (b) and Figure 9 The preset dynamic gestures shown in (b) all have the same starting gesture, meaning their starting gesture categories are consistent. In other words, in Figure 8 and Figure 9Of the six preset dynamic gestures shown, one includes four starting gestures.

[0174] Detecting the second image refers to detecting whether the second image contains any of the preset dynamic gestures' starting gestures, and if so, the category of that starting gesture. Combined with... Figure 8 and Figure 9 Taking the preset dynamic gesture shown as an example: Detecting the second image means detecting whether the second image includes... Figure 8 and Figure 9 Any of the four starting gestures shown, and if a starting gesture exists, that starting gesture is Figure 8 and Figure 9 Which of the four starting gestures is being described? In other words, the category of the starting gesture in this application indicates which type of starting gesture it is. In a specific implementation, each category of starting gesture can be assigned a number, combined with... Figure 8 and Figure 9 Taking the four starting gestures shown as examples: assuming Figure 8 (a) Figure 8 (b) and Figure 9 The starting gesture shown in (b) is denoted as category #1 (represented by "00"). Figure 8 The starting gesture shown in (c) is denoted as category #2 (represented by "01"). Figure 8 The starting gesture shown in (d) is denoted as category #3 (represented by "10"). Figure 9 The initial gesture shown in (a) is denoted as category #4 (represented by "11"). After detecting the second image, the category of the second valid gesture in the second image can be represented by a number. In other words, the second detection result obtained by detecting the second image can be directly represented by the category number of the detected second valid gesture.

[0175] It should be noted that this application does not limit the specific implementation method of detecting the second image to obtain the second detection result. In one possible example, the electronic device inputs the second image into a second model and outputs a second detection result. The second model is a multi-classification model, meaning that the second detection result can display multiple results: the second image does not include the second valid gesture, or the category of the second valid gesture in the second image is category #1, or the category of the second valid gesture in the second image is category #2, or the category of the second valid gesture in the second image is category #3, and so on. Compared to a binary classification model, a multi-classification model can output the specific category of the detected valid gesture, allowing the electronic device to determine which initial gesture the user performed based on the detection result, in order to execute subsequent gesture perception processes. For example, the second model can be a neural network model.

[0176] S409, determine whether a second valid gesture has been detected.

[0177] For example, after acquiring the second image and performing detection to obtain the second detection result, it is determined whether a second valid gesture was detected from the second image based on the first detection result.

[0178] If the electronic device detects a second valid gesture from the second image, it triggers the execution steps S410 to S413.

[0179] If the electronic device fails to detect the second valid gesture from the second image, in one possible implementation, the electronic device discards the cached first image and restarts gesture detection. For example, the electronic device re-controls the camera to switch back to the first mode and re-executes the scheme provided in this application from step S401 (this example is denoted as scheme A); or, for another example, the electronic device continues to control the camera in the second mode and continuously captures images in the second mode. In this case, the scheme executed by the electronic device is the same as... Figure 1 The method 100 described is similar and will not be repeated here (this example will be referred to as scheme B). In another possible implementation, the electronic device retains the cached first image and returns to step S407 to reacquire a second frame, then triggers the execution of S408 and S409. If a second valid gesture is detected in the reacquired second image, the subsequent steps S410 to S413 are triggered. If no second valid gesture is detected, the electronic device discards the cached first image and restarts the gesture detection (as described in scheme A or scheme B above).

[0180] In the above scheme, an invalid frame is allowed between two valid frames. That is, in one implementation, if the electronic device acquires two consecutive frames of images through the camera, where the first frame is an image with a first resolution and the second frame is an image with a second resolution, and a valid gesture is detected in both frames, then subsequent steps S410 to S413 are triggered. In another implementation, if the electronic device acquires three consecutive frames of images through the camera, where the first frame is an image with a first resolution, and the second and third frames are images with a second resolution, and a valid gesture is detected in the first and third frames, but not in the second frame, then subsequent steps S410 to S413 are still triggered, i.e., it is compatible with the case where the three consecutive frames are <valid low resolution, invalid high resolution, valid high resolution> in sequence.

[0181] In summary, in the embodiments of this application, the first image and the second image used to trigger subsequent steps S410 to S413 are two adjacent frames, or, in the embodiments of this application, the first image and the second image used to trigger subsequent steps S410 to S413 are separated by one frame.

[0182] S410, interpolate the first image to obtain the third image.

[0183] For example, the electronic device interpolates the first image to obtain a third image, wherein the resolution of the third image is the same as that of the second image (i.e., the second resolution), or the resolution of the third image is greater than a preset value. In other words, the electronic device up-interpolates the low-resolution image (the first image at the first resolution) that detects a valid gesture to obtain a high-resolution image.

[0184] In one possible implementation, step S410 is triggered by step S409, that is, when the electronic device detects a second valid gesture from the second image, it retrieves the first image from the cache and then performs interpolation processing on the first image to obtain the third image. This implementation is beneficial for reducing power consumption. Specifically, after the electronic device detects the first valid gesture from the first image, it caches the first image and controls the camera to switch to the second mode to capture the second image. Since the first valid gesture detected in the first image may not be the gesture made by the user when performing an air gesture operation, for example, the first valid gesture may be the gesture made by the user when unintentionally waving their hand across the camera, in this case, the second valid gesture may not be detected in the second image. Therefore, in this implementation, S410 is triggered only when it is determined that a second valid gesture has been detected from the second image, which can reduce unnecessary interpolation operations, save resources, and reduce power consumption.

[0185] In another possible implementation, step S410 is triggered by step S404, that is, when the electronic device detects the first valid gesture from the first image, it directly performs an interpolation operation on the first image. This is because in some scenarios, the initial gesture of the preset dynamic gesture is a relatively special gesture, and the probability of user error is relatively small. Therefore, performing interpolation processing on the first image in advance after detecting the first valid gesture can improve the efficiency of gesture perception and further improve the success rate of gesture perception when the user performs a rapid gesture.

[0186] It is understandable that since the third image is obtained by interpolation of the first image, and the premise for executing step S410 is that the first valid gesture is detected in the first image, the third image also includes the first valid gesture.

[0187] S411, Detect the third image to obtain the third detection result.

[0188] For example, after the electronic device interpolates the first image to obtain a third image, it performs detection on the third image to obtain a third detection result, which is used to indicate the category of the first valid gesture.

[0189] It is understandable that, due to the relatively low resolution of the first image, it is difficult to accurately identify the specific category of the first valid gesture in the first image. Therefore, in this embodiment of the application, by interpolating the first image to obtain a high-resolution image (i.e., the third image), and then detecting the high-resolution image, the specific category of the first valid gesture can be determined more accurately, thereby improving the accuracy of gesture perception.

[0190] It should be noted that this application does not limit the specific implementation method of detecting the third image to obtain the third detection result. In one possible example, the electronic device inputs the third image into the second model and outputs the third detection result; that is, the electronic device can use the same model to detect both the second and third images. The second model is a multi-classification model. Compared with a binary classification model, a multi-classification model can output the specific category of the detected valid gesture, thereby enabling the electronic device to determine which initial gesture the user performed based on the detection result, so as to execute subsequent gesture perception processes.

[0191] S412, determine whether the categories of the first valid gesture and the second valid gesture are the same.

[0192] For example, after the electronic device completes the detection of the second and third images respectively, it determines whether to execute the gesture perception process based on the detection results (i.e., the second detection result and the third detection result). This gesture perception process is used to detect the target dynamic gesture and execute a preset operation corresponding to the target dynamic gesture. The target dynamic gesture is one of a plurality of preset dynamic gestures. For a description of the gesture perception process, please refer to the description in step S413; it will not be repeated here.

[0193] When the first valid gesture and the second valid gesture are of the same type, that is, when the first valid gesture and the second valid gesture are the same type of gesture, and the gesture is the starting gesture of the target dynamic gesture in the preset dynamic gesture, the electronic device will trigger the subsequent gesture perception process.

[0194] Therefore, in the solution provided in this application embodiment, the starting gesture of the target dynamic gesture is determined by a dual-frame confirmation mechanism. That is, the starting gesture of the target dynamic gesture is detected in both high-resolution images (the second image and the third image) before the subsequent process is continued.

[0195] In this scheme, only two image frames need to be acquired to trigger the subsequent gesture perception process. This is because method 400 interpolates the first image (i.e., the low-resolution image) to obtain the third image (the high-resolution image). Thus, only one more high-resolution image frame (i.e., the second image) needs to be acquired to obtain two high-resolution images. This scheme not only maintains the gesture perception function of the electronic device at a low power consumption cost, but also improves the efficiency of gesture perception. Even if the user's gesture operation is relatively fast, it can ensure a high detection success rate, thus achieving a balance between power consumption and performance.

[0196] Optionally, if the category of the second valid gesture indicated by the first detection result is the same as the category of the first valid gesture indicated by the second detection result, the electronic device may also display an indicator icon, wherein the indicator icon is used to prompt the user to continue inputting the target dynamic gesture (i.e., inputting a gesture other than the starting gesture in the target dynamic gesture), or the indicator icon is used to indicate to the user that the electronic device has detected the starting gesture of the target dynamic gesture.

[0197] This indicator icon prompts the user to perform other gestures after the electronic device has confirmed the initial gesture of the target dynamic gesture, thus improving the user experience. For example, after observing the indicator icon, the user knows that the electronic device has detected the initial gesture of the target dynamic gesture and is ready to detect other gestures within the target dynamic gesture. At this point, the user can continue to perform other gestures to trigger the preset operation corresponding to the target dynamic gesture.

[0198] This application does not limit the specific style of the indicator image; it can be represented by text or icons, and is not limited thereto. As an example, the style of the indicator icon corresponds to the category of the target dynamic gesture. Figures 10 to 14 The application scenarios of the gesture recognition function shown are illustrated with several indicator icons. Figure 10 For example: Figure 10 As shown in (a), after the electronic device confirms the detection of the initial gesture, it displays an indicator icon S1001 to prompt the user to continue performing other gestures. The user continues to perform other gestures according to the indicator icon S1001. Figure 10 The fist gesture shown in (b) is illustrated in the figure. As can be seen from the figure, the gesture style in the indicator icon S1001 is palm upward, which is consistent with the type of gesture operated by the user. This is convenient for observing whether the electronic device has accurately recognized the user's initial gesture, so that if the electronic device fails to recognize the gesture, the user can repeat the air gesture operation.

[0199] In one implementation, for gestures of the same category, the starting icon style is consistent, such as... Figure 11 In (a) the indicator icon S1101 and Figure 14 The indicator icons S1401 in (a) are shown in the image, and their styles are similar to... Figure 10 The indicator icon S1001 in (a) is consistent with this. Figure 12 The style of the indicator icon S1201 shown in (a) is consistent with the user's two-finger open gesture. Figure 13 The style of the indicator icon S1301 shown in (a) is consistent with the style of the user's palm facing down, and will not be described in detail here.

[0200] S413, execute the gesture perception process.

[0201] For example, if the second detection result indicates that the second image includes a second valid gesture, and the category of the second valid gesture is the same as the category of the first valid gesture indicated by the third detection result, a gesture perception process is executed. In this case, the first and second valid gestures are of the same category and are the starting gesture of the target dynamic gesture.

[0202] The gesture perception function of the electronic device described in this application embodiment refers to the electronic device detecting the starting gesture of the target dynamic gesture in two frames of images, and then detecting other gestures in the target dynamic gesture other than the starting gesture (if there are other gestures). If other gestures are detected, the preset operation corresponding to the target dynamic gesture is executed.

[0203] In one example, the target dynamic gesture includes a first gesture, which is both the starting and ending gesture of the target dynamic gesture. That is, in this example, the target dynamic gesture consists of a single gesture, such as... Figure 7 Either of the two preset dynamic gestures. In this example, if the electronic device determines that the first valid gesture and the second valid gesture are of the same category, it determines the starting gesture (i.e., the first gesture) of the detected target dynamic gesture. In this case, the electronic device directly executes the preset operation corresponding to the target dynamic gesture.

[0204] In another example, the target dynamic gesture includes a first gesture and a second gesture, where the first gesture is the starting gesture of the target dynamic gesture, and the second gesture is the ending gesture of the target dynamic gesture. That is, in this example, the target dynamic gesture consists of two gestures, for example, it could be... Figure 8The electronic device selects one of four preset dynamic gestures. In this example, if the electronic device determines that the first and second valid gestures are of the same category, it acquires a fourth image frame captured by the camera in the second mode and performs detection on the fourth image to obtain a fifth detection result. The fifth detection result indicates whether a fourth valid gesture exists in the fourth image, and if so, the category of the fourth valid gesture. The fourth valid gesture is any one of the termination gestures of one or more preset dynamic gestures. If the fifth detection result indicates that the fourth valid gesture is the second gesture, the electronic device controls the execution of a preset operation corresponding to the target dynamic gesture.

[0205] In another example, the target dynamic gesture includes a third gesture, a fourth gesture, and a fifth gesture. The third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the intermediate gesture, and the fifth gesture is the ending gesture. That is, in this example, the target dynamic gesture consists of three gestures, for example, it could be... Figure 9 The electronic device selects one of two preset dynamic gestures. In this example, if the electronic device determines that the first and second effective gestures are of the same category, it acquires a fifth image and a sixth image captured by the camera in the second mode. It then performs detection on the fifth and sixth images respectively to obtain a sixth detection result and a seventh detection result. The sixth detection result indicates whether a fifth effective gesture exists in the fifth image, and if so, its category. The fifth effective gesture is any one of the intermediate gestures of one or more preset dynamic gestures. The seventh detection result indicates whether a sixth effective gesture exists in the sixth image, and if so, its category. The sixth effective gesture is any one of the terminating gestures of one or more preset dynamic gestures. If the sixth detection result indicates that the fifth effective gesture is the fourth gesture, and the seventh detection result indicates that the sixth effective gesture is the fifth gesture, the device executes the preset operation corresponding to the target dynamic gesture. It is understood that the fifth and sixth images are not necessarily adjacent frames, as the camera may capture multiple images including the fifth effective gesture and multiple images including the sixth effective gesture. Therefore, the fifth and sixth images may be separated by x frames, where x can be a preset value.

[0206] It is understood that this application embodiment uses the first detection result to indicate whether the first image includes a first valid gesture as an example for explanation. However, in another possible implementation, the first detection result may only indicate whether the first image includes a gesture. That is, in another implementation, after a gesture is detected from the first image (regardless of whether it is a valid gesture), the camera is triggered to switch to the second mode. After detecting the starting gesture of the target dynamic gesture, the camera is triggered to switch to the third mode. The third mode is used to acquire an image at a third resolution, which is greater than the second resolution. For example, the electronic device controls the camera to acquire a first image in the first mode and detect whether there is a gesture within the camera range. In the second mode, it acquires a second image and detects whether the gesture is the starting gesture of the target dynamic gesture. In the third mode, it acquires a third image and detects the gesture change, that is, whether the gesture change includes the intermediate and ending gestures of the target dynamic gesture. By adjusting the camera's working mode step by step, the resolution and / or frame rate of the acquired images are improved, thereby reducing device power consumption as much as possible while improving the accuracy of gesture recognition.

[0207] S414, determine whether the number of detected frames is less than the threshold value.

[0208] Optionally, if the categories of the first and second valid gestures are different, the electronic device can return to step S407, that is, reacquire the high-resolution (i.e., second resolution) image captured by the camera in the second mode, and detect the captured high-resolution image to determine whether a valid gesture has been detected. If no valid gesture is detected after the number of detections exceeds a preset threshold (e.g., n), the camera is controlled to switch from the second mode back to the first mode to save power. For example, if the categories of the first and second valid gestures are different, the electronic device acquires n consecutive frames of images captured by the camera in the second mode, and sequentially detects the n frames to obtain n fourth detection results. The n fourth detection results correspond one-to-one with the n frames. Each fourth detection result is used to indicate whether a third valid gesture exists in the corresponding image, and if a third valid gesture exists, its category, where n is an integer greater than or equal to 1. If the n fourth detection results indicate that no third valid gesture exists in any of the n frames, the camera is controlled to switch from the second mode back to the first mode.

[0209] In summary, this application provides a gesture perception method. In this method, an electronic device controls a camera to capture images in a first mode to detect whether a valid gesture exists within the camera's range. Since the first mode is a low-resolution mode, the electronic device can continuously capture images with low power consumption. After a valid gesture is detected in the first image (a frame captured in the first mode), the electronic device switches the shooting mode and controls the camera to capture a second image in a second mode to detect whether a valid gesture exists within the camera's range and the type of the valid gesture. On the other hand, the electronic device interpolates the first image to obtain a high-resolution third image and detects the type of valid gesture in the third image. Finally, based on the detection results corresponding to the second and third images, it is determined whether to execute the gesture detection process.

[0210] This solution automatically switches between high-resolution and low-resolution camera modes at different detection stages to meet the image quality requirements of each stage, thus reducing device power consumption. Furthermore, the solution can determine whether a gesture sensing process has been triggered based on only two frames, ensuring a high detection success rate even with rapid user gestures, achieving a balance between power consumption and performance.

[0211] Compared to the above solutions, this application also provides another solution. In this solution, the electronic device controls the camera to continuously acquire images at a second resolution in a second mode. That is, the electronic device continuously acquires high-resolution (i.e., second-resolution) images and detects the acquired images to determine whether to trigger the subsequent gesture perception process. An exemplary description follows:

[0212] The electronic device controls the camera to acquire a seventh image in the second mode, and performs a downsampling operation on the seventh image to obtain an eighth image. The resolution of the eighth image is the first resolution, or the resolution of the eighth image is less than a preset value. That is to say, by performing a downsampling operation on the seventh image to reduce the image resolution (e.g., from the second resolution to the first resolution), the purpose of reducing the resolution is to use the first model to perform effective gesture detection on the image (please refer to the following description for details).

[0213] Then, the electronic device detects the eighth image to obtain an eighth detection result. This eighth detection result indicates whether the eighth image contains an eighth valid gesture, which is any one of the starting gestures of a plurality of preset dynamic gestures. For example, the electronic device inputs the eighth image into a first model and outputs the eighth detection result, where the first model is a binary classification model. The explanation of the first model can be found in the above embodiments and will not be repeated here. Further, if the eighth detection result indicates that the eighth image includes an eighth valid gesture, the electronic device acquires another frame of a ninth image through a camera in a second mode and detects the ninth image to obtain a ninth detection result. This ninth detection result indicates whether the ninth valid gesture exists in the ninth image, and, if so, the category of the ninth valid gesture. For example, the electronic device inputs the ninth image into a second model and outputs the ninth detection result, where the second model is a multi-classification model. The explanation of the second model can be found in the above embodiments and will not be repeated here.

[0214] If the ninth detection result indicates that the ninth image includes the ninth valid gesture, the electronic device performs detection on the eighth image to obtain a tenth detection result, which is used to indicate the category of the eighth valid gesture. If the category of the eighth valid gesture is the same as the category of the ninth valid gesture, the electronic device determines to trigger the gesture perception process.

[0215] Therefore, in the above scheme, high-resolution images are continuously acquired, and during the first-stage detection process (detecting the presence or absence of valid gestures), the high-resolution images can be downsampled so that the first model can be used to detect the downsampled images. Since the first model is a binary classification model, it consumes less power than a multi-class classification model. Because in gesture perception scenarios, most of the time, the first-stage detection is performed, and using a binary classification model can effectively reduce power consumption. After a valid gesture is detected in the image obtained after downsampling from the seventh image, the electronic device can use the second model to detect the gesture category in the seventh image and the next frame image (i.e., the ninth image), and determine whether to trigger the subsequent gesture perception process based on the detection results. In other words, this scheme does not require a mode switching process, and only requires two frames of images to determine whether to trigger the subsequent gesture perception process. At the same time, it can also reduce the power consumption of the first-stage detection process, achieving a balance between detection effect and power consumption.

[0216] Corresponding to the gesture sensing method described in the above embodiments, this application also provides a hardware structure diagram of an electronic device for implementing the gesture sensing method, as shown below. Figure 16 The electronic device 500 is shown in the figure.

[0217] Electronic device 500 includes a processor 510, a camera 520, a display screen 530, and a memory 540. Optionally, it also includes a sensor mode 550, which may include, for example, a distance sensor 551. The illustrated components can be implemented in hardware, software, or a combination of both. Electronic device 500 includes 1 to N cameras and 1 to M displays, where N and M are both positive integers greater than 1.

[0218] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 500. In other embodiments of this application, the electronic device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. For example, processing Figure 16 The electronic device 500, as shown in the module or unit, may also include one or more of the following: mobile communication module, wireless communication module, antenna, audio module, speaker, microphone, headphone jack, buttons, motor, indicator, subscriber identification module (SIM) card interface, external memory interface, charging management module, battery, power management module, etc. The sensor module 550 may also include one or more of the following sensors: pressure sensor, gyroscope sensor, magnetic sensor, accelerometer, distance sensor, proximity light sensor, fingerprint sensor, touch sensor, ambient light sensor, etc.

[0219] The processor 510 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a DSP, a baseband processor, and / or a neural network processing unit (NPU).

[0220] The different processing units can be independent devices or integrated into one or more processors. For example, processor 510 is used to execute the gesture perception method in the embodiments of this application.

[0221] The memory 540 can refer to internal memory, also known as "RAM," and can be used to store executable program code, including instructions. The processor 510 calling the executable program code stored in the memory 540 can cause the electronic device to execute the gesture perception method described in the embodiments of this application. The memory 540 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.).

[0222] As one possible implementation, in the embodiments of this application, the electronic device uses tightly coupled memory (TCM) and double data rate (DDR) memory.

[0223] Among them, DDR memory is a double data rate synchronous dynamic random access memory with a very high data transfer rate. Its full name is double data rate synchronous dynamic random access memory (DDR SDRAM).

[0224] The TCM (Temporary Memory Module) is contained within the memory's address map space and can be accessed as fast memory. TCM provides low-latency memory to the processor without the unpredictability inherent in caches. It can be used to store critical routines, such as interrupt handlers or real-time tasks where avoiding cache uncertainty is crucial. Furthermore, TCM can be used to store temporary register data, data types with local attributes unsuitable for caching, and important data structures such as interrupt stacks.

[0225] It should be noted that, from a storage capacity perspective, TCM memory has a smaller capacity, while DDR memory has a larger capacity. From a data transfer rate perspective, DDR memory has a higher data transfer rate than TCM memory. From a power consumption perspective, DDR memory consumes more power than TCM memory. Electronic devices typically use DDR memory, which therefore consumes more power; embodiments of this application use TCM memory for data caching in some scenarios to reduce power consumption.

[0226] For example, in some scenarios of early-stage pre-sensing services (effective gesture recognition scenarios), both dynamic memory and static memory in electronic devices use TCM memory to store continuous multi-frame images captured by the camera in the first mode. These images are QQVGA image specifications, where QQVGA image specifications are smaller image sizes. It can be understood that QQVGA image specifications have lower image resolution and occupy less memory.

[0227] For example, in some scenarios involving post-processing identification of specific business operations (gesture category recognition scenarios), the electronic device's dynamic and static memory are combined using DDR memory and TCM memory to jointly store multiple consecutive frames of images captured by the camera in the second mode. These images are multiple consecutive frames in QVGA image format, where QVGA image format has a larger image size compared to QQVGA image format. It can be understood that QVGA image format has a relatively higher image resolution than QQVGA image format, resulting in a larger image memory footprint.

[0228] Understandably, QQVGA format images have the lowest resolution and require the least memory.

[0229] In this embodiment, the camera can automatically switch between two modes. For convenience, the process of the camera acquiring images at a first resolution in the first mode and performing effective gesture recognition on these images at the first resolution is referred to as the first stage, and the process of the camera acquiring images at a second resolution in the second mode and performing gesture category recognition on these images at the second resolution is referred to as the second stage.

[0230] In the first stage, on the one hand, the camera captures small-sized images in QQVGA format and stores them in TCM memory, thus occupying very little memory space; on the other hand, the processor analyzes multiple consecutive frames of images to determine whether the user has triggered a certain service. For example, if the image analysis detects a valid gesture in the image, the processor can determine that the user has triggered the air gesture service, and then trigger the subsequent gesture perception service.

[0231] In the second stage, on the one hand, the camera captures a large image in QVGA format and stores it in DDR memory and TCM memory; on the other hand, the processor analyzes the captured large image to identify the specific content of the triggered service. If the image analysis detects that the palm changes from an open state to a gripping state, the processor can identify the gripping gesture and use it to trigger a screenshot.

[0232] Therefore, it can be seen that in scenarios where the camera is always on, although the camera will continuously capture a large number of images, the solution of this application can greatly reduce power consumption by acquiring small-sized continuous multi-frame images in the first stage and storing them in low-power TCM memory, and performing image analysis on the small-sized continuous multi-frame images.

[0233] Display screen 530 is used to display images, videos, etc. Display screen 594 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In this embodiment, the user can control the content displayed on display screen 530 through gesture sensing function, such as zooming in and out of the image displayed on the display screen, taking a screenshot of the currently displayed screen, or swiping the content displayed on the display screen upwards, etc.

[0234] The electronic device 500 also includes various sensors that can convert different physical signals into electrical signals. For example, a distance sensor 551 is used to measure distance. The electronic device 500 can measure distance using infrared or laser. For example, in a shooting scenario, the electronic device 500 can use the distance sensor 551 to measure distance for fast focusing. In the solution provided in the embodiments of this application, the distance sensor 551 can be used to measure the distance of the user's gesture. For example, in… Figure 11 In the described scenario, users can control the start and pause of the video by extending their palm and pressing forward. Figure 11 The distance between the starting gesture shown in (a) and the electronic device is greater than [missing information]. Figure 11 The distance between the end gesture and the electronic device shown in (b) is such that, in order to accurately identify the target dynamic gesture, in addition to capturing the gesture image through a camera and recognizing the captured image, it is also necessary to use a distance sensor 551 to measure the change in the distance between the gesture and the electronic device.

[0235] Electronic device 500 implements display functions through a GPU, display screen 530, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 594 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 510 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0236] Electronic device 500 can achieve shooting function through ISP, camera 520, video codec, GPU, display 530 and application processor.

[0237] Camera 520 is used to capture still images or videos. An object is projected onto a photosensitive element by an optical image generated through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 520, where N is a positive integer greater than 1.

[0238] The camera 520 in this embodiment can switch between at least two modes: a first mode and a second mode. The first mode is used to capture images at a first resolution, and the second mode is used to capture images at a second resolution. The processor 510 can control the camera 520 to switch between these two modes according to preset logic, and control the camera to capture images in both modes.

[0239] In this embodiment, the camera 520 can achieve functions such as air gesture control of the screen through an always-on camera (AON) function. The AON function relies on an image signal processing unit (ISP). The ISP can run various algorithms to process image signals in real time. The camera module connects to the ISP via a camera serial interface (CSI), and the ISP connects to an access point (AP). After receiving the image captured by the camera module, the ISP performs image signal processing. The AP can control the ISP's operating mode and obtain its operating status through its built-in inter-integrated circuit (I2C).

[0240] The above is a detailed description of the embodiments of this application using electronic device 500 as an example. It should be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on electronic device 500. Electronic device 500 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations.

[0241] The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0242] In this embodiment of the application, in order to realize the gesture perception function of the electronic device, it is necessary to make hardware and software improvements to the camera and processor of the electronic device. The hardware improvements of the electronic device will be described first below.

[0243] For example, Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 17 As shown, the electronic device may include an improved camera 601 and an improved processor 602. The improved camera 601 refers to the addition of control circuitry and operating circuitry corresponding to new shooting modes to an existing camera module, achieving a low-power configuration. For example, the existing camera module has shooting mode 1, and the camera module includes operating circuitry corresponding to mode 1. If a shooting mode 2 is added, correspondingly, switching between mode 1 and mode 2 is involved. Therefore, the improved camera module includes not only the operating circuitry corresponding to mode 1, but also the newly added operating circuitry corresponding to mode 2, as well as control circuitry corresponding to the switching between the two modes. It should be understood that, depending on actual application requirements, more than two shooting modes can be set, and this embodiment of the application does not impose any limitations on this.

[0244] In one possible implementation, the improved camera 601 includes two operating modes: a first mode and a second mode. The resolution of the image captured by the camera 601 in the first mode is lower than the resolution of the image captured in the second mode. Optionally, the frame rate of the image captured by the camera 601 in the first mode is lower than the frame rate of the image captured in the second mode. The camera 601 can switch between these two modes. In this example, the first frame rate is lower than the second frame rate, and the first resolution is lower than the second resolution.

[0245] For example, in a gesture perception scenario, if the conditions for activating the camera are met (such as the gesture perception function being enabled), the camera 601 operates in the first mode, continuously acquiring images at the first resolution at the first frame rate (i.e., the camera is constantly scanning) to detect whether there is a first valid gesture within the range of the camera 601; if the first valid gesture is detected in a frame of the captured first image, the camera 601 switches from the first mode to the second mode, acquiring a second image at the second resolution at the second frame rate, and detecting whether the second image includes a second valid gesture and the category of the second valid gesture when it exists.

[0246] Based on the above example, camera 601 can dynamically adjust the resolution (and optionally the frame rate) of the captured images to adapt to different detection needs. For example, camera 601 can capture images at a lower resolution (and frame rate) to detect whether there is a first valid gesture within the range of camera 601, and camera 601 can capture images at a higher resolution (and frame rate) to identify whether there is a second valid gesture in the image and the specific category of the second valid gesture. The low resolution (i.e., the first resolution) of camera 601 can be 120×160, and the high resolution (i.e., the second resolution) can be 240×320 or 480×640. For example, the low frame rate (i.e., the first frame rate) of camera 601 can be 1fps, and the high frame rate (i.e., the second frame rate) can be 30fps.

[0247] It should be noted that this application does not specifically limit the resolution range and frame rate range of the images captured by the camera. That is, there is no limit to the minimum and maximum frame rates, or the minimum and maximum resolutions, of the images captured by the camera. In practical applications, the frame rate range and resolution range can be reasonably set according to requirements.

[0248] It should be noted that the camera 601 can be a front-facing camera or a rear-facing camera of an electronic device, and this embodiment does not limit it.

[0249] The improved processor 602 can be a SoC. In this embodiment, when the electronic device triggers the camera 601 to perform a persistent scan and acquire an image of the current scene, the camera 601 can send the image to the SoC for image analysis to detect whether there is a valid gesture in the image and the type of the valid gesture, etc.

[0250] To achieve low power consumption, the SoC can support low-power always-on ISP (AONISP), see attached document. Figure 18 The camera 601 transmits low-resolution images to the AON ISP. The AON ISP performs format conversion on the image but does not apply any image effects. It then stores the converted image in on-chip static random-access memory (on-chip SRAM). The SoC can also support ultra-low-power cores, with computation, algorithm execution, and image storage all operating in low-power mode. Furthermore, the SoC can support a low-power embedded neural network processor (eNPU).

[0251] It should be noted that the information interaction and execution process between the above modules / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method and system embodiments section, and they will not be repeated here.

[0252] The SoC involved in the embodiments of this application will be described in detail below.

[0253] For example, Figure 18 This is a schematic diagram of a SoC structure provided in an embodiment of this application. Figure 18 As shown, the SoC includes a first processing module, a second processing module, and a third processing module. The first processing module includes an I2C bus interface, an AON ISP, on-chip SRAM, a digital signal processor (DSP), and an eNPU. The second processing module includes an image signal processor (ISP), a neural network processor (NPU), and a central processing unit (CPU). The third processing module includes a GPU and an NPU. In the SoC, the power consumption of the first processing module is lower than that of the second processing module. Specifically, the power consumption of the eNPU in the first processing module is lower than that of the NPU in the second processing module, and the power consumption of the AON ISP in the first processing module is lower than that of the ISP in the second processing module.

[0254] As an example, the first processing module can be used to process the first image at the first resolution captured by the camera 901. Exemplarily, in the first mode, the camera 601 captures the first image at the first resolution. The AON ISP obtains the first image at the first resolution through the I2C bus interface. After processing by the AON ISP, the eNPU detects the processed first image at the first resolution, for example, detecting whether a first valid gesture exists in the first image. The on-chip SRAM in the first processing module can be used to store the processed first image at the first resolution. For example, after detecting a first valid gesture from the first image, the first image is stored in the on-chip SRAM. The DSP can be used to notify the eNPU to perform image detection, receive the detection results reported by the eNPU, and report the detection results to the upper-layer application. The first processing module employs a low-power configuration to reduce the power consumption of the electronic device.

[0255] As an example, the second processing module can be used to process the second image at the second resolution captured by the camera 601. Exemplarily, in the second mode, the camera 601 captures the second image at the second resolution, which is then processed by the IPS. The NPU then performs recognition on the processed second image at the second resolution, for example, identifying whether a second valid gesture exists in the second image and the type of the second valid gesture. Before sending the data (such as image data) to the memory, the second processing module can perform security processing (e.g., encryption), storing the securely processed data in a security buffer in the memory.

[0256] As an example, the third processing module can be used to obtain a first image from on-chip SRAM, and interpolate the first image to obtain a third image with a second resolution via GPU. Then, the NPU can recognize the interpolated third image with a second resolution, such as recognizing the category of a first valid gesture in the third image.

[0257] It should be noted that the embodiments of this application do not limit the format of the image data transmitted between the first processing module or the second processing module and the camera, or the image data transmitted between the first processing module and the third processing module. For example, the image data can be camera serial interface (CSI) or mobile industry processor interface (MIPI) data.

[0258] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the Android system as an example of a layered architecture software system to illustrate the software structure of an electronic device. Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The layered architecture divides the software system of the electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces.

[0259] Reference Figure 19 The electronic device in this application embodiment includes an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, a sensor hub, and a hardware layer.

[0260] The application layer can include a series of applications, which run by calling the application programming interface (API) provided by the application framework layer.

[0261] In this embodiment, the application layer may include a gesture sensing application and a sensing module. The gesture sensing application is connected to the sensing module and is registered in the sensing module. The sensing module manages the state and transmits data. For example, when the sensing module learns from the first processing module in the sensor hub that there is a valid gesture within the camera range, the sensing module notifies the second processing module in the HAL so that the second processing module can acquire an image of the second resolution and identify the category of the second valid gesture. Finally, the sensing module reports the recognition result to the gesture sensing application.

[0262] In some embodiments, the application layer also includes other applications ( Figure 19 (Not shown), for example, gaze-on-screen applications and gaze-on-AOD applications. In one possible scenario, multiple applications may correspond to the same algorithm. For example, gaze-on-screen applications and gaze-on-AOD applications may correspond to gaze detection algorithms, and the perception module can be used for unified scheduling and management of gaze detection algorithms. In another possible scenario, different applications may correspond to different algorithms. For example, gesture perception applications may correspond to gesture recognition algorithms (including algorithms for identifying the presence or absence of valid gestures and for identifying the category of valid gestures), while gaze-on-screen applications may correspond to gaze detection algorithms. Both algorithms involve acquiring image data from the underlying camera, and the perception module can be used for scheduling and management of the priorities among multiple algorithms. It is also possible that the gesture recognition algorithm and the gaze detection algorithm have the same priority, and the perception module can instruct the underlying camera to simultaneously report image data to both the gesture perception application and the gaze-on-screen application. In this embodiment, the algorithm for identifying the presence or absence of valid gestures is deployed in the first processing module, and the algorithm for identifying the category of valid gestures is deployed in the second and third processing modules. The algorithms shown in this embodiment are merely examples.

[0263] In some embodiments, if the electronic device includes a foldable screen, the application layer further includes a fourth processing module. The fourth processing module is used to obtain the physical state of the foldable screen reported by the first processing module, as well as the state (on or off) of the inner and outer screen cameras. In addition, the fourth processing module is also used to notify the second processing module of the state of the inner and outer screen cameras.

[0264] In some embodiments, the application may also include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS, which may be system applications or third-party applications. This application embodiment does not limit this.

[0265] The application framework layer provides APIs and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 19 As shown, the application framework layer may include a camera service (CameraService), which is used for priority scheduling and management of all applications that need to use the camera.

[0266] In some embodiments, the application framework layer may also include, for example, a window manager, a content provider, a resource manager, a notification manager, a view system, etc., and this application embodiment does not limit this.

[0267] The hardware abstraction layer may include an AO service, a second processing module, and a third processing module. The AO service can be used to control the enabling or disabling of the gesture category recognition algorithm in the second processing module, the enabling or disabling of the gesture presence / absence recognition algorithm in the third processing module, the enabling or disabling of the gesture presence / absence recognition algorithm in the first processing module, and data transmission between upper and lower layers. The second processing module can be used to process higher resolution and / or higher frame rate images, such as the second image described above, to detect the category of valid gestures in the second image. The second processing module is also used for camera mode switching; for example, the second processing module receives an instruction from the sensing module or the first processing module and controls the camera to switch from a first mode to a second mode.

[0268] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to function. In this embodiment, the kernel layer may include a camera driver, which drives a camera in an electronic device to operate in a first or second mode to capture images at different frame rates and / or resolutions.

[0269] In addition, the kernel layer may also include display drivers, audio drivers, sensor drivers, motor drivers, etc., and this application embodiment does not limit this. Among them, the sensor driver can drive, for example, a proximity sensor to emit light signals to detect whether the user is currently holding the electronic device close to their ear for a call, etc. The sensor driver can also drive, for example, a gyroscope sensor to detect the attitude data of the electronic device; the sensor driver can also drive, for example, an ambient light sensor to detect the ambient light brightness to detect whether the electronic device is in a dark environment, such as when the mobile phone is in a pocket.

[0270] Sensor hubs are used to centrally control sensors, reducing the CPU load. A sensor hub is equivalent to a microprogrammed control unit (MCU), which can run programs to drive multiple sensors. In other words, a sensor hub can support the mounting of multiple sensors. It can be placed as a standalone chip between the CPU and various sensors, or it can be integrated into the application processor (AP) within the CPU.

[0271] In this embodiment, the sensor hub may include a first processing module. This first processing module can process images with lower resolution and / or lower frame rates, such as the first image described above, to detect whether a first valid gesture exists in the first image. Compared to the second processing module, the first processing module is a low-power processing module, which either runs continuously or in a low-power mode. As an example, when the gesture sensing function is enabled, the first processing module is also used to acquire data reported by various sensors and determine various states of the electronic device based on this data, such as screen state, unlock state, and usage state. If the device state meets a first condition, the first processing module can send a first shooting command to the camera, instructing the camera to continuously scan in a low-power shooting mode (such as the first mode) to detect whether a first valid gesture exists within the camera's range. As an example, for a foldable device, the first processing module can determine whether to send a first shooting command to the camera (on the inner or outer screen) by detecting whether the physical state of the foldable screen, the screen state, and the device state meet a second condition. As an example, when the first processing module detects a change in the physical state of the foldable phone's screen, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the first processing module can control the foldable phone's camera (such as the camera on the phone's outer screen and / or the camera on the phone's inner screen) to turn on or off.

[0272] The hardware layer can include, for example, cameras, various sensors, and AON ISPs.

[0273] Understandable, Figure 19 The layers in the illustrated hierarchy and the modules or components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations. Figure 19 The modules included in each layer shown are those involved in the embodiments of this application. The modules included in each layer do not constitute a limitation on the structure of the electronic device or the hierarchy of module deployment (illustrative example). In some embodiments, Figure 19 The modules shown can be deployed individually, or several modules can be deployed together. Figure 19 The module division in this example is one illustration. In some embodiments, Figure 19 The names of the modules shown are for illustrative purposes only.

[0274] Based on the structure of the electronic device described above, the gesture perception method provided in this application will be described below with reference to specific embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0275] Figure 20 This is a flowchart illustrating a gesture sensing method provided in an embodiment of this application. Figure 20 As shown, the gesture sensing method provided in this embodiment includes:

[0276] S701, The target application registers the gesture perception function in the perception module.

[0277] In this embodiment, the target application is the gesture-aware application at the application layer. In some embodiments, the target application obtains gesture-aware function information from a server (or the cloud). After obtaining the gesture-aware function information, the target application can register the gesture-aware function with the sensing module so that the sensing module can run residently and perform gesture-aware related tasks. The gesture-aware function information includes code information that implements the gesture-aware function, and may also include preset operations corresponding to different air gesture operations.

[0278] S702, The sensing module determines whether to activate the gesture sensing function.

[0279] If the sensing module determines to activate the gesture sensing function, then execute S703.

[0280] For example, refer to Figure 5 In (a) and (b), if the user selects to enable the gesture sensing function in the settings interface of the system application, the system application can send a notification to the sensing module so that the sensing module knows that the phone has enabled the gesture sensing function.

[0281] S703, The sensing module sends a first instruction to the first processing module.

[0282] In one implementation, when the gesture sensing function is enabled, the sensing module sends a first instruction to the first processing module. The first instruction instructs the first processing module to detect whether there is a identifiable first valid gesture within the camera range of the electronic device. Sending the first instruction from the sensing module can control the activation of the first processing module, including powering on the module, assigning a working scene, and preparing resources.

[0283] In another implementation, when the gesture sensing function is enabled and a first condition is met, the sensing module sends a first instruction to the first processing module. In this embodiment, the first condition includes at least one of the following:

[0284] The screen of the electronic device is on.

[0285] Electronic devices are unlocked;

[0286] The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal;

[0287] The ambient light sensor data of the electronic device is greater than the third threshold;

[0288] The screen of the electronic device is facing a preset direction;

[0289] The electronic device runs the target application.

[0290] In this embodiment, if the electronic device has enabled gesture sensing and meets the first condition described above, the camera of the electronic device is triggered to continuously capture the first image. By adding the first condition, the electronic device is prevented from continuously capturing the first image when it is not necessary, thereby further reducing the device's power consumption.

[0291] S704, the first processing module responds to the first instruction and sends a first shooting command to the camera, the first shooting command being used to instruct the camera to operate in a first mode.

[0292] The first processing module sends a first shooting command to control the camera to start, including powering on the camera, switching modes, setting the output resolution and frame rate, etc. The first mode is the low-resolution shooting mode, in which the camera continuously captures images at a lower resolution (optionally also including a lower frame rate).

[0293] In some embodiments, if the electronic device is a foldable device, the foldable device includes a first screen and a second screen, the first screen being provided with a first camera and the second screen being provided with a second camera.

[0294] In one possible scenario, the detection of the first condition described in S703 can be replaced by: the first processing module detecting that the inner screen of the electronic device is in a lit state and the electronic device is in an unfolded state. Correspondingly, S704 could be: the first processing module sending a first shooting command to the first camera. In some embodiments, before the first processing module sends the first shooting command to the first camera, it further includes: detecting that the state of the electronic device meets a second condition, the second condition including at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor (on the inner screen) of the electronic device and the reflected light signal is greater than a first threshold, and / or, the signal strength of the reflected signal is less than a second threshold, and / or, the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the inner screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0295] In one possible scenario, the detection of the first condition described in S703 can be replaced by: the first processing module detecting that the outer screen of the electronic device is on and the electronic device is in a folded state. Correspondingly, S704 could be: the first processing module sending a first shooting command to the second camera. In some embodiments, before the first processing module sends the first shooting command to the second camera, it further includes: detecting that the state of the electronic device meets a second condition, the second condition including at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity sensor (on the outer screen) of the electronic device and the reflected light signal is greater than a first threshold, and / or, the signal strength of the reflected signal is less than a second threshold, and / or, the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the outer screen of the electronic device is facing a preset direction; the electronic device is running a preset application.

[0296] S705, The camera sends the first image to the first processing module.

[0297] In response to a first shooting command, the camera captures a first image at a first resolution in a first mode (at a first frame rate). For example, the first frame rate may be set to 5fps, and the first resolution may be set to 200×180 or 640×480.

[0298] S706, The first processing module identifies whether the first image contains a first valid gesture.

[0299] If the first processing module recognizes a first valid gesture in the first image, then S707 is executed. Otherwise, the first processing module continues to acquire an image of a first resolution from the camera and identifies whether a first valid gesture exists in the acquired image, unless the camera is controlled to be turned off.

[0300] In some embodiments, the first processing module has a preset first model, which is an effective gesture recognition model. The first model is a binary classification model. The first processing module detects the first image through the first model and determines whether there is a first effective gesture in the first image based on the recognition result output by the first model.

[0301] In some embodiments, the first model may be trained using a lightweight neural network model to identify whether a first valid gesture exists in an image. The first model can be deployed on the eNPU of the first processing module and has good real-time performance.

[0302] As an example, the training process for the first model may include:

[0303] Step a: Construct the training set and test set for the first model. Both the training set and the test set include sample images and the corresponding annotation results (i.e., whether the gesture is present or absent in the sample image). The sample images in the training set and the test set are different.

[0304] Step b: Train the first model based on the initial first model and the training set. Specifically, use the sample images from the training set as the input to the initial first model, and use the annotation results corresponding to the sample images from the training set as the output of the initial first model to train the first model.

[0305] Step c: Based on the first model trained in step b and the test set, verify the prediction results of the first model. When the model loss function converges, stop training the first model.

[0306] S707. When a first valid gesture is recognized from the first image, the first processing module sends a first message to the sensing module. The first message is used to notify the sensing module that a first valid gesture exists within the camera range of the electronic device.

[0307] S708, The sensing module sends a second instruction to the second processing module.

[0308] The second instruction is used to instruct the second processing module to identify whether a second valid gesture exists within the camera's range, and if so, the type of the second valid gesture. The first processing module sending the second instruction can control the activation of the second processing module, including powering on the module, issuing the working scene, and preparing resources.

[0309] S709, the second processing module responds to the second instruction and sends a second shooting command to the camera, the second shooting command being used to instruct the camera to operate in the second mode.

[0310] The second processing module sends a second shooting command to control the camera to switch modes, that is, from the first mode to the second mode. In the second mode, the camera continuously captures images at a higher resolution, that is, the second image.

[0311] S710: The camera sends a second image to the second processing module.

[0312] In response to a second shooting command, the camera captures a second image at a second frame rate and a second resolution in a second mode, and sends the second image to the second processing module. In this embodiment, the second image is a single frame.

[0313] S711, The second processing module identifies whether a second valid gesture exists in the second image and the type of the second valid gesture.

[0314] In some embodiments, the second processing module has a preset second model, and the second processing module uses the second model to identify whether a second valid gesture exists in the second image and the category of the second valid gesture.

[0315] As an example, the second model can be trained using a lightweight neural network model based on deep learning methods, used to identify the category of valid gestures in an image. As an example, the second model can include model #1, model #2, and model #3, where model #1 is used to identify the overall frame and orientation of the hand, model #2 is used to identify three-dimensional hand nodes, and model #3 is used to classify the identified hand nodes to determine the specific category of the gesture.

[0316] Model #1 can be a model capable of recognizing single-frame images, primarily providing accurately cropped hand images for the second model. It can also recognize various hand sizes, identify hand occlusion, and accurately locate the hand by recognizing features such as the arm, torso, or other personal characteristics. Model #2 can be a Landmark model that can recognize 21 hand nodes and the positional occlusion between these nodes. For example, Figure 21 A schematic diagram of the hand nodes is shown. Model #2 can identify the coordinates of 21 hand nodes. Model #3 can infer the actions such as straightening or bending each finger based on the hand node data extracted by Model #2, and match these actions with preset gestures to predict the gesture category.

[0317] As an example, the second model can be deployed on the NPU of the second processing module, which has good real-time performance.

[0318] In some embodiments, the second processing module identifies whether a second valid gesture exists in the second image and the category of the second valid gesture, which may include: identifying a hand region from the second image to remove non-hand regions from the second image; performing gesture analysis on the identified hand region, such as inputting the image of the identified hand region into the second model described above to determine the category of the gesture in the image.

[0319] In some embodiments, the second processing module may use background subtraction to separate the foreground (i.e., the hand region) and the background (i.e., the non-hand region) in the second image.

[0320] For example, if the second image does not contain a second valid gesture, the second processing module sends a fourth message (not shown in the figure) to the perception module. The fourth message is used to notify the perception module that the second image does not contain a second valid gesture.

[0321] S712, if a second valid gesture exists in the second image, the second processing module sends a second message to the perception module, which is used to notify the perception module that a second valid gesture exists in the second image and the type of the second valid gesture.

[0322] S713, the perception module sends a third instruction to the third processing module, which instructs the third processing module to perform interpolation processing on the first image and to identify the category of the first valid gesture based on the interpolated high-resolution (second resolution) image.

[0323] It is understood that S713 can be triggered by step S712 or by step S707, and this application does not limit it.

[0324] S714, The third processing module performs interpolation processing on the first image to obtain the third image.

[0325] For example, in response to a third instruction from the sensing module, the third processing module retrieves a first image from memory (such as on-chip SRAM in the first processing module) and interpolates the first image to obtain a third image with a second resolution.

[0326] S715, the third processing module detects the third image to identify the category of the first valid gesture.

[0327] In some embodiments, the third processing module has a preset second model, and the third processing module uses the second model to identify the category of the first valid gesture in the third image.

[0328] S716. After detecting the category of the first valid gesture, the third processing module sends a third message to the perception module, which is used to notify the perception module of the category of the first valid gesture.

[0329] S717, The perception module determines whether to trigger the gesture perception process.

[0330] For example, the perception module determines whether the categories of the first valid gesture and the second valid gesture are the same. If the categories of the first valid gesture and the second valid gesture are the same and both are the starting gestures of the target dynamic gesture, then the gesture perception process is triggered. If they are different, the gesture perception process is not triggered.

[0331] An example of the gesture perception process is as follows: Upon determining that the gesture perception process has been triggered, the perception module can send an instruction to the second processing module. The second processing module, based on the instruction from the perception module, acquires one or more frames of images from the camera, performs gesture category recognition on the acquired images, and notifies the perception module of the recognition result. The perception module receives the recognition result sent by the second processing module. If it detects a valid gesture other than the initial gesture, it sends an instruction to the target application to instruct the electronic device to recognize the target dynamic gesture. Optionally, this instruction may include an identifier of the target dynamic gesture. The target application stores the correspondence between the target dynamic gesture and preset operations (see the example in the above embodiment for details). In response to the instruction from the perception module, the target application executes the preset operation corresponding to the target dynamic gesture.

[0332] In the above embodiments, if the electronic device has registered a gesture sensing function and has enabled the gesture sensing function, the sensing module can send a first shooting command to the camera through the first processing module (low-power processing module) so that the camera can capture a first image at a lower frame rate and / or resolution. If the first processing module recognizes a first valid gesture in the first image, it can notify the second processing module so that the second processing module can send a second shooting command to the camera so that the camera can capture a second image at a higher frame rate and / or resolution. The second processing module identifies the category of the second valid gesture in the second image. On the other hand, the third processing module interpolates the first image into a high-resolution third image and identifies the category of the first valid gesture based on the third image. If the category of the first valid gesture is the same as the category of the second valid gesture, the gesture sensing process is triggered.

[0333] This application also provides a gesture sensing method, which is applied to electronic devices with flexible screens. The following description uses a foldable phone as an example. The foldable phone has a first screen and a second screen. The first screen is a foldable screen that can be folded inwards. The first screen and the second screen are respectively equipped with a first physical camera and a second physical camera for acquiring image data. The gesture sensing method of this embodiment involves the processing logic of the device's underlying modules when the foldable screen state changes. The following description, in conjunction with the appendix... Figure 22 Please provide an explanation.

[0334] For example, Figure 22 This is a flowchart illustrating a gesture sensing method provided in an embodiment of this application. Figure 22 As shown, the gesture perception method of this embodiment may include the following steps:

[0335] S801, The first processing module acquires sensor data to determine the physical state changes of the foldable screen.

[0336] In this embodiment, sensor data includes, for example, magnetic sensors and Hall effect sensors. Acquiring sensor data determines whether the physical state of the foldable screen has changed. Changes in the physical state of the foldable screen include transitioning from a folded state to an unfolded state, or vice versa.

[0337] S802a, the first processing module controls the opening or closing of the first physical camera based on the physical state changes of the foldable screen.

[0338] S802b: The first processing module controls the opening or closing of the second physical camera based on the physical state changes of the foldable screen.

[0339] In one possible implementation, if the physical state of the foldable screen changes from a folded state to an unfolded state, the first processing module can control the activation of the first physical camera.

[0340] In one possible implementation, the second physical camera is already turned on. If the physical state of the foldable screen changes from a folded state to an unfolded state, the first processing module can control the second physical camera to turn off while simultaneously controlling the first physical camera to turn on.

[0341] In one possible implementation, the first physical camera is already turned on. If the physical state of the foldable screen changes from an unfolded state to a folded state, the first processing module can control the first physical camera to be turned off and the second physical camera to be turned on at the same time.

[0342] S802c, the first processing module reports the physical state of the foldable screen and the status of the inner and outer screen cameras to the third processing module.

[0343] It should be noted that this embodiment does not impose any restrictions on the execution order of S802a to S802c.

[0344] S803, the fourth processing module sends a notification to the second processing module, informing it of the status of the inner and outer screen cameras.

[0345] This embodiment illustrates the interaction process between various modules inside a foldable phone when the physical state of the screen changes. Through this interaction, precise control of the camera's low-power continuous scanning function is achieved, enabling the phone to recognize the user's air gestures when using the foldable phone. This allows for control of the phone in different usage scenarios, such as audio and video playback control and image display control, thereby improving the user experience.

[0346] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0347] This application provides a computer program product that, when run on a device, enables the device to perform the steps described in the various method embodiments above.

[0348] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0349] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0350] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0351] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0352] Furthermore, it should be noted that the various numerical designations used in this application (such as the terms "first," "second," "third," "fourth," and other terminology used in the specification, claims, and accompanying drawings, if any) are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the sequence of execution; the execution order of each process should be determined by its function and internal logic.

[0353] The terms “comprising” and “having”, and any variations thereof, mean “including, but not limited to”, unless otherwise specifically emphasized, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0354] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0355] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The specific operational methods in the method embodiments of this application can also be applied to the device embodiments or system embodiments.

Claims

1. A gesture sensing method applied to an electronic device, the electronic device including a camera, characterized in that, The method includes: The camera is controlled to operate in a first mode, which is used to acquire images at a first resolution. A first image frame captured by the camera in the first mode is obtained, and the first image is detected to obtain a first detection result. The first detection result is used to indicate whether there is a first valid gesture in the first image. The first valid gesture is any one of the starting gestures of one or more preset dynamic gestures. If the first valid gesture is detected from the first image, the camera is controlled to switch from the first mode to the second mode, the second mode being used to capture an image at a second resolution, the first resolution being lower than the second resolution; A second image frame captured by the camera in the second mode is acquired, and the second image is detected to obtain a second detection result. The second detection result is used to indicate whether there is a second valid gesture in the second image, and the category of the second valid gesture when there is a second valid gesture. The second valid gesture is any one of the starting gestures of the plurality of preset dynamic gestures. The first image is interpolated to obtain a third image, and the third image is detected to obtain a third detection result. The third detection result is used to indicate the category of the first valid gesture. The resolution of the third image is the same as the resolution of the second image or the resolution of the third image is greater than a preset value. Based on the second detection result and the third detection result, it is determined whether to execute the gesture perception process. The gesture perception process is used to detect the target dynamic gesture and execute the preset operation corresponding to the target dynamic gesture. The target dynamic gesture is one of the multiple preset dynamic gestures.

2. The method according to claim 1, characterized in that, The step of interpolating the first image to obtain the third image includes: After determining that the second valid gesture exists in the second image based on the second detection result, the first image is interpolated to obtain the third image; or, After determining that the first image includes the first valid gesture based on the first detection result, the first image is directly interpolated to obtain the third image.

3. The method according to claim 1 or 2, characterized in that, The step of detecting the first image to obtain a first detection result includes: The first image is input into the first model and the first detection result is output. The first model is a binary classification model. The step of detecting the second image to obtain a second detection result includes: The second image is input into the second model and the second detection result is output. The second model is a multi-classification model. The step of detecting the third image to obtain a third detection result includes: The third image is input into the second model and the third detection result is output.

4. The method according to any one of claims 1 to 3, characterized in that, If the category of the second valid gesture is different from the category of the first valid gesture, the method further includes: The camera acquires n consecutive frames of images captured in the second mode, and sequentially detects the n frames to obtain n fourth detection results. The n fourth detection results correspond one-to-one with the n frames. Each fourth detection result is used to indicate whether the corresponding image has a third valid gesture, and the category of the third valid gesture when it exists. The n is an integer greater than or equal to 1. If the n fourth detection results indicate that the third valid gesture does not exist in any of the n frames of images, the camera is controlled to switch from the second mode back to the first mode.

5. The method according to any one of claims 1 to 4, characterized in that, The first image and the second image are two adjacent frames, or there is a one-frame interval between the first image and the second image.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining whether to execute the gesture perception process based on the second detection result and the third detection result includes: If the second detection result indicates that the second image includes the second effective gesture, and the category of the second effective gesture is the same as the category of the first effective gesture indicated by the third detection result, the gesture perception process is executed, and the first effective gesture and the second effective gesture are the starting gestures of the target dynamic gesture.

7. The method according to claim 6, characterized in that, If the category of the first valid gesture indicated by the first detection result is the same as the category of the second valid gesture indicated by the second detection result, the method further includes: Display an indicator icon to prompt the user to continue inputting the target dynamic gesture.

8. The method according to claim 6 or 7, characterized in that, The target dynamic gesture includes a first gesture and a second gesture, wherein the first gesture is the starting gesture of the target dynamic gesture and the second gesture is the ending gesture of the target dynamic gesture; The execution of the gesture perception process includes: A fourth image frame captured by the camera in the second mode is acquired, and the fourth image is detected to obtain a fifth detection result. The fifth detection result is used to indicate whether there is a fourth valid gesture in the fourth image, and the category of the fourth valid gesture when the fourth valid gesture exists. The fourth valid gesture is any one of the termination gestures of the one or more preset dynamic gestures. If the fifth detection result indicates that the fourth valid gesture is the second gesture, the system controls the execution of a preset operation corresponding to the target dynamic gesture.

9. The method according to claim 6 or 7, characterized in that, The target dynamic gesture includes a third gesture, a fourth gesture, and a fifth gesture. The third gesture is the starting gesture of the target dynamic gesture, the fourth gesture is the intermediate gesture of the target dynamic gesture, and the fifth gesture is the ending gesture of the target dynamic gesture. The execution of the gesture perception process includes: A fifth image and a sixth image captured by the camera in the second mode are acquired, and the fifth image and the sixth image are detected respectively to obtain a sixth detection result and a seventh detection result. The sixth detection result is used to indicate whether there is a fifth valid gesture in the fifth image, and the category of the fifth valid gesture when it exists. The fifth valid gesture is any one of the intermediate gestures of the one or more preset dynamic gestures. The seventh detection result is used to indicate whether there is a sixth valid gesture in the sixth image, and the category of the sixth valid gesture when it exists. The sixth valid gesture is any one of the termination gestures of the one or more preset dynamic gestures. If the sixth detection result indicates that the fifth valid gesture is the fourth gesture, and the seventh detection result indicates that the sixth valid gesture is the fifth gesture, then the preset operation corresponding to the target dynamic gesture is executed.

10. The method according to any one of claims 1 to 9, characterized in that, The first mode is also used to acquire images at a first frame rate, and the second mode is also used to acquire images at a second frame rate, wherein the first frame rate is lower than the second frame rate.

11. The method according to any one of claims 1 to 10, characterized in that, Controlling the camera to operate in a first mode includes: In response to the operation of activating the gesture sensing function, the camera is controlled to operate in the first mode.

12. The method according to any one of claims 1 to 11, characterized in that, Before controlling the camera to operate in the first mode, the method further includes: The electronic device is detected to meet a first condition, which includes at least one of the following: The screen of the electronic device is in the on state; The electronic device has been unlocked; The time difference between the light signal emitted by the proximity sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the proximity sensor does not receive the reflected signal; wherein, the proximity sensor is used to determine whether the detected object is close; The ambient light sensor of the electronic device detects data that is greater than the third threshold. The screen of the electronic device faces a preset direction; The electronic device runs a preset application.

13. The method according to any one of claims 1 to 12, characterized in that, The electronic device is a foldable device, which includes a first screen and a second screen located on the front and back of the electronic device respectively in the unfolded state. The first screen is a foldable screen. The camera includes a first physical camera and a second physical camera, which are logical front-facing cameras corresponding to the first screen and the second screen, respectively. Controlling the camera to operate in a first mode includes: If the first screen is detected to be on and the electronic device is in an unfolded or hovered state, control the first physical camera to operate in the first mode; or, When the second screen is detected to be on and the electronic device is in a folded state, the second physical camera is controlled to operate in the first mode.

14. The method according to claim 13, characterized in that, The method further includes: When the first physical camera is running in the first mode, if the electronic device is detected to switch from an unfolded state or a hovering state to a folded state, the second physical camera is controlled to run in the first mode, and the first physical camera is turned off. When the second physical camera is running in the first mode, if the electronic device is detected to switch from a folded state to an unfolded state or a hovering state, the first physical camera is controlled to run in the first mode, and the second physical camera is turned off.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 8.

16. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 8.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 8.

Images