Application control method and device, electronic equipment and storage medium

Abstract

The invention relates to an application control method and device, electronic equipment and a storage medium. The method comprises the steps that an image containing a gesture of a user is acquired; identifying a gesture in the image, and determining a control event corresponding to the gesture; wherein the control event is an event associated with a small window mode; and according to the control event, controlling an application currently displayed by the electronic equipment to execute an operation associated with the small window mode. The method is good in intelligence, and can improve the user experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of application operation, and in particular to an application control method and device, an electronic device, and a storage medium. Background Art

[0002] In recent years, with the development of electronic technology, more and more applications (APPs) are installed in electronic devices, and the control of applications is becoming more and more diversified, with increasing entertainment and experience. For example, most mobile phones support small window mode to meet the needs of users to operate multiple applications at the same time without frequently switching applications. However, how to further improve the convenience of application control is still worth exploring. Summary of the invention

[0003] The present disclosure provides an application control method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a method for controlling an application, including:

[0005] Get an image containing the user's gesture;

[0006] Recognize the gesture in the image, and determine the control event corresponding to the gesture; wherein the control event is an event associated with the small window mode;

[0007] According to the control event, the application currently displayed by the electronic device is controlled to perform an operation associated with the small window mode.

[0008] In some embodiments, the image includes multiple images, and the identifying the gesture in the image and determining the control event corresponding to the gesture includes:

[0009] Identify the shape of the fingers in each image;

[0010] Determining a morphological change trend of fingers in the plurality of images;

[0011] A control event is determined according to a morphological change trend of the fingers in the multiple images.

[0012] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0013] When the distance between the fingers in the plurality of images gradually decreases, determining that the control event is an event for opening a small window mode;

[0014] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0015] According to the event of opening the small window mode, the application currently displayed by the electronic device is controlled to switch to the small window mode display.

[0016] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0017] When the distance between the fingers in the plurality of images gradually increases, determining that the control event is an event for closing the small window mode;

[0018] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0019] According to the event of closing the small window mode, the application displayed in the small window mode by the electronic device is controlled to exit the small window mode.

[0020] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0021] If there is a first image in the plurality of images in which the shape of the finger remains unchanged but the position of the finger in the image changes, determining that the control event is a window moving event according to the change in the position of the finger in the image;

[0022] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0023] According to the window moving event, the application currently displayed in the small window mode of the electronic device is controlled to change the window position.

[0024] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0025] There is a second image in the plurality of images in which the finger shape is changed relative to the finger shape in the first image, and the control event is determined to be a movement stop event; wherein the acquisition time of the second image is later than the acquisition time of the first image;

[0026] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0027] According to the movement stop event, the application currently displayed and moving in the small window mode of the electronic device is controlled to stop moving.

[0028] In some embodiments, if the application currently displayed by the electronic device includes multiple applications displayed in a small window mode, controlling the application currently displayed by the electronic device to perform an operation associated with the small window mode according to the control event includes:

[0029] According to the control event, the focus application among the multiple applications currently displayed in the small window mode of the electronic device is controlled to perform operations associated with the small window mode.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided a control device for an application, including:

[0031] An acquisition module configured to acquire an image containing a user's gesture;

[0032] a determination module configured to recognize a gesture in the image and determine a control event corresponding to the gesture; wherein the control event is an event associated with a small window mode;

[0033] The control module is configured to control the application currently displayed on the electronic device to perform operations associated with the small window mode according to the control event.

[0034] In some embodiments, the image includes multiple images, and the determination module is further configured to identify the shape of the finger in each image; determine the shape change trend of the finger in the multiple images; and determine the control event based on the shape change trend of the finger in the multiple images.

[0035] In some embodiments, the determination module is further configured to determine that the control event is an event for opening the small window mode when the distance between the fingers in the multiple images gradually decreases; the control module is further configured to control the application currently displayed on the electronic device to switch to the small window mode display according to the event for opening the small window mode.

[0036] In some embodiments, the determination module is further configured to determine that the control event is an event for closing the small window mode when the distance between the fingers in the multiple images gradually increases; the control module is further configured to control the application displayed in the small window mode of the electronic device to exit the small window mode based on the event for closing the small window mode.

[0037] In some embodiments, the determination module is further configured to, when there is a first image among the multiple images in which the finger shape remains unchanged but the position of the finger in the image changes, determine that the control event is a window movement event based on the change in the position of the finger in the image; the control module is further configured to control the application currently displayed in the small window mode of the electronic device to change the window position based on the window movement event.

[0038] In some embodiments, the determination module is further configured to determine that the control event is a movement stop event when there is a second image in the multiple images in which the finger shape is changed relative to the finger shape in the first image; wherein the acquisition time of the second image is later than the acquisition time of the first image; and the control module is further configured to control the application currently displayed and moving in the small window mode of the electronic device to stop moving according to the movement stop event.

[0039] In some embodiments, if the application currently displayed by the electronic device includes multiple applications displayed in small window mode, the control module is also configured to control the focus application among the multiple applications currently displayed in small window mode by the electronic device to perform operations associated with the small window mode according to the control event.

[0040] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0041] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method as described in the first aspect above.

[0042] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0043] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in the first aspect above.

[0044] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0045] In the embodiment of the present disclosure, by acquiring an image containing a user's gesture and determining a corresponding control event based on the gesture in the image, the application currently displayed on the electronic device can perform operations associated with the small window mode according to the control event without the user having to touch the electronic device to perform controls related to the small window mode. This has good intelligence and can enhance the user experience.

[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] Figure 1 This is an example of a control method flow of an application provided by an embodiment of the present disclosure. Figure 1 .

[0049] Figure 2It is an architecture diagram of an Android system provided by an embodiment of the present disclosure.

[0050] Figure 3 This is an example of a control method flow of an application provided by an embodiment of the present disclosure. Figure 2 .

[0051] Figure 4 is a diagram of a control device of an application shown in an embodiment of the present disclosure.

[0052] Figure 5 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] Figure 1 This is an example of a control method flow of an application shown in an embodiment of the present disclosure. Figure 1 ,like Figure 1 As shown, the following steps are included:

[0055] S11, acquiring an image including a user's gesture;

[0056] S12, identifying a gesture in the image, and determining a control event corresponding to the gesture; wherein the control event is an event associated with a small window mode;

[0057] S13. According to the control event, control the application currently displayed by the electronic device to perform an operation associated with the small window mode.

[0058] In some embodiments of the present disclosure, the control method of the application can be applied to electronic devices that display applications, such as user equipment (UE), mobile devices, user terminals, mobile phones, tablet computers, personal digital assistants (PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. The electronic device includes a camera, and after the electronic device captures an image through the camera, it recognizes the gesture in the image, and then controls the currently displayed application to perform the operation of the associated small window mode according to the gesture. Among them, the camera in the electronic device can be, for example, a TOF (Time of Flight) camera, or an image sensor based on structural optics.

[0059] In other embodiments of the present disclosure, the control method of the application can be applied to other devices associated with the electronic device that displays the application, such as a server. After the electronic device captures the image through the camera, it sends the image to the server, which recognizes the gesture included in the image and determines the control event corresponding to the gesture and sends it to the electronic device, so that the application currently displayed by the electronic device can perform the operation of the associated small window mode according to the control event. In the embodiment of the present disclosure, the application control method is applied to the electronic device that displays the application as an example for explanation.

[0060] In step S11, the electronic device acquires an image containing a user's gesture, which is an image that can be captured within the acquisition range of the camera of the electronic device, and the gesture is an air gesture. In the image containing the user's gesture, the gesture refers to the shape of the user's fingers, which may include dynamic gestures or static gestures. For example, the gesture may include a fist gesture, an extended palm gesture, a "1" gesture, a heart gesture, and the like.

[0061] In the embodiment of the present disclosure, the image acquired by the electronic device may be a single image or an image sequence including multiple images, wherein the image sequence includes images acquired in chronological order, and adjacent images may be spaced apart by a preset time length.

[0062] In step S12, the electronic device recognizes the gesture in the image. For example, in some embodiments, the electronic device may first detect the hand, and further determine the posture (i.e., gesture) of the hand when the hand is detected. When detecting the hand, for example, the characteristics of the skin color of the hand may be used to detect the hand based on the fact that the color range of the hand is relatively uniform and clustered. An edge-based gesture detection method may also be used to calculate the contour of the image in the image using an edge detection operator to determine whether it conforms to the contour of the hand based on the contour. When determining the posture of the hand, for example, a template matching method may be used to compare the detected hand with different gestures in the template library, and the gesture with the highest matching degree may be determined as the gesture included in the image.

[0063] In other embodiments, an image containing a user's gesture may be input into a gesture recognition model to determine the gesture included in the image, wherein the gesture recognition model may be trained based on a deep learning network. For example, based on training sample data including different gesture images and label values, networks such as convolutional neural networks (CNN) and deep learning networks (DNN) are trained and parameterized to obtain a gesture recognition model, wherein the label value is a preset gesture recognition result.

[0064] It should be noted that if the electronic device acquires a single image, the gesture recognized based on the single image can be understood as a static gesture, and the electronic device can determine the corresponding control event based only on the static gesture. If the electronic device acquires multiple images, the electronic device can determine whether it is a static gesture or a dynamic gesture by performing a difference comparison on the gestures recognized in each image, and determine the corresponding control event based on the static gesture or the dynamic gesture. Among them, a dynamic gesture refers to a gesture in which the shape of the finger changes over time.

[0065] In the disclosed embodiment, the control event corresponding to the gesture is an event associated with the small window mode, for example: the control event corresponding to gesture "1" is an event of enlarging the application window of the small window mode; the control event corresponding to gesture "2" is an event of shrinking the application window of the small window mode; the control event corresponding to the palm-to-palm gesture is an event of opening the small window mode; the control event corresponding to the palm-to-palm gesture is an event of closing the small window mode; the control event corresponding to the static gesture movement is an event of moving the application window of the small window mode.

[0066] It should be noted that the electronic device stores the mapping relationship between gestures and control events, and based on the mapping relationship, the electronic device can determine the control event corresponding to the current gesture. However, the embodiment of the present disclosure does not limit the control event corresponding to the gesture in the mapping relationship.

[0067] In step S13, after determining the control event, the electronic device can control the application currently displayed by the electronic device to perform the operation associated with the small window mode. The application currently displayed by the electronic device can be one or more applications in the small window mode, or an application displayed in the full screen mode; the application currently displayed by the electronic device can be a video application, a reading application, or other application that supports the small window mode, which is not limited in the embodiments of the present disclosure.

[0068] It can be understood that in the disclosed embodiment, by acquiring an image containing a user's gesture and determining a corresponding control event based on the gesture in the image, the application currently displayed by the electronic device can perform operations associated with the small window mode according to the control event, without the user having to touch the electronic device to perform controls related to the small window mode. This is highly intelligent and can improve the user experience. In addition, if an image containing a gesture is acquired by an electronic device that displays the application, the utilization rate of the hardware configuration related to the camera application in the electronic device can be improved.

[0069] In some embodiments, the image includes multiple images, and the identifying the gesture in the image and determining the control event corresponding to the gesture includes:

[0070] Identify the shape of the fingers in each image;

[0071] Determining a morphological change trend of fingers in the plurality of images;

[0072] A control event is determined according to a morphological change trend of the fingers in the multiple images.

[0073] In the disclosed embodiment, the electronic device identifies the shape of the finger in each image, and can determine the shape change trend of the finger by comparing the shape differences of the finger in different images, so as to find the control event corresponding to the shape change trend in the mapping relationship table between gestures and control events. Among them, the shape change trend of the finger includes the change trend of the finger relative to the electronic device that collects the image, and may also include the change trend of the finger relative to itself. In addition, the shape change trend of the finger may include the shape change trend of a single finger, and may also include the shape change trend between multiple fingers. The shape change trend may include a trend of stillness, which corresponds to a static gesture, and may also include a trend of change, which corresponds to a dynamic gesture.

[0074] It can be understood that in the embodiments of the present disclosure, determining control events based on the morphological change trends of fingers in multiple images can reduce the occurrence of situations where, for example, the control of an electronic device is triggered when a gesture in a single image is an unintentional gesture by the user. In this way, the accuracy of application control can be improved.

[0075] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0076] When the distance between the fingers in the plurality of images gradually decreases, determining that the control event is an event for opening a small window mode;

[0077] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0078] According to the event of opening the small window mode, the application currently displayed by the electronic device is controlled to switch to the small window mode display.

[0079] In an embodiment of the present disclosure, a dynamic gesture in which the distance between fingers in multiple images gradually decreases, that is, the fingers change from an open state to a closed state, the embodiment of the present disclosure determines that the corresponding control event is an event for opening the small window mode based on the dynamic gesture. When the application currently displayed on the electronic device is full-screen, the electronic device can control the application currently displayed on the electronic device to switch to small window mode display based on the event for opening the small window mode.

[0080] It can be understood that the small window function is triggered by a dynamic gesture of changing the finger from an open shape to a closed shape. The user operation is simple and it is easy to identify the trend of gesture shape changes, thereby improving the accuracy of gesture control and enhancing the user experience.

[0081] In some implementations, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0082] When the distance between the fingers in the plurality of images gradually increases, determining that the control event is an event for closing the small window mode;

[0083] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0084] According to the event of closing the small window mode, the application displayed in the small window mode by the electronic device is controlled to exit the small window mode.

[0085] In an embodiment of the present disclosure, a dynamic gesture in which the distance between fingers in multiple images gradually increases, that is, the fingers change from a closed shape to an expanded shape. The embodiment of the present disclosure determines that the corresponding control event is an event for exiting the small window mode based on the dynamic gesture. When the application currently displayed on the electronic device is displayed in the small window mode, the electronic device can control the application currently displayed on the electronic device to exit the small window mode based on the event for exiting the small window mode.

[0086] It should be noted that exiting the small window mode of an application may be by switching to full screen mode, or by closing the display of the application in the foreground but running it in the background, or by closing the application, which is not limited in the present embodiment.

[0087] It can be understood that the exit small window function is triggered by a dynamic gesture of changing the finger from a closed shape to an expanded shape. The user operation is simple and easy to identify the trend of gesture shape changes. It can also improve the accuracy of gesture control and enhance the user experience.

[0088] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0089] If there is a first image in the plurality of images in which the finger shape remains unchanged but the position of the finger in the image changes, determining that the control event is a window moving event according to the change in the position of the finger in the image;

[0090] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0091] According to the window moving event, the application currently displayed in the small window mode of the electronic device is controlled to change the window position.

[0092] As mentioned above, the trend of the change in the shape of the finger may include the trend of the change of the finger relative to the electronic device that collects the image. If the position of the finger relative to the electronic device that collects the image changes, the position of the finger in the image will be different in the corresponding multiple images collected. In the embodiment of the present disclosure, if there is a first image in the multiple images in which the shape of the finger remains unchanged but the position of the finger in the image changes, then the control event is determined to be a window move event based on the change in the position of the finger in the image. When the application currently displayed on the electronic device is displayed in small window mode, the electronic device can control the small window application currently displayed on the electronic device to change the window position based on the window move event.

[0093] It should be noted that in the disclosed embodiment, the fingers whose finger shapes remain unchanged but whose positions in the image change can be a preset number of fingers on the hand, such as 5 fingers, or a preset single target finger. The finger shape can be a close-together shape, or any preset shape of a single target finger.

[0094] In addition, the change in the position of the finger in the image may include a change in distance corresponding to the direction, and the corresponding window movement event may carry movement direction and / or movement distance parameters, and the present disclosure does not limit the movement direction and movement distance. For example, if the finger moves 100 pixels to the right in the image, the corresponding window movement event may be a distance mapped by moving a predetermined distance to the right or moving 100 pixels; or, the window movement event may be a distance mapped by moving 100 pixels in a predetermined direction. In the embodiment of the present disclosure, the electronic device may store a mapping between the direction of finger movement and the direction of window movement, and / or a mapping between the finger movement pixels and the window movement distance.

[0095] It can be understood that the function of triggering the mobile window position by gestures in which the finger's own shape remains unchanged but the electronic device that collects the image changes is highly intelligent and easy to operate, thus improving the user experience.

[0096] In some embodiments, determining a control event according to a morphological change trend of fingers in the plurality of images includes:

[0097] There is a second image in the plurality of images in which the finger shape is changed relative to the finger shape in the first image, and the control event is determined to be a movement stop event; wherein the acquisition time of the second image is later than the acquisition time of the first image;

[0098] The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes:

[0099] According to the movement stop event, the application currently displayed and moving in the small window mode of the electronic device is controlled to stop moving.

[0100] In the embodiment of the present disclosure, a window movement event can be determined based on a gesture in which the shape of the finger remains unchanged but the position of the finger in the image changes. When controlling an application currently displayed in a small window mode on an electronic device to change the window position according to the window movement event, the window position may change if the position of the finger in the image changes, and the window position may no longer change if the finger stops after changing the position in the image. Correspondingly, if there is a second image among multiple images in which the finger shape is changed relative to the finger shape in the first image, the control event is determined to be a movement stop event, wherein the acquisition time of the second image is later than the acquisition time of the first image.

[0101] In the embodiment of the present disclosure, when the application currently displayed by the electronic device is in small window mode and the window is moving, the electronic device can control the small window application currently displayed by the electronic device to stop moving based on the movement stop event. For example, the embodiment of the present disclosure can move and drag the small window to any position on the display screen of the electronic device based on the gesture of closing and not letting go, and fix the small window at the current position after letting go.

[0102] It can be understood that in the embodiment of the present disclosure, after triggering the movement of the window position of the application displayed in the small window mode, the movement stop event is triggered by comparing the shape changes of the finger in the subsequently collected images, which is convenient for the user to control the window movement, has high intelligence, and is simple for the user to operate.

[0103] In some embodiments, if the application currently displayed by the electronic device includes multiple applications displayed in a small window mode, controlling the application currently displayed by the electronic device to perform an operation associated with the small window mode according to the control event includes:

[0104] According to the control event, a focus application among a plurality of applications currently displayed in a small window mode on the electronic device is controlled to perform operations associated with the small window mode.

[0105] In the disclosed embodiment, only the focus application among the applications displayed in multiple small window modes is controlled, rather than all small window applications being controlled uniformly. Since the focus application is the application that the user is currently paying attention to, this method can make the control more in line with user needs and enhance the intelligence of the control.

[0106] Taking the control method of the embodiment of the present disclosure applied to an electronic device of an Android system as an example, Figure 2 is an architecture diagram of an Android system provided by an embodiment of the present disclosure, such as Figure 2 As shown in Figure 1, the Android system architecture can be roughly divided into five layers, including: Linux kernel layer, hardware abstraction layer (HAL layer), system runtime layer (Native layer), application framework layer and application layer. The functions of each layer are as follows:

[0107] Linux kernel layer: provides low-level drivers for various hardware of Android devices, such as display, audio, camera, Bluetooth, power management, etc.;

[0108] Hardware abstraction layer: contains multiple library modules, each of which implements a set of interfaces for a specific type of hardware component, such as the interface of a TOF sensor, the interface of a structured optical image sensor, etc. When the system requests access to device hardware, the system loads the corresponding library module for the hardware.

[0109] System runtime layer: mainly provides dynamic libraries such as Android runtime library and Dalvik virtual machine, allowing developers to use JAVA language to write Android applications. For example, it can provide sensor services, including application services for TOF sensors and application services for structured optical image sensors.

[0110] Application framework layer: This layer mainly provides various application programming interfaces (APIs) that may be used when building applications. Some core applications that come with Android are completed using these APIs. Android's application framework provides a series of class libraries required for developing Android applications, using a reuse mechanism. Developers can quickly develop applications, conveniently and efficiently use the components of the Android platform itself, or inherit various application components of the platform itself to achieve personalized extensions.

[0111] Application layer: includes one or more user-oriented applications, including but not limited to: applications that come with the phone or applications downloaded by the user. Applications can be displayed in different modes at the application layer, for example, both application 1 and application 2 are displayed in small window mode, or one of the applications is displayed in full screen mode.

[0112] At present, the customization level of Android has become higher and higher, from the initial UI customization to the application framework layer. In the embodiment of the present disclosure, the control method of the aforementioned application is executed at the application framework layer. Figure 2 As shown, gesture management, event management and window management are performed at the application framework layer, wherein gesture management is to recognize gestures in images, event management is to determine control events corresponding to gestures, and window management is to control the application currently displayed on the electronic device to execute operations in associated small window mode.

[0113] Figure 3 This is an example of a control method flow of an application provided by an embodiment of the present disclosure. Figure 2 , which can be applied to Figure 2 In the Android architecture shown in Figure 3 As shown, the following steps are included:

[0114] S21: The user performs a palm closing gesture.

[0115] In an embodiment of the present disclosure, the electronic device can obtain multiple images including gestures through a built-in camera, and when it is detected that the distance between fingers in the multiple images gradually decreases, it is determined that the user is performing a palm-closing gesture, and the corresponding control event is an event of opening the small window mode.

[0116] S22: Whether the front-end interface supports small windows; if so, execute step S23; if not, do not execute the operation.

[0117] In the disclosed embodiment, after the electronic device detects the palm closing gesture, it further determines whether the currently displayed application supports the small window mode based on the event of opening the small window mode. If it does, it continues to execute step S23, otherwise no response is made.

[0118] S23, the display status is switched from full screen to small window.

[0119] In an embodiment of the present disclosure, if the electronic device determines that the current control event is an event for opening the small window mode, and the currently displayed application supports the small window mode, the currently displayed application is switched to the small window mode display. It should be noted that the embodiment of the present disclosure can trigger the small window mode of multiple applications by repeating the above operation.

[0120] S24. Move following the user's gesture position.

[0121] In the disclosed embodiment, when the electronic device detects the movement of the pinch gesture, it determines that the corresponding control event is a window movement event. Based on this, the electronic device can make the application displayed in the small window mode move with the movement of the palm according to the window movement event. Among them, the electronic device can determine that it is the movement of the pinch gesture when it detects that there is a first image in which the finger shape remains unchanged but the position of the finger in the image changes among the multiple images.

[0122] S25. The user gesture changes from closing to expanding.

[0123] In the disclosed embodiment, when the electronic device detects that the gesture changes from closing to expanding, the corresponding control event is determined to be a moving stop event. Wherein, when the electronic device detects that there is a second image in which the finger shape changes relative to the finger shape in the first image among the multiple images, it determines that the user gesture changes from closing to expanding, and the corresponding control event is a moving stop event.

[0124] S26. Fix the small window position.

[0125] In the embodiment of the present disclosure, after the electronic device determines that the current control event is a movement stop event, the movement of the application displayed in the small window mode is suspended to fix the position of the small window. It should be noted that the embodiment of the present disclosure can exit the small window mode of multiple applications by repeating the above-mentioned S24-S26 operations.

[0126] It can be understood that in the embodiments of the present disclosure, after the electronic device determines the corresponding control event based on the air gesture, the application currently displayed on the electronic device can perform operations associated with the small window mode according to the control event, such as opening the small window mode, closing the small window mode, moving the window of the application displayed in the small window mode, etc., without the user touching the electronic device to perform related controls. It has good intelligence and can enhance the user experience.

[0127] Figure 4 is a control device diagram of an application shown in an embodiment of the present disclosure, comprising Figure 4 It is known that, including:

[0128] An acquisition module 101 is configured to acquire an image including a user's gesture;

[0129] The determination module 102 is configured to recognize a gesture in the image and determine a control event corresponding to the gesture; wherein the control event is an event associated with a small window mode;

[0130] The control module 103 is configured to control the application currently displayed on the electronic device to perform an operation associated with the small window mode according to the control event.

[0131] In some embodiments, the image includes multiple images, and the determination module 102 is further configured to identify the shape of the finger in each image; determine the shape change trend of the finger in the multiple images; and determine the control event based on the shape change trend of the finger in the multiple images.

[0132] In some embodiments, the determination module 102 is further configured to determine that the control event is an event for opening the small window mode when the distance between the fingers in the multiple images gradually decreases; the control module 103 is further configured to control the application currently displayed on the electronic device to switch to the small window mode display according to the event for opening the small window mode.

[0133] In some embodiments, the determination module 102 is further configured to determine that the control event is an event for closing the small window mode when the distance between the fingers in the multiple images gradually increases; the control module 103 is further configured to control the application displayed in the small window mode of the electronic device to exit the small window mode according to the event for closing the small window mode.

[0134] In some embodiments, the determination module 102 is further configured to, when there is a first image in the multiple images in which the finger shape remains unchanged but the position of the finger in the image changes, determine that the control event is a window movement event based on the change in the position of the finger in the image; the control module 103 is further configured to control the application currently displayed in the small window mode of the electronic device to change the window position based on the window movement event.

[0135] In some embodiments, the determination module 102 is further configured to determine that the control event is a movement stop event when there is a second image in the multiple images in which the finger shape is changed relative to the finger shape in the first image; wherein the acquisition time of the second image is later than the acquisition time of the first image; and the control module 103 is further configured to control the application currently displayed and moving in the small window mode of the electronic device to stop moving according to the movement stop event.

[0136] In some embodiments, if the application currently displayed by the electronic device includes multiple applications displayed in small window mode, the control module 103 is also configured to control the focus application among the multiple applications currently displayed in small window mode by the electronic device to perform operations associated with the small window mode according to the control event.

[0137] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0138] Figure 5 8 is a block diagram of an electronic device shown in an embodiment of the present disclosure. For example, the electronic device 800 may be a smart phone, a personal computer, etc.

[0139] Reference Figure 5 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0140] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0141] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0142] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0143] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0144] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0145] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0146] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0147] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0148] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0150] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the control method of the aforementioned application.

[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A control method for an application, characterized in that: The method comprises: Get an image containing the user's gesture; Recognize the gesture in the image, and determine the control event corresponding to the gesture; wherein the control event is an event associated with the small window mode; According to the control event, the application currently displayed by the electronic device is controlled to perform an operation associated with the small window mode.

2. The method according to claim 1, characterized in that The image includes a plurality of images, and the identifying the gesture in the image and determining the control event corresponding to the gesture includes: Identify the shape of the fingers in each image; Determining a morphological change trend of fingers in the plurality of images; A control event is determined according to a morphological change trend of the fingers in the multiple images.

3. The method according to claim 2, characterized in that The step of determining a control event according to the morphological change trend of the fingers in the plurality of images includes: When the distance between the fingers in the plurality of images gradually decreases, determining that the control event is an event for opening a small window mode; The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes: According to the event of opening the small window mode, the application currently displayed by the electronic device is controlled to switch to the small window mode display.

4. The method according to claim 2, characterized in that: The step of determining a control event according to the morphological change trend of the fingers in the plurality of images includes: When the distance between the fingers in the plurality of images gradually increases, determining that the control event is an event for closing the small window mode; The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes: According to the event of closing the small window mode, the application displayed in the small window mode by the electronic device is controlled to exit the small window mode.

5. The method according to claim 2, characterized in that: The step of determining a control event according to the morphological change trend of the fingers in the plurality of images includes: If there is a first image in the plurality of images in which the shape of the finger remains unchanged but the position of the finger in the image changes, determining that the control event is a window moving event according to the change in the position of the finger in the image; The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes: According to the window moving event, the application currently displayed in the small window mode of the electronic device is controlled to change the window position.

6. The method according to claim 5, characterized in that The step of determining a control event according to the morphological change trend of the fingers in the plurality of images includes: There is a second image in the plurality of images in which the finger shape is changed relative to the finger shape in the first image, and the control event is determined to be a movement stop event; wherein the acquisition time of the second image is later than the acquisition time of the first image; The controlling, according to the control event, an application currently displayed by the electronic device to perform an operation associated with the small window mode includes: According to the movement stop event, the application currently displayed and moving in the small window mode of the electronic device is controlled to stop moving.

7. The method according to claim 1, characterized in that If the application currently displayed by the electronic device includes a plurality of applications displayed in a small window mode, controlling the application currently displayed by the electronic device to perform an operation associated with the small window mode according to the control event includes: According to the control event, a focus application among a plurality of applications currently displayed in a small window mode on the electronic device is controlled to perform operations associated with the small window mode.

8. A control device for an application, characterized in that: The device comprises: An acquisition module configured to acquire an image containing a user's gesture; a determination module configured to recognize a gesture in the image and determine a control event corresponding to the gesture; wherein the control event is an event associated with a small window mode; The control module is configured to control the application currently displayed on the electronic device to perform operations associated with the small window mode according to the control event.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 7.

Images