Air interaction method and electronic equipment

Abstract

The invention discloses an air interaction method and electronic equipment, relates to the field of software, and is used for realizing an air mouse of the electronic equipment so as to completely operate the electronic equipment across the air. The air interaction method comprises the steps that the electronic equipment detects a first gesture of a palm through a camera, responds to the first gesture and displays a first icon of an air mouse so as to prompt a user whether to start the air mouse or not; the electronic device detects that the palm clicks with a second gesture through the camera, the electronic device responds to the second gesture clicks with the palm, and the electronic device displays a mouse cursor to start the air mouse and stops displaying the first icon of the air mouse; the electronic equipment detects that a palm moves in a first gesture or a second gesture through a camera, and in response to the movement of the palm, a mouse cursor is displayed to move to a target position along with the movement of the palm; and the electronic equipment detects that the palm executes the preset action with the second gesture through the camera, responds to the palm to execute the preset action with the second gesture, and executes the preset operation.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on November 3, 2023, with application number 202311461292.4 and invention name “A human-computer interaction method and electronic device”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of software, and in particular to a remote interaction method and electronic device. Background Art

[0003] When using electronic devices such as mobile phones, it is sometimes inconvenient for users to directly operate the electronic devices by touching the screen, for example, when their hands are wet or they are far away from the mobile phone. Although some electronic devices support air gestures, they can only support a few functions. This greatly limits the application scope of air gestures and reduces the frequency of their use. Summary of the invention

[0004] The embodiments of the present application provide an air interaction method and an electronic device, which are used to implement an air mouse of an electronic device to operate the electronic device completely in the air.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for air interaction is provided, which is applied to an electronic device including a front camera, and the method includes: the electronic device detects a first gesture of a palm through the camera, and in response to the first gesture, displays a first icon of an air mouse to prompt a user whether to start the air mouse; the electronic device detects a single click of the palm with a second gesture through the camera, and in response to the single click of the palm with the second gesture, the electronic device displays a mouse cursor to start the air mouse, and stops displaying the first icon of the air mouse; the electronic device detects a movement of the palm with the first gesture or the second gesture through the camera, and in response to the movement of the palm, displays a mouse cursor that follows the movement of the palm to move to a target position; the electronic device detects a preset action performed by the palm with the second gesture through the camera, and in response to the palm performing the preset action with the second gesture, performs a preset operation.

[0007] The air interaction method provided by the embodiment of the present application realizes an air mouse, which is started by a first palm gesture and a second palm gesture, controls the movement of the mouse cursor by the movement of the palm, and simulates pressing the left mouse button by the second gesture, thereby realizing the operation of the simulated mouse. Compared with air gestures, the control of electronic devices is more flexible and convenient.

[0008] In a possible implementation, the air mouse is exited and the mouse cursor is stopped from being displayed if the exit condition is met. Because the user does not always use air interaction and still mainly touches the screen, there is an exit mechanism for the air mouse.

[0009] In a possible implementation, the exit condition is satisfied, including: the electronic device fails to detect the first gesture and the second gesture through the camera for a preset time, or the electronic device detects that the user touches the screen through the screen. The electronic device fails to detect the first gesture and the second gesture through the camera, which may be due to the palm moving out of the shooting range of the camera, or it may be due to the palm making gestures other than the first gesture and the second gesture. If the preset time is reached, it can be considered that the user is no longer using the air mouse, and the air mouse is automatically exited. The priority of the user touching the screen is higher than the priority of the air mouse, so when the electronic device detects that the user touches the screen through the screen, the air mouse is exited.

[0010] In a possible implementation, the target position is the left and right edges of the screen, and the palm performs the preset action with the second gesture, which means that the palm controls the mouse cursor to slide inward from the left and right edges of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm controlling the mouse cursor to slide inward from the left and right edges of the screen with the second gesture, returning to the previous level interface. This is a system-level preset operation, which is equivalent to sliding inward from the left and right edges of the screen with a single finger.

[0011] In a possible implementation, the target position is the bottom edge of the screen, and the palm performs the preset action with the second gesture, which means that the palm controls the mouse cursor to slide upward from the bottom edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen with the second gesture, returning to the desktop. This is a system-level preset operation, which is equivalent to sliding upward from the bottom edge of the screen with a single finger.

[0012] In a possible implementation, the target position is the bottom edge of the screen, and the palm performs the preset action with the second gesture, which means that the palm controls the mouse cursor to slide upward from the bottom edge of the screen and pauses with the second gesture, and in response to the palm performing the preset action with the second gesture, a preset operation is performed, including: in response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen and pause with the second gesture, the recent task is displayed. This is a system-level preset operation, which is equivalent to sliding upward from the bottom edge of the screen with a single finger and pausing.

[0013] In a possible implementation, the target position is the top edge of the screen, and the palm performs the preset action with the second gesture, which means that the palm controls the mouse cursor to slide downward from the top edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm controlling the mouse cursor to slide downward from the top edge of the screen with the second gesture, the notification center or the control center is opened. This is a system-level preset operation, which is equivalent to sliding downward from the top edge of the screen with a single finger.

[0014] In a possible implementation, the target position is the bottom edge of the screen, and the palm performs the preset action with the second gesture, which means that the palm controls the mouse cursor to slide horizontally at the bottom edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm controlling the mouse cursor to slide horizontally at the bottom edge of the screen with the second gesture, switching to the previous task or the next task. This is a system-level preset operation, which is equivalent to sliding a single finger horizontally at the bottom edge of the screen.

[0015] In a possible implementation, the target position is on the object, and the palm performs the preset action with the second gesture, which means the palm clicks with the second gesture. In response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm clicking with the second gesture, the object is selected or opened. This is a non-system-level preset operation, which is equivalent to clicking the object with a single finger on the screen.

[0016] In a possible implementation, the target position is on the object, the palm performs the preset action with the second gesture means that the palm moves with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm moving with the second gesture, the object moves following the mouse cursor. This is a non-system-level preset operation, which is equivalent to pressing and dragging the object with a single finger on the screen.

[0017] In a possible implementation, the target position is on the object, the palm performs the preset action with the second gesture means that the palm maintains the second gesture, and in response to the palm performing the preset action with the second gesture, a preset operation is performed, including: in response to the palm maintaining the second gesture, more is displayed, or a menu is displayed, or an editing state is entered. This is a non-system-level preset operation, which is equivalent to long pressing the object with a single finger on the screen.

[0018] In a possible implementation, the target position is on the object, and the palm performs the preset action with the second gesture, which means the palm double-clicks with the second gesture. In response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm double-clicking with the second gesture, scaling the object. This is a non-system-level preset operation, which is equivalent to double-clicking the object with one finger or pinching the object with two fingers on the screen.

[0019] In a possible implementation, the target position is any position, the palm performs the preset action with the second gesture means that the palm moves with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm moving with the second gesture, scrolling or turning pages. This is a non-system-level preset operation, which is equivalent to sliding a single finger at any position on the screen.

[0020] In a possible implementation, the target position is on the text, and the palm performs the preset action with the second gesture, which means that the palm moves with the second gesture, the palm double-clicks with the second gesture, or the palm triple-clicks with the second gesture. In response to the palm performing the preset action with the second gesture, a preset operation is performed, including: in response to the palm moving with the second gesture, or in response to the palm double-clicking with the second gesture, or in response to the palm triple-clicking with the second gesture, the text is selected. This is a non-system-level preset operation, which is equivalent to pressing and sliding a single finger on the text.

[0021] In a possible implementation, the target position is any position, the palm performs the preset action with the second gesture means the palm slides downward with the second gesture, and in response to the palm performing the preset action with the second gesture, the preset operation is performed, including: in response to the palm sliding downward with the second gesture, refreshing the interface. This is a non-system-level preset operation, which is equivalent to sliding down with one finger at any position on the screen.

[0022] In a possible implementation, the target position is on the object, and the palm performs the preset action with the second gesture, which means that the palm slides over multiple objects with the second gesture, or, in the editing state, the palm clicks the object with the second gesture, and in response to the palm performing the preset action with the second gesture, a preset operation is performed, including: in response to the palm sliding over multiple objects with the second gesture, or, in response to the palm clicking the object with the second gesture in the editing state, batch selecting multiple objects. This is a non-system-level preset operation, which is equivalent to long pressing a single finger on the screen and sliding over multiple objects, or, in the editing state, clicking an object with a single finger.

[0023] In a possible implementation, the electronic device further includes: when the mouse cursor is displayed, the electronic device displays a second icon of the mouse cursor in the status bar to prompt the user that the air mouse is currently turned on. When the mouse cursor is not displayed, the electronic device stops displaying the second icon of the mouse cursor in the status bar to prompt the user that the air mouse is currently turned off.

[0024] In a possible implementation, the method further includes: when the electronic device detects that the distance between the palm and the camera is less than a first threshold, a first prompt message is displayed, and the first prompt message is used to indicate that the distance between the palm and the camera is too close; when the electronic device detects that the distance between the palm and the camera is greater than a second threshold, a second prompt message is displayed, and the second prompt message is used to indicate that the distance between the palm and the camera is too far. If the distance between the palm and the screen is too close or too far, the user will be prompted to change the distance between the palm and the screen.

[0025] In a possible implementation, the electronic device further includes: detecting whether the palm is shaking back and forth through a camera; in response to the palm shaking back and forth, the electronic device changes the style of the mouse cursor; in response to the palm stopping shaking back and forth, the electronic device displays the mouse cursor in a default style, so that the mouse cursor is easier to be found by the user.

[0026] In a possible implementation, the method further includes: in response to the first palm gesture, the electronic device displays a mouse cursor in a default style; and in response to the second palm gesture, the electronic device changes the style of the mouse cursor, so that the user can distinguish whether the electronic device correctly recognizes the first gesture and the second gesture by the difference in the styles of the mouse cursor.

[0027] In a possible implementation, the electronic device further includes: detecting a palm sliding screen gesture through a camera, and performing an air sliding screen operation in response to the palm sliding screen gesture. This implementation combines the air mouse with the air sliding screen gesture, making it more convenient to operate the electronic device.

[0028] In a possible implementation, the electronic device further includes: detecting the air-return gesture of the palm through the camera, and returning to the previous level interface in response to the air-return gesture of the palm. This implementation combines the air mouse with the air-return gesture to more conveniently operate the electronic device.

[0029] In a second aspect, an electronic device is provided, including a screen, a front camera, a processor and a memory, wherein the processor is used to control the camera to capture images and the screen to display images; instructions are stored in the memory, and when the processor executes the instructions, the method described in the first aspect and any one of its embodiments is executed.

[0030] According to a third aspect, a computer-readable storage medium is provided, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the method according to the first aspect and any embodiment thereof.

[0031] In a fourth aspect, a computer program product is provided, comprising instructions, which, when executed on the electronic device, cause the electronic device to execute the method as described in the first aspect and any of its embodiments.

[0032] In a fifth aspect, a chip system is provided, the chip system including a processor for supporting an electronic device to implement the functions involved in the first aspect. In one possible design, the device also includes an interface circuit, which can be used to receive signals from other devices (such as a memory) or send signals to other devices (such as a communication interface). The chip system may include a chip and may also include other discrete devices.

[0033] The technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of an air gesture and an air mouse provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of each gesture in an air gesture provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of various gestures in an air mouse provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of a mouse cursor of an air mouse provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of an air mouse icon provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of a mouse cursor moving along with the movement of a palm provided in an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of an access path of an air mouse configuration interface provided in an embodiment of the present application;

[0043] Fig.10 A schematic diagram of an operation guide process for starting an air mouse provided in an embodiment of the present application;

[0044] Fig.11 A schematic diagram of an operation guidance process of moving a mouse cursor provided in an embodiment of the present application;

[0045] Fig.12A schematic diagram of an operation guidance process for simulating a left-button click function of a mouse provided in an embodiment of the present application;

[0046] Fig.13 A schematic diagram of prompting a user to change the distance between the palm and the screen provided in an embodiment of the present application;

[0047] Fig.14 A schematic diagram of making a mouse cursor more obvious provided in an embodiment of the present application;

[0048] Fig.15 A schematic diagram of recommending an air gesture to a user provided in an embodiment of the present application;

[0049] Fig.16 A schematic diagram of a flow chart of a remote interaction method provided in an embodiment of the present application;

[0050] Fig.17 A schematic diagram of an example of an air mouse provided in an embodiment of the present application;

[0051] Fig.18 A schematic diagram of displaying recent tasks and opening the notification center, which refers to displaying the notification center and switching tasks, provided in an embodiment of the present application;

[0052] Fig.19 A schematic diagram of a flow chart of another remote interaction method provided in an embodiment of the present application;

[0053] Fig. 20 A schematic diagram of an example of combining an air mouse and an air gesture provided in an embodiment of the present application;

[0054] Fig.21 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] First, some concepts involved in this application are described.

[0056] The terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0057] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0058] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0059] The electronic device provided in the embodiment of the present application can be mobile or fixed. The electronic device can be deployed on land (for example, indoors or outdoors, handheld or vehicle-mounted, etc.), on the water (for example, ships, etc.), or in the air (for example, airplanes, balloons, and satellites, etc.). The electronic device can be called user equipment (UE), access terminal, terminal unit, subscriber unit, terminal station, mobile station (MS), mobile station, terminal agent or terminal device, etc. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a smart bracelet, a smart screen, a smart watch, a headset, a smart speaker, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical, a terminal in smart grid, a terminal in transportation safety, a terminal in smart city, a terminal in smart home, etc. The embodiment of the present application does not limit the specific type and structure of the electronic device. A possible structure of the electronic device is described below.

[0060] like Figure 1 As shown, taking the electronic device as a mobile phone as an example, at the hardware level, the electronic device 101 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a power management module 240, a battery 241, a wireless charging coil 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295. Optionally, in some embodiments, an audio digital signal processor (ADSP) 243 is also included.

[0061] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 210 may include one or more processing units, for example, the processor 210 may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), an application processor (AP), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a baseband processor, and a neural network processor (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. For example, the processor 210 may be an application processor AP. Alternatively, the above-mentioned processor 210 may be integrated in a system on chip (SoC). Alternatively, the above-mentioned processor 210 may be integrated in an integrated circuit (IC) chip. The processor 210 may include an analog front end (AFE) and a micro-controller unit (MCU) in an IC chip.

[0063] ADSP 243 can be coupled with audio module 270 and sensor module 280, and ADSP 243 can be used to process audio signals and sensor data. When the processor is in a dormant state, ADSP 243 can still keep working, thereby reducing the power consumption of the electronic device.

[0064] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0065] The wireless communication function of the electronic device 101 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, the baseband processor, and the like.

[0066] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 101 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0067] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 101. The wireless communication module 260 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc., applied to the electronic device 101. In some embodiments, the antenna 1 of the electronic device 101 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the electronic device 101 can communicate with the network and other devices through wireless communication technology.

[0068] The external memory interface 220 can be used to connect an external memory card, such as a micro SanDisk (Micro SD) card, to expand the storage capacity of the electronic device 101. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0069] The internal memory 221 may be used to store computer executable program codes, which include computer instructions. The processor 210 executes various functional applications and data processing of the electronic device 101 by running the computer instructions stored in the internal memory 221. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0070] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0071] The electronic device 101 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone interface 270D and the application processor.

[0072] The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210. The speaker 270A, also known as the "speaker", is used to convert the audio electrical signal into a sound signal. The receiver 270B, also known as the "earpiece", is used to convert the audio electrical signal into a sound signal. The microphone 270C, also known as the "microphone", is used to convert the sound signal into an electrical signal. The electronic device 101 can be provided with at least one microphone 270C. The headphone interface 270D is used to connect a wired headset. The headphone interface 270D can be a USB interface 230, or it can be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0073] The button 290 includes a power button, a volume button, etc. The button 290 may be a mechanical button. It may also be a touch button. The electronic device 101 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 101. The motor 291 may generate a vibration prompt. The motor 291 may be used for an incoming call vibration prompt or for touch vibration feedback. The indicator 292 may be an indicator light, which may be used to indicate the charging status, the change in power, or may be used to indicate messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect a SIM card. The SIM card may be inserted into or removed from the SIM card interface 295 to achieve contact and separation with the electronic device 101. The electronic device 101 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 may support a Nano SIM card, a Micro SIM card, a SIM card, etc. In some embodiments, the electronic device 101 uses an embedded SIM (eSIM) card. The eSIM card can be embedded in the electronic device 101 and cannot be separated from the electronic device 101.

[0074] The electronic device 101 can realize the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute computer instructions to generate or change display information.

[0075] The sensor module 280 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a structured light sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, an angle sensor, etc. When the display screen 294 is a folding screen, the angle sensor may detect the folding angle of the display screen 294, and the folding angle ranges from 0 to 180 degrees.

[0076] The battery 241 may include one or more batteries to power the load.

[0077] The power management module 240 is used to receive charging input from a charger. The charger may be a wireless charger, such as a wireless charging base, other electronic devices 101 with reverse wireless charging function, etc. The power management module 240 may receive wireless charging input through a wireless charging coil 242 of the electronic device. The charger may also be a wired charger, for example, the power management module 240 may receive charging input from a wired charger through the USB interface 230. The power management module 240 is also called a charging chip.

[0078] The power management module 240 can charge the battery 241 and also power the electronic device. The power management module 240 receives input from the battery 241 and powers the processor 210, the internal memory 221, the external memory interface 220, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 240 can also be used to monitor the capacity, voltage, battery cycle number, battery health status (leakage, impedance) and other parameters of the battery 241. In some other embodiments, the power management module 240 can also be set in the processor 210.

[0079] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. In some embodiments, the electronic device 101 may include one or more display screens 294. In this application, the display screen 294 with a touch detection function is referred to as a screen.

[0080] The camera 293 is used to capture static images or dynamic videos (collectively referred to as images). In some embodiments, the electronic device 101 may include 1 or N cameras 293, where N is a positive integer greater than 1. The camera 293 generally includes an optical lens and a photosensitive element, and the photosensitive element may be any photosensitive device such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). In the process of capturing an image, the reflected light of the photographed object is projected onto the photosensitive element after passing through the optical lens, and the photosensitive element converts the received light signal into an electrical signal, which is sent to the processor 210 after being processed by the ISP, and the processor 210 obtains a frame of image. In this application, the camera 293 refers to a front camera, and the front camera means that the camera and the screen are facing the same direction. In this way, when it is inconvenient for the user to operate the electronic device by touching the screen, and thus needs to interact in the air, the palm side of the palm faces the camera and the screen, which is in line with the user's usual habit of operating the electronic device by touching the screen.

[0081] The processor 210 may also be provided with a memory for storing computer instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store computer instructions or data that the processor 210 has just used or circulated. If the processor 210 needs to use the computer instructions or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0082] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.

[0083] For scenarios where it is inconvenient for users to directly operate electronic devices by touching the screen when using electronic devices such as mobile phones, although electronic devices in the prior art support air gestures, they can only support a few functions, such as screenshots, page up or page down, etc., which greatly limits the application scope of air gestures and reduces the frequency of use of air gestures. In the present application, the processor 210 executes the air interaction method provided in the embodiment of the present application by executing the program and computer instructions stored in the internal memory 221, so that air gestures and air mouse can be realized, and the electronic device can be completely operated in the air.

[0084] like Figure 2 As shown in A, the air gesture means that the user does not touch the screen of the electronic device, and the camera 293 of the electronic device collects an image including the palm, and the processor recognizes the palm gesture based on the image, and controls the electronic device according to the changes in the gesture in multiple frames of images. Figure 2 As shown in B, the air mouse refers to the user not touching the screen of the electronic device, the camera 293 of the electronic device collects an image including the palm, and the processor recognizes the gesture and position of the palm according to the image, and displays the mouse cursor 200. The position of the mouse cursor 200 changes with the position of the palm in the multi-frame image, and simulates the operation of the mouse according to the change of the gesture in the multi-frame image, thereby controlling the electronic device. In addition, the processor 210 can also obtain the distance between the palm and the electronic device through the distance sensor to assist the air gesture and air mouse.

[0085] like Figure 3 As shown, taking the electronic device running the Android operating system as an example, the software architecture run by the processor 210 includes an application layer, a framework layer, a system runtime layer, a hardware abstract layer (HAL) layer, and a kernel layer.

[0086] The application layer can include a series of application packages, such as camera, call, game, short video application, chat application and other applications.

[0087] The framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The framework layer includes some predefined functions. For example, the framework layer includes a window manager, a content provider, a view system, a notification manager, etc. In particular, the framework layer includes an air interaction module for executing the air interaction method of the present application, thereby realizing air gestures and air mouse.

[0088] The system runtime layer includes C / C++ program libraries and runtime libraries. Many core components and services of the Android operating system are built from native code and need to be written in C and C++ as C / C++ program libraries. When an application is installed for the first time, it will be pre-compiled into a runtime library in the form of machine code. This process is called pre-compilation. In this way, when the application is started and executed, it can be accelerated by running the machine code.

[0089] The HAL layer is used to abstract the hardware. The HAL layer hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, which is hardware-independent. For example, the HAL layer includes display modules, camera modules, audio modules, sensor modules, etc.

[0090] The kernel layer is a layer between hardware and software. For example, the kernel layer includes display driver, camera driver, audio driver, sensor driver, etc., and the present application embodiment does not impose any restrictions on this.

[0091] Exemplary air gestures include but are not limited to: air sliding screen, air screenshot, air return, air pressing, etc. For details, please refer to Fig. 9 As shown in C. Figure 4 As shown, sliding the screen in the air includes waving your palm left and right ( Figure 4 A in the middle) to slide the screen left and right, and to wave your palm up and down ( Figure 4 Screen capture refers to making a fist (as shown in B) to achieve the operation of sliding the screen up and down. Figure 4 To return to the screen, turn your palm over ( Figure 4 D in the middle) to return to the previous interface. Press the palm of your hand in the direction of the palm ( Figure 4 (as shown in E) to achieve audio or video playback progress control, etc.

[0092] It should be noted that the “up”, “down”, “left” and “right” described in the embodiments of the present application are relative directions based on the normal use of the electronic device by the user, which can be understood in conjunction with the drawings of the specific embodiments. The embodiments of the present application do not impose any limitations on this.

[0093] The gestures of the air mouse mainly include the first gesture and the second gesture. The first gesture is equivalent to not pressing any mouse button. When the palm moves with the first gesture, it is equivalent to not pressing any mouse button and not selecting any object. The mouse cursor moves with the movement of the palm. For more information about the movement of the mouse cursor with the movement of the palm, please refer to Figure 8The second gesture is equivalent to pressing the left mouse button. When the palm performs a single click, double click, or triple click with the second gesture, it is equivalent to pressing the left mouse button once, twice, or three times, respectively. When the palm maintains the second gesture, it is equivalent to continuously pressing the left mouse button. When the palm moves with the second gesture, it is equivalent to continuously pressing the left mouse button, and the mouse cursor moves with the movement of the palm.

[0094] For example, Figure 5 As shown in AB, the first gesture can be the index finger and thumb not pinching (also called light touch), and the remaining fingers making a fist, referred to as two fingers not pinching. Figure 5 The middle A shows that the two fingers are not pinched and not moved. Figure 5 B in the middle shows two fingers moving without pinching.

[0095] For example, Figure 5 As shown in the middle CG, the second gesture can be to pinch the index finger and thumb together, and make a fist with the remaining fingers, referred to as two-finger pinch. Figure 5 The middle C shows pinching with two fingers once (i.e. single click with the second gesture). Figure 5 D in the middle shows pinching twice with two fingers (i.e. double-clicking with the second gesture). Figure 5 The middle E shows pinching three times with two fingers (i.e., three taps with the second gesture). Figure 5 F in the middle shows continuous pinching with two fingers (i.e. maintaining the second gesture). Figure 5 The middle G shows continuous pinching and movement of two fingers (i.e., moving with the second gesture).

[0096] It should be noted that the present application uses the above-mentioned gestures as examples for illustration, and is not intended to be limited to these gestures. Other gestures may also be used as long as different gestures can be distinguished.

[0097] In order to make the air mouse compatible with the existing mouse cursor, the style of the mouse cursor displayed by the electronic device is as follows: Figure 6 As shown in A, in response to the first palm gesture, the electronic device displays the mouse cursor 200 in a default style, such as a circle, gray, or semi-transparent, which is not limited in this application. Figure 6 As shown in B, in response to the second palm gesture, which is equivalent to pressing the left mouse button, the electronic device changes the style of the mouse cursor 200 (for example, shrinks the mouse cursor), so that the user can distinguish whether the electronic device recognizes the correct first gesture and the second gesture through the different styles of the mouse cursor.

[0098] It should be noted that the changes in the mouse cursor style involved in this application include but are not limited to: changing the size of the mouse cursor (enlarging or reducing), changing the shape of the mouse cursor, changing the pattern of the mouse cursor, changing the transparency of the mouse cursor, changing the color of the mouse cursor, flashing the mouse cursor, etc.

[0099] In order to make the air mouse compatible with existing mouse operations, the style of the icon displayed by the electronic device is as follows: Figure 7 As shown in Figure A, if the mouse cursor 200 is not on the icon, the electronic device displays the default style of the icon. Figure 7 As shown in FIG. 2B , if the mouse cursor 200 is on the icon, the electronic device changes the icon to the first style (for example, enlarging the icon). Figure 7 As shown in Figure C, if the mouse cursor 200 is on the icon and the palm gesture is the second gesture, which is equivalent to pressing the left mouse button, the electronic device changes the icon to the second style (for example, deepening the icon color, reducing the icon, etc.). Figure 7 As shown in D, if the mouse cursor is prohibited from operating the icon, the electronic device can change the icon to a third style (eg, lighten the icon color).

[0100] like Figure 8 As shown in A and B, for the first gesture ( Figure 8 Middle A) and the second gesture ( Figure 8 B), the position of the mouse cursor on the screen is determined by the position of the palm (especially the index finger) in the image captured by the camera. For the first plane where the screen of the electronic device is located, the position of the mouse cursor refers to the position of the mouse cursor in the coordinate system of the first plane. The coordinate system of the first plane can take the upper left corner of the screen as the coordinate origin, one side of the screen passing through the coordinate origin (for example, the side in the horizontal direction) as the X-axis, and another side of the screen passing through the coordinate origin (for example, the side in the vertical direction) as the Y-axis. The image captured by the camera is the second plane, and the coordinate system of the second plane can take the upper left corner of the image captured by the camera as the coordinate origin, one side of the image passing through the coordinate origin (for example, the side in the horizontal direction) as the x-axis, and another side of the image passing through the coordinate origin (for example, the side in the vertical direction) as the y-axis. It can be seen that the coordinate system of the first plane and the coordinate system of the second plane are symmetrical along the perpendicular bisector of the screen. According to the mapping relationship between the two coordinate systems, the position of the mouse cursor on the screen can be obtained according to the position of the palm (especially the index finger) in the image captured by the camera.

[0101] The movement of the mouse cursor following the movement of the palm means that: the movement direction of the mouse cursor on the screen is the same as the movement direction of the palm in the second plane, and the movement distance of the mouse cursor on the screen is proportional to the movement distance of the palm in the second plane, that is, when the distance between the palm and the screen remains unchanged, the greater the movement distance of the palm in the second plane, the greater the movement distance of the mouse cursor on the screen. Among them, the movement of the palm refers to the movement of the palm in the second plane where the image captured by the camera is located, which is essentially the movement of the image of the palm in the image captured by the camera. The movement of the mouse cursor refers to the movement of the mouse cursor on the screen. For example, relative to the user, when the palm moves to the left, the mouse cursor also moves to the left, and when the palm moves to the right, the mouse cursor also moves to the right.

[0102] like Fig. 9 As shown, the air mouse and the original air gesture can be integrated into an air interaction function, for example, Fig. 9 As shown in the AD, the access path to the air mouse configuration interface is "Settings->Smart Assistant->Air Gestures". Fig. 9 As shown in Figure D, in the configuration interface of the air mouse, prompt information (such as prompt 1, prompt 2) can be displayed to guide the user how to use the air mouse. Fig. 9 As shown in FIG. 5 , when the user clicks the air mouse button to turn on the air mouse for the first time, or when the user clicks the “gesture operation guide” button, the electronic device may display a prompt message to prompt the user whether to gradually learn the air mouse through the operation guide. If the user clicks the “start learning” button, the operation guide process of the air mouse begins. Figure 10-12 shown.

[0103] like Fig.10 As shown, the operation guide process for starting the air mouse is as follows: Fig.10 As shown in A, the electronic device displays a prompt message. The prompt message is used to indicate that the distance between the palm and the camera is within a preset range (e.g., 20 cm-40 cm) so that the electronic device can correctly recognize the palm gesture. Fig.10 As shown in Figure B, the electronic device displays an example of the first gesture and prompt information. The prompt information is used to prompt that when the first gesture of the palm is made, the first icon 300 of the air mouse is displayed in a suspended manner. The example of the first gesture is used to prompt the user how to make the first gesture. When the electronic device detects the first gesture of the palm through the camera, the first icon 300 of the air mouse is displayed in a suspended manner. Fig.10 As shown in Figure C, the electronic device displays an example of single-clicking with the second gesture and prompt information, and the prompt information is used to prompt the user to confirm the start of the air mouse when the palm single-clicks with the second gesture. The example of single-clicking with the second gesture is used to prompt the user how to make a single-click with the second gesture. Fig.10As shown in D, when the electronic device detects a single click of the palm with the second gesture through the camera, the mouse cursor is displayed and the icon of the suspended air mouse stops being displayed. The purpose of confirming the activation of the air mouse by at least two gestures is to avoid the user's mistakenly displaying a gesture that happens to match the gesture for activating the air mouse, thereby activating the air mouse by mistake.

[0104] like Fig.11 As shown in FIG. 1 , the operation guidance process of moving the mouse cursor by moving the palm with the first gesture is as follows: Fig.11 As shown in A, the electronic device displays prompt information to prompt the user to move the palm with the first gesture, and the electronic device displays the mouse cursor to move along with the movement of the palm. Fig.11 As shown in B, the electronic device displays a prompt message to prompt the user to move the mouse cursor to the "settings" icon by moving the palm with the first gesture. Fig.11 As shown in C, when the mouse cursor follows the movement of the palm and moves to the "settings" icon, the electronic device displays a prompt message to prompt that the operation of moving the mouse cursor is successful.

[0105] like Fig.12 As shown, the operation guidance process of simulating the left mouse button click function by single-clicking with the second gesture is as follows: Fig.12 As shown in Figures AC, the electronic device displays a prompt message to prompt the user to click and confirm the "settings" icon by clicking with the second gesture. Fig.12 As shown in D, after the palm completes the single click with the second gesture, the electronic device displays a prompt message to prompt the user to complete all operation guidance processes.

[0106] In addition, during the operation guidance process, or when the user is using the air mouse normally, if the distance between the palm and the screen is too close or too far, the user will be prompted to change the distance between the palm and the screen. Fig.13 As shown in Figure A, when the electronic device detects through the camera that the distance between the palm and the screen is less than a first threshold (for example, 20 cm), a prompt message is displayed to indicate that the distance between the palm and the screen is too close. Fig.13 As shown in B, when the electronic device detects through the camera that the distance between the palm and the screen is greater than a second threshold (e.g., 40 cm), a prompt message is displayed to prompt that the distance between the palm and the screen is too far. The second threshold is greater than the first threshold. The electronic device can detect the distance between the palm and the screen through the distance sensor or structured light sensor described above.

[0107] like Fig.14As shown in A and B, sometimes the mouse cursor may be close to the background color, so that the user cannot find the mouse cursor 200. At this time, the user can shake the palm back and forth, and the electronic device detects whether the palm is shaking back and forth through the camera. In response to the shaking of the palm, the electronic device changes the style of the mouse cursor 200 (for example, enlarges the mouse cursor), so that the mouse cursor is easier to be found by the user. Fig.14 As shown in C, in response to the palm stopping shaking back and forth, the electronic device displays the mouse cursor 200 in a default style.

[0108] like Fig.15 As shown, when the user uses the air mouse, the electronic device can also recommend air gestures to the user, for example, Fig.15 As shown in A and B in the figure, the electronic device can prompt the user to turn on the air return function in the message of playing machine skills. When the user clicks the "Go to Turn on" button, it jumps to the configuration interface of the air return. Fig.15 As shown in C and D, the electronic device can prompt the user to turn on the air sliding screen function in the message of mobile phone skills. When the user clicks the "Go to Turn on" button, it jumps to the air sliding screen configuration interface.

[0109] The following uses the actual use of the air mouse feature by a user as an example to illustrate the air interaction method provided by the embodiment of the present application. Fig.16 As shown, the air interaction method provided in the embodiment of the present application includes:

[0110] S101. An electronic device detects a first palm gesture through a camera, and displays a first icon of an air mouse in response to the first gesture.

[0111] The electronic device can keep the camera always on in the screen-off state or the screen-on state to detect the palm gesture. Fig.17 As shown in A, when the electronic device detects the first palm gesture through the camera, the first icon 300 of the air mouse is displayed in a floating manner. The purpose of displaying the first icon 300 is to prompt the user whether to start the air mouse.

[0112] S102: The electronic device detects, through a camera, a single click of a palm with a second gesture. In response to the single click of the palm with the second gesture, the electronic device displays a mouse cursor and stops displaying the first icon of the air mouse.

[0113] For example, Fig.17As shown in B, when the electronic device detects a single click with the second gesture of the palm through the camera, the air mouse is activated, the mouse cursor 200 is displayed, and the first icon 300 of the suspended air mouse is stopped. The single click with the second gesture of the palm is equivalent to the user's confirmation operation on whether to activate the air mouse. By confirming that the user activates the air mouse through at least two gestures, it can be avoided that the gesture displayed by the user by mistake happens to match the gesture for activating the air mouse, thereby activating the air mouse by mistake.

[0114] Optionally, the electronic device can continuously change the style of the mouse cursor to prompt the user the current location of the mouse cursor. Fig.17 As shown in C, when the electronic device displays the mouse cursor, it can display the second icon 400 of the air mouse in the status bar to prompt the user that the air mouse is currently turned on. When the electronic device does not display the mouse cursor, it stops displaying the second icon of the mouse cursor in the status bar to prompt the user that the air mouse is currently turned off.

[0115] S103: The electronic device detects through a camera that the palm moves in the first gesture or the second gesture, and in response to the movement of the palm, displays a mouse cursor to move to a target position following the movement of the palm.

[0116] For more information about how to make the mouse cursor move with your hand, refer to Figure 8 The target position can refer to the examples in the second column of Table 1 and Table 2, which will not be described here.

[0117] S104: The electronic device detects through a camera that the palm performs a preset action with a second gesture, and in response to the palm performing the preset action with the second gesture, performs a preset operation of the air mouse.

[0118] For example, Table 1 lists how users can operate the screen, mouse, touchpad, or perform air interaction in order to achieve system-level preset operations such as returning to the previous interface, returning to the desktop, displaying recent tasks, opening the notification center or control center, and switching tasks. Among them, returning to the previous interface means returning to the parent interface from the child interface of the application. Returning to the desktop means returning to the desktop from the interface of the application. Fig.18 As shown in A, displaying recent tasks means displaying a thumbnail interface or list of tasks (task 1-task 3) executed within a preset time. Fig.18 As shown in B, opening the notification center means displaying the drop-down list of the notification center. Fig.18 As shown in C, opening the control center means displaying the drop-down list of the control center. Fig.18 As shown in D, switching tasks means switching to the task that was executed last time.

[0119] In the examples of air interaction in the last column, except for the ones marked as air gestures, all are air mice. The action of the air mouse involved in the last column is "the palm performs the preset action with the second gesture", and the content in the first column of the same row is "perform the preset operation of the air mouse".

[0120] For example, for the first row of Table 1: In S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the left and right edges of the screen (i.e., the target position). In S104, in response to the palm controlling the mouse cursor to slide inward from the left and right edges of the screen with the second gesture (i.e., the palm performs the preset action with the second gesture), the mouse returns to the previous interface (i.e., performs the preset operation of the air mouse). Fig. 20 The examples are not repeated here.

[0121] Exemplarily, for the second row of Table 1: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the bottom edge of the screen (i.e., the target position); in S104, in response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen with the second gesture (i.e., the palm performs the preset action with the second gesture), it returns to the desktop (i.e., executes the preset operation of the air mouse).

[0122] Exemplarily, for the third row of Table 1: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the bottom edge of the screen (i.e., the target position); in S104, in response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen and pause with the second gesture (i.e., the palm performs the preset action with the second gesture), the most recent task is displayed (i.e., executing the preset operation of the air mouse).

[0123] Exemplarily, for the fourth row of Table 1: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the top edge of the screen (i.e., the target position); in S104, in response to the palm controlling the mouse cursor to slide downward from the top edge of the screen with the second gesture, or, in response to the palm clicking with the second gesture (i.e., the palm performs a preset action with the second gesture), the notification center or the control center is opened (i.e., the preset operation of the air mouse is executed).

[0124] For example, for the fifth row of Table 1: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the bottom edge of the screen (i.e., the target position), in S104, in response to the palm controlling the mouse cursor to slide horizontally at the bottom edge of the screen with the second gesture (i.e., the palm performs a preset action with the second gesture), the task is switched (i.e., the preset operation of the air mouse is performed). For example, in response to the palm controlling the mouse cursor to slide to the left at the bottom edge of the screen with the second gesture, the next task is switched, and in response to the palm controlling the mouse cursor to slide to the right at the bottom edge of the screen with the second gesture, the previous task is switched.

[0125] Table 1

[0126]

[0127]

[0128] Table 2 lists how users can operate the screen, mouse, touchpad or perform air interaction, except for non-system-level preset operations. Among them, the object can be any operable element such as controls (such as buttons, icons, etc.), folders, files, pictures, texts, hyperlinks, etc., without specific limitation. In the examples of air interaction in the last column, all are air mice except for the marked air gestures. The action of the air mouse involved in the last column is "the palm performs the preset action with the second gesture", and the content in the first column of the same row is "execute the preset operation of the air mouse".

[0129] Exemplarily, for the first row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the object (i.e., the target position); in S104, in response to the palm clicking with the second gesture (i.e., the palm performs the preset action with the second gesture), the object is selected or opened (i.e., the preset operation of the air mouse is executed).

[0130] Exemplarily, for the second row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the object (i.e., the target position); in S104, in response to the palm moving with a second gesture (i.e., the palm performs a preset action with the second gesture), the object is moved following the mouse cursor (i.e., the preset operation of the air mouse is executed).

[0131] Exemplarily, for the third row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the object (i.e., the target position); in S104, in response to the palm maintaining the second gesture (i.e., the palm performs the preset action with the second gesture), more is displayed, or a menu is displayed, or the editing state is entered (i.e., the preset operation of the air mouse is executed).

[0132] Exemplarily, for the fourth row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the object (i.e., the target position); in S104, in response to the palm double-clicking with the second gesture (i.e., the palm performs the preset action with the second gesture), the object is scaled (i.e., the preset operation of the air mouse is executed).

[0133] For example, for the fifth row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to any position (i.e., the target position); in S104, in response to the palm moving with the second gesture (i.e., the palm performs the preset action with the second gesture), the mouse cursor scrolls or turns the page (i.e., performs the preset operation of the air mouse). Fig. 20 The examples are not repeated here.

[0134] Exemplarily, for the sixth row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the text (i.e., the target position); in S104, in response to the palm moving with the second gesture, or, in response to the palm double-clicking with the second gesture (word selection), or, in response to the palm triple-clicking with the second gesture (paragraph selection) (i.e., the palm performs the preset action with the second gesture), the text is selected (i.e., the preset operation of the air mouse is executed).

[0135] Exemplarily, for the seventh row of Table 2: in S103, in response to the movement of the palm, the mouse cursor moves to any position (i.e., the target position) following the movement of the palm; in S104, in response to the palm sliding downward with the second gesture (i.e., the palm performs the preset action with the second gesture), the interface is refreshed (i.e., the preset operation of the air mouse is executed).

[0136] Exemplarily, for the eighth row of Table 2: in S103, in response to the movement of the palm, the mouse cursor follows the movement of the palm and moves to the object (i.e., the target position); in S104, in response to the palm sliding over multiple objects with the second gesture, or, in response to the palm single-clicking to click on the object with the second gesture in the editing state (i.e., the palm performs a preset action with the second gesture), multiple objects are selected in batches (i.e., the preset operation of the air mouse is executed).

[0137] Table 2

[0138]

[0139]

[0140] S105: If the exit condition is met, the air mouse is exited and the mouse cursor stops being displayed.

[0141] Because users do not always use air interaction and still mainly touch the screen, there is an exit mechanism for the air mouse. The exit conditions include but are not limited to: the electronic device fails to detect the first gesture and the second gesture through the camera for a preset time, or the electronic device detects that the user touches the screen through the screen.

[0142] The electronic device cannot detect the first gesture and the second gesture through the camera, which may be due to the palm moving out of the camera's shooting range, or it may be due to the palm making gestures other than the first gesture and the second gesture. If the preset time is reached, it can be considered that the user is no longer using the air mouse, and the air mouse is automatically exited. The priority of the user touching the screen is higher than the priority of the air mouse, so when the electronic device detects that the user touches the screen through the screen, the air mouse is exited.

[0143] For example, Fig.17 As shown in D, when the exit condition is met, the air mouse is exited, and the mouse cursor 200 and the air mouse icon 400 in the status bar stop being displayed.

[0144] The following describes the above-mentioned air interaction method in combination with specific application scenarios of air mouse and air gesture. Fig.19 As shown, the air interaction method provided in the embodiment of the present application includes S201-S207, wherein S201-S205 are air mice, S206-S207 are air gestures, and any step of S206-S207 can be executed independently and is not dependent on S201-S205:

[0145] S201: An electronic device displays a first interface, where the first interface includes an object.

[0146] For example, Fig. 20 As shown in A, the object is the "Settings" icon.

[0147] S202: The electronic device detects a first palm gesture through a camera, and displays a first icon of the air mouse in response to the first gesture.

[0148] This step refers to S101 and will not be described in detail here.

[0149] S203: The electronic device detects, through a camera, a single click of a palm with a second gesture. In response to the single click of the palm with the second gesture, the electronic device displays a mouse cursor and stops displaying the first icon of the air mouse.

[0150] This step refers to S102 and will not be described in detail here.

[0151] S204: The electronic device detects through a camera that the palm moves in the first gesture or the second gesture, and in response to the movement of the palm, displays a mouse cursor that follows the movement of the palm and moves to a target position.

[0152] For example, Fig. 20 As shown in B, the mouse cursor 200 follows the movement of the palm and moves to the "settings" icon (ie, object).

[0153] S205: The electronic device detects through a camera that the palm performs a preset action with a second gesture, and in response to the palm performing the preset action with the second gesture, performs a preset operation of the air mouse.

[0154] For example, Fig. 20 As shown in C and D, the electronic device detects through the camera that the palm clicks with the second gesture, and in response to the palm clicks with the second gesture, opens the setting interface (second interface).

[0155] S206. The electronic device detects an air sliding gesture of the palm on the screen through a camera, and performs an air sliding screen operation in response to the air sliding gesture of the palm on the screen.

[0156] For example, Fig. 20 As shown in E and F, in response to the air sliding gesture of the palm of the hand (sliding the palm up and down), the sliding settings interface is displayed.

[0157] S207. The electronic device detects the air-return gesture of the palm through the camera, and returns to the previous level interface in response to the air-return gesture of the palm.

[0158] For example, Fig. 20 As shown in G and A, in response to the air gesture of the palm (flipping the palm), returning to the previous interface is Fig. 20 The first interface of China Institute A.

[0159] For other air gestures, the electronic device detects the screen capture air gesture of the palm through the camera, and in response to the screen capture air gesture of the palm, takes a screen capture of the current interface. The electronic device detects the air pressing air gesture of the palm through the camera, and in response to the air pressing air gesture of the palm, controls the playback progress of the audio or video.

[0160] It should be noted that because the user is likely to switch back and forth between the air mouse and air gestures when performing air interaction, the air mouse does not need to be exited when performing air gestures, thereby avoiding the process of frequently requiring the user to confirm whether to turn on the air mouse and improving the user experience.

[0161] The air interaction method and electronic device provided by the embodiment of the present application realize an air mouse, which is started by a first palm gesture and a second palm gesture, controls the movement of the mouse cursor by the movement of the palm, and simulates pressing the left mouse button by the second gesture, thereby realizing the operation of the simulated mouse. Compared with air gestures, the control of electronic devices is more flexible and convenient.

[0162] like Fig.21 As shown, the embodiment of the present application also provides a chip system. The chip system 500 includes at least one processor 5001 and at least one interface circuit 5002. The at least one processor 5001 and the at least one interface circuit 5002 can be interconnected through a line. The processor 5001 is used to support the electronic device to implement each step in the above method embodiment, such as Fig.16 , Fig.19 In the method shown, at least one interface circuit 5002 may be used to receive signals from other devices (eg, memory) or to send signals to other devices (eg, communication interface). The chip system may include a chip and may also include other discrete devices.

[0163] The present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on the electronic device, the electronic device executes each step in the method embodiment, such as executing Fig.16 , Fig.19 The method shown.

[0164] The present application also provides a computer program product including instructions. When the instructions are executed on the electronic device, the electronic device executes each step in the method embodiment, such as executing Fig.16 , Fig.19 The method shown.

[0165] Regarding the technical effects of the chip system, computer-readable storage medium, and computer program product, refer to the technical effects of the previous method embodiments.

[0166] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0167] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0170] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0171] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0172] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0173] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A remote interaction method, characterized in that: Applied to an electronic device including a front camera, the method comprises: The electronic device detects a first palm gesture through the camera, and in response to the first gesture, displays a first icon of the air mouse to prompt the user whether to start the air mouse; The electronic device detects, through the camera, that the palm clicks with a second gesture, and in response to the palm clicking with the second gesture, the electronic device displays a mouse cursor to start the air mouse, and stops displaying the first icon of the air mouse; The electronic device detects, through the camera, that the palm moves in the first gesture or the second gesture, and in response to the movement of the palm, displays the mouse cursor to move to a target position following the movement of the palm; The electronic device detects through the camera that the palm performs a preset action with a second gesture, and executes a preset operation in response to the palm performing the preset action with the second gesture.

2. The method according to claim 1, characterized in that: Also includes: If the exit condition is met, the air mouse is exited and the mouse cursor stops being displayed.

3. The method according to claim 2, characterized in that The exit conditions are met, including: The electronic device fails to detect the first gesture and the second gesture through the camera for a preset time, or the electronic device detects that the user touches the screen through the screen.

4. The method according to any one of claims 1 to 3, characterized in that: The target position is the left and right edges of the screen, the palm performs the preset action with the second gesture means that the palm controls the mouse cursor to slide inward from the left and right edges of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm controlling the mouse cursor to slide inward from the left and right edges of the screen with a second gesture, the screen returns to the previous level interface.

5. The method according to any one of claims 1 to 3, characterized in that: The target position is the bottom edge of the screen, the palm performing the preset action with the second gesture means that the palm controls the mouse cursor to slide upward from the bottom edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen with a second gesture, the user returns to the desktop.

6. The method according to any one of claims 1 to 3, characterized in that: The target position is the bottom edge of the screen, the palm performing the preset action with the second gesture means that the palm controls the mouse cursor to slide upward from the bottom edge of the screen and pause with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm controlling the mouse cursor to slide upward from the bottom edge of the screen and pause using a second gesture, recent tasks are displayed.

7. The method according to any one of claims 1 to 3, characterized in that: The target position is the top edge of the screen, the palm performing the preset action with the second gesture means that the palm controls the mouse cursor to slide downward from the top edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm controlling the mouse cursor to slide downward from the top edge of the screen with a second gesture, a notification center or a control center is opened.

8. The method according to any one of claims 1 to 3, characterized in that: The target position is the bottom edge of the screen, the palm performing the preset action with the second gesture means that the palm controls the mouse cursor to slide horizontally at the bottom edge of the screen with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm controlling the mouse cursor to slide horizontally along the bottom edge of the screen with a second gesture, a previous task or a next task is switched.

9. The method according to any one of claims 1 to 3, characterized in that: The target position is on the object, the palm performing the preset action with the second gesture refers to the palm clicking with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm clicking with the second gesture, an object is selected or opened.

10. The method according to any one of claims 1 to 3, characterized in that: The target position is on the object, the palm performing the preset action with the second gesture means that the palm moves with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm moving with the second gesture, the object is moved following the mouse cursor.

11. The method according to any one of claims 1 to 3, characterized in that: The target position is on the object, the palm performing the preset action with the second gesture means that the palm maintains the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm maintaining the second gesture, more is displayed, or a menu is displayed, or an editing state is entered.

12. The method according to any one of claims 1 to 3, characterized in that: The target position is on the object, the palm performing the preset action with the second gesture refers to the palm double-clicking with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm double-clicking with a second gesture, the object is scaled.

13. The method according to any one of claims 1 to 3, characterized in that: The target position is any position, the palm performing the preset action with the second gesture means that the palm moves with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm moving in a second gesture, scrolling or turning pages.

14. The method according to any one of claims 1 to 3, characterized in that: The target position is on the text, the palm performing the preset action with the second gesture means that the palm moves with the second gesture, the palm double-clicks with the second gesture, or the palm triple-clicks with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm moving with the second gesture, or in response to the palm double-clicking with the second gesture, or in response to the palm triple-clicking with the second gesture, the text is selected.

15. The method according to any one of claims 1 to 3, characterized in that: The target position is any position, the palm performing the preset action with the second gesture means that the palm slides downward with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm sliding downward with a second gesture, the interface is refreshed.

16. The method according to any one of claims 1 to 3, characterized in that: The target position is on the object, the palm performing the preset action with the second gesture means that the palm slides over the multiple objects with the second gesture, or, in the editing state, the palm clicks the object with the second gesture, and in response to the palm performing the preset action with the second gesture, performing the preset operation includes: In response to the palm sliding over the multiple objects with the second gesture, or in response to the palm clicking on the object with the second gesture in the editing state, the multiple objects are selected in batches.

17. The method according to any one of claims 1 to 16, characterized in that: Also includes: When the electronic device displays the mouse cursor, the electronic device displays a second icon of the mouse cursor in the status bar; When the mouse cursor is not displayed, stop displaying the second icon of the mouse cursor in the status bar.

18. The method according to any one of claims 1 to 17, characterized in that: Also includes: The electronic device detects that the distance between the palm and the camera is less than a first threshold, and displays first prompt information, where the first prompt information is used to indicate that the distance between the palm and the camera is too close; The electronic device detects that the distance between the palm and the camera is greater than a second threshold, and displays second prompt information, where the second prompt information is used to indicate that the distance between the palm and the camera is too far.

19. The method according to any one of claims 1 to 18, characterized in that: Also includes: The electronic device detects whether the palm is shaking back and forth through the camera; In response to the palm shaking back and forth, the electronic device changes the style of the mouse cursor; In response to the palm stopping shaking back and forth, the electronic device displays the mouse cursor in a default style.

20. The method according to any one of claims 1 to 19, characterized in that: Also includes: In response to the first palm gesture, the electronic device displays a mouse cursor in a default style; In response to a second palm gesture, the electronic device changes the style of the mouse cursor.

21. The method according to any one of claims 1 to 20, characterized in that: Also includes: The electronic device detects the air sliding gesture of the palm on the screen through the camera, and performs an air sliding screen operation in response to the air sliding gesture of the palm on the screen.

22. The method according to any one of claims 1 to 21, characterized in that: Also includes: The electronic device detects the air-returning gesture of the palm through the camera, and returns to the previous level interface in response to the air-returning gesture of the palm.

23. An electronic device, characterized in that: It includes a screen, a front camera, a processor and a memory, wherein the processor is used to control the camera to capture images and control the screen to display images; the memory stores instructions, and when the processor executes the instructions, the electronic device executes the method as described in any one of claims 1-22.

24. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 22.

25. A computer program product, characterized in that The method comprises instructions which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 22.

Images