Air gesture interaction method and electronic equipment

Abstract

The invention provides an air gesture interaction method and electronic equipment, relates to the technical field of terminals, and aims to map a palm of a user into a virtual air mouse cursor and control the air mouse cursor to move along a moving track of an air gesture, so that continuous interaction operation can be realized without memorizing more air gestures by the electronic equipment, and the user experience is improved. And the man-machine interaction efficiency is improved. The method comprises the following steps: in response to a first air gesture input by a user, displaying an air mouse cursor on a first display interface by the electronic equipment; in response to a second air gesture input by the user, the electronic equipment moves the air mouse cursor along a moving track of the second air gesture; and along with the movement of the second air gesture, the electronic equipment moves the air mouse cursor to a first position area of the first display interface. And furthermore, in response to a third air gesture input by the user, the electronic equipment triggers the operation of the air mouse cursor in the first position area, and a second display interface is displayed.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and more particularly to an air gesture interaction method and electronic device. Background Art

[0002] With the continuous development of electronic technology, users' use of electronic devices such as mobile phones is no longer limited to operating by touching the touch screen. At present, some mobile phones can recognize the gestures performed by users in three-dimensional space (i.e., air gestures) through devices such as cameras or ultrasonic sensors, thereby realizing air-based operations on some functions of the mobile phone.

[0003] Normally, one air gesture controls the phone to perform an operation. If the user wants to operate the phone continuously through the air, the phone needs to memorize more air gestures, which not only takes up memory but also reduces the efficiency of human-computer interaction. Summary of the invention

[0004] The present application provides an air gesture interaction method and electronic device, which maps the user's palm into a virtual wireless mouse, displays an air mouse cursor on the display screen of the electronic device through air gestures, and controls the air mouse cursor to move along the moving trajectory of the air gestures, thereby achieving the purpose of continuously operating the electronic device when it is inconvenient for the user. Continuous interactive operations are possible without the electronic device having to memorize a large number of air gestures, thereby improving the efficiency of human-computer interaction.

[0005] This application adopts the following technical solutions:

[0006] In the first aspect, a method for interactive air gestures is provided, which can be executed by an electronic device equipped with a camera, such as a mobile phone, a tablet computer, a laptop computer, etc., or by a chip, a chip system or a processor that can implement the interactive air gesture method provided in the present application, or by a logic module or software that can implement all or part of the functions of the electronic device. The following is an introduction with the execution subject being an electronic device. The method includes: the electronic device displays a first display interface, and in response to a first air gesture input by a user, the electronic device displays an air mouse cursor on the first display interface. In response to a second air gesture input by the user, the electronic device moves the air mouse cursor along the moving trajectory of the second air gesture; as the second air gesture moves, the electronic device moves the air mouse cursor to the first position area of ​​the first display interface. Furthermore, in response to a third air gesture input by the user, the electronic device triggers the operation of the air mouse cursor in the first position area and displays the second display interface.

[0007] In this way, by mapping the user's palm into a virtual air mouse cursor and controlling the air mouse cursor to move along the moving trajectory of the air gesture, the purpose of continuously operating the electronic device when it is inconvenient for the user is achieved. Continuous interactive operations can be performed without the electronic device remembering a large number of air gestures, thereby improving the efficiency of human-computer interaction.

[0008] In a possible implementation of the first aspect, in response to a first air gesture input by a user, the electronic device displays an air mouse cursor at a first preset position on the first display interface, including: in response to a first air gesture input by a user, the electronic device displays an air mouse cursor at a first preset position on the first display interface.

[0009] Optionally, in response to a first air gesture input by the user, the electronic device displays an air mouse cursor at a reference position of the first display interface; wherein the reference position is used to indicate the position where a reference point of the user's palm is mapped to the display screen of the electronic device; the reference point may be the palm of the user's palm; or, the reference point may be the endpoint of the user's index finger.

[0010] In this way, by displaying the air mouse cursor at the first preset position of the first display interface, the user can be prompted that the air mouse cursor can be moved to perform interactive operations, thereby improving the efficiency of human-computer interaction.

[0011] In a possible implementation of the first aspect, a rectangular coordinate system is pre-set in the electronic device; wherein, in response to a second air gesture input by the user, the electronic device moves the air mouse cursor along the moving trajectory of the second air gesture, including: in response to the second air gesture input by the user, the electronic device determines the starting position of the second air gesture mapped in the rectangular coordinate system.

[0012] As the second air gesture moves, the electronic device determines at least one moving position of the second air gesture during the moving process; the electronic device moves the air mouse cursor along the moving trajectory of the second air gesture based on the starting position and at least one moving position.

[0013] In a possible implementation of the first aspect, before the electronic device displays an air mouse cursor on the first display interface in response to a first air gesture input by the user, the method further includes: the electronic device detects a preset gesture input by the user, the preset gesture is used to enter an air mouse mode of the air gesture; in response to the preset gesture input by the user, the electronic device displays a preset icon at a second preset position on the first display interface,

[0014] In this way, after the mobile phone enters the air mouse mode of air gesture, the mobile phone can prompt the user that it is currently in air mouse mode by displaying a preset icon; in this way, the user can continue to input the next air gesture through the preset icon, which improves the efficiency of human-computer interaction and enhances the user experience.

[0015] In a possible implementation manner of the first aspect, the method further includes: when the electronic device fails to detect the air gesture for a preset time period, the electronic device exits the air mouse mode.

[0016] Alternatively, in response to a user clicking operation on the touch screen of the electronic device, the electronic device exits the air mouse mode.

[0017] Alternatively, in response to a fourth air gesture input by the user, the electronic device exits the air mouse mode.

[0018] Optionally, the fourth air gesture is a palm flip gesture; or, the fourth air gesture is a fist hovering gesture.

[0019] In a possible implementation manner of the first aspect, the method further includes: the electronic device hiding the empty mouse cursor.

[0020] In this way, by hiding the air mouse cursor in the current display interface, the user is prompted that the air mouse mode of the air gesture has been exited.

[0021] In a possible implementation manner of the first aspect, in the second air gesture, the palm of the user is opposite to the touch screen of the electronic device.

[0022] Optionally, the reason why the palm of the user's hand is opposite to the touch screen of the mobile phone in the second air gesture is that the palm of the user's hand is easier to identify in the multiple frames of images captured by the camera. By determining the position of the palm, the air mouse cursor can be controlled to move with the movement of the second air gesture.

[0023] In a possible implementation manner of the first aspect, the second air gesture includes extending the index finger and bringing four fingers other than the index finger together.

[0024] In a possible implementation manner of the first aspect, the first air gesture includes pinching two fingers together and bringing three fingers together except the two fingers; wherein the two fingers include the thumb and the index finger.

[0025] Optionally, the first air gesture includes pinching two fingers together and then opening them.

[0026] Optionally, the first air gesture includes opening and hovering with two fingers.

[0027] Optionally, the first air gesture includes a five-finger pinch.

[0028] In a possible implementation manner of the first aspect, the first preset gesture includes spreading two fingers and bringing three fingers other than the two fingers together; wherein the two fingers include a thumb and an index finger.

[0029] In a possible implementation of the first aspect, in response to a third air gesture input by the user, the electronic device triggers the operation of the air mouse cursor in the first position area and displays the second display interface, including: in response to the third air gesture input by the user, the electronic device triggers a click operation of the air mouse cursor in the first position area; in response to the click operation, the electronic device displays the second display interface.

[0030] In a possible implementation of the first aspect, the third air gesture includes pinching two fingers together, and bringing three fingers together except the two fingers; wherein the two fingers include the thumb and the index finger.

[0031] Optionally, the third air gesture includes pinching and then expanding two fingers.

[0032] Optionally, the third air gesture includes opening and hovering with two fingers.

[0033] Optionally, the third air gesture includes a five-finger pinch.

[0034] Optionally, the third air gesture includes clicking along the z-axis direction of the spatial rectangular coordinate system; wherein the z-axis direction is perpendicular to the horizontal direction and the vertical direction when the electronic device is displayed.

[0035] In a second aspect, an electronic device is provided, which has the function of implementing any one of the functions in the first aspect, and the function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0036] According to a third aspect, an electronic device is provided, comprising a memory and one or more processors; wherein the memory is used to store computer instructions, and when the electronic device is running, the processor executes the computer instructions of the memory so that the electronic device executes any one of the methods described in the first aspect.

[0037] In a fourth aspect, a chip system is provided, comprising: at least one processor and an interface, the interface being used to receive instructions and transmit them to at least one processor; at least one processor executes the instructions so that the electronic device executes any one of the methods described in the first aspect.

[0038] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute any of the methods described in the first aspect.

[0039] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0040] Among them, the technical effects brought about by any implementation method in the above-mentioned second to sixth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a camera collecting images provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a software framework of an electronic device provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a flow chart of an air gesture interaction method provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of an application scenario of an air gesture interaction method provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a flow chart of another air gesture interaction method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0050] Fig.10 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0051] Fig.11 A schematic diagram of a flow chart of another air gesture interaction method provided in an embodiment of the present application;

[0052] Fig.12 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0053] Fig.13 A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0054] Fig.14A schematic diagram of an application scenario of another air gesture interaction method provided in an embodiment of the present application;

[0055] Fig.15 A flowchart of another air gesture interaction method provided in an embodiment of the present application;

[0056] Fig.16 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The real-time method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0058] The embodiments of the present application provide an air gesture interaction method that can be applied to electronic devices such as mobile phones, tablet computers, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, vehicle-mounted devices, virtual reality devices, etc. The embodiments of the present application do not impose any restrictions on this.

[0059] For example, Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0060] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193 and a display screen 194, etc.

[0061] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units, for example, the processor 110 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] Among them, the controller can be the nerve center and command center of the electronic device 100.

[0064] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface 130, etc.

[0066] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0067] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0068] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 can receive input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0069] The power management module 141 may be used to monitor performance parameters such as battery capacity, battery cycle times, battery charging voltage, battery discharging voltage, and battery health status (eg, leakage, impedance).

[0070] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0071] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include one or more filters, switches, power amplifiers, low noise amplifiers (LAN), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor, etc. for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be placed in the processor 110. In some embodiments, at least some of the functions of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0072] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating one or more communication processing modules. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0073] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology.

[0074] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0075] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0076] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0077] The camera 193 is used to capture static images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras, where N is a positive integer greater than 1. The camera 193 may be a front camera or a rear camera. The ISP is used to process data fed back by the camera 193.

[0078] like Figure 2 As shown, the camera 193 generally includes a lens and a photosensitive element (sensor), and the photosensitive element can be any photosensitive device such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0079] Still Figure 2 As shown, in the process of taking photos or videos, the reflected light of the photographed object can generate a light signal after passing through the lens, and the light signal is projected onto the photosensitive element, which converts the received light signal into an electrical signal. Then, the camera 193 sends the obtained electrical signal to the ISP for processing, and finally obtains each frame of the image. Optionally, the ISP can also perform algorithm optimization on the noise, brightness, color, etc. of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0080] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0081] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0082] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can enable the electronic device 100 to execute the method provided in some embodiments of the present application, with various functional applications and data processing, etc. by running the above instructions in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory; for example, a double data rate synchronous dynamic random access memory (DDR SDRAM), a tightly coupled memory (TCM), etc. It may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, universal flash storage (UFS), etc. In other embodiments, the processor 110 enables the electronic device 100 to execute the methods provided in the embodiments of the present application, as well as various functional applications and data processing by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0083] Among them, TCM is included in the address mapping space of the internal memory 121 and can be accessed as a fast memory. TCM is used to provide low-latency memory to the processor 110. Since TCM does not have the unpredictability unique to cache, TCM can be used to store important routines, such as terminal processing routines or real-time tasks that need to avoid cache uncertainty. In addition, TCM can be used to save temporary register data, data types whose local attributes are not suitable for cache, and important data structures such as interrupt stacks.

[0084] Generally, TCM has a smaller capacity and DDR has a larger capacity; the data transmission rate of DDR is greater than that of TCM; and the power consumption of DDR is greater than that of TCM. In the embodiment of the present application, DDR or TCM is used for data caching, which can be set according to the actual situation without limitation.

[0085] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0086] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0087] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0088] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0089] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with one or more microphones 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0090] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunication industry association of the USA (CTIA) standard interface.

[0091] The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., but the embodiments of the present application do not impose any restrictions on this.

[0092] Of course, the electronic device 100 provided in the embodiment of the present application may also include one or more devices such as a positioning module 181, a button 190, a motor 191, an indicator 192 and a SIM card interface 195, and the embodiment of the present application does not impose any restrictions on this.

[0093] The methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0094] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0095] Figure 3 1 is a software structure diagram of the electronic device 100 provided in an embodiment of the present application. The layered architecture can divide the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the layered architecture is, from top to bottom, an application layer (referred to as application layer, application), an application framework layer (referred to as framework layer, framework), a hardware abstraction layer (hardware abstraction layer, HAL) and a kernel layer (kernel, also called a driver layer).

[0096] Application layer: can include a series of application packages. For example, the application layer can include system UI, always on (AO) applications and common applications. Common applications can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and desktop launcher applications (not shown in the figure).

[0097] Among them, system UI is the core application responsible for managing the status bar and navigation bar of electronic devices. System UI can be used to feedback the system and application status and interact with users. For example, system UI can be responsible for displaying the status bar and updating relevant information according to the system status, such as time, battery level, network status, etc. As another example, system UI can be used to respond to user operations and display and manage notification information in the notification bar, including displaying notifications, clearing notifications, clicking notifications and other functions. As another example, system UI is responsible for controlling the display and hiding of the navigation bar, and responding to user touch events to realize navigation functions. As another example, when the application enters full-screen mode, system UI is responsible for hiding the status bar and navigation bar to provide a better immersive experience.

[0098] In the embodiment of the present application, the system UI is also used to display the air mouse cursor and the preset icon. For example, when the electronic device enters the air mouse mode, the system UI displays the air mouse cursor or the preset icon according to the air gesture reported by the framework layer.

[0099] AO applications are used to implement functions such as air gestures and staring at the screen. For example, after the AO application is turned on, the camera of the electronic device can be controlled to be in a normally open state, and images can be collected in real time, and the user's air gestures can be recognized, so that the electronic device can be controlled to perform corresponding functions without the user touching the electronic device.

[0100] The framework layer provides a programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3 As shown, the framework layer may include input management service (IMS), window management service (WMS) and activity management service (AMS), etc. Optionally, the framework layer may also include content provision service, view system, phone management service, resource management service, notification management service, etc. (not shown in the figure).

[0101] Among them, the input management service is mainly responsible for input event monitoring, input event parsing, input event distribution, etc. For example, in the embodiment of the present application, the input management service is mainly responsible for distributing the air mouse event to the system UI, so that the system UI displays the air mouse cursor or preset icon according to the air mouse event.

[0102] The window management service is used to manage window programs. The window management service can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The activity management service is mainly responsible for managing activities, and is responsible for the startup, switching, scheduling of various components in the system, and the management and scheduling of application programs, etc.

[0103] The hardware abstraction layer may include multiple library modules, and the library modules may be a hardware compositor (Hwcomposer, HWC), a camera library module, etc. The operating system of the electronic device may load the corresponding library modules for the device hardware, so as to achieve the purpose of the application framework layer accessing the device hardware.

[0104] Optionally, the hardware abstraction layer may include an air mouse control module, which is used to send commands to the kernel layer to instruct the kernel layer to initialize the air mouse driver. Among them, initializing the air mouse driver means restoring the air mouse driver to its initial settings or state. For example, setting the various parameters configured by the air mouse driver to predefined initial states.

[0105] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a camera driver, an air mouse driver, and a sensor driver. Among them, the camera driver is used to drive hardware such as a camera to perform the function of image acquisition. For example, the camera driver is used to configure image parameters (such as resolution, size, etc.) for hardware such as a camera, so as to drive the hardware such as a camera to acquire images based on the configured image parameters. Among them, the air mouse driver is used to drive the electronic device to enter / exit the air mouse mode.

[0106] In some embodiments of the present application, the above application layer, framework layer, hardware abstraction layer, and kernel layer may be deployed in an application processor (AP), that is, the AP controls the modules in each of the above software layers to execute corresponding functions. In some other embodiments of the present application, the software framework of the electronic device may further include various modules deployed in an audio digital signal processor (ADSP), and the ADSP controls each module to execute corresponding functions. Exemplarily, the ADSP may include a Sensing Hub.

[0107] Among them, in addition to receiving and processing data from various sensors (such as an acceleration sensor, a gravity sensor, a gyroscope sensor, etc.), the Sensing Hub can also receive and process data from the camera.

[0108] Optionally, Sensing Hub can be used to recognize air gestures input by the user based on images captured by the camera. After Sensing Hub recognizes the air gestures input by the user, it can also transmit the air gestures to related applications in the application layer (such as system UI), so that the applications can respond to the air gestures input by the user to perform corresponding operations (such as displaying an air mouse cursor, preset icons, etc.).

[0109] In the present application embodiment, Figure 3 As shown, the Sensing Hub may include a first air mouse detection module, in which an AI model (or AI engine) is deployed. Exemplarily, the AI ​​model can be used to recognize the air gestures input by the user in real time based on the images collected by the camera.

[0110] Exemplarily, the AI ​​model can obtain each frame of image captured by the camera in real time by calling the camera driver of the kernel layer. Furthermore, the AI ​​model can recognize the air gestures input by the user based on these images. For example, the AI ​​model can continuously obtain N (N>1) frames of images collected in the recent time (such as the last 200ms), and identify the pattern of the user's palm in these N frames of images, and determine the air gesture input by the user based on the recognized pattern of the user's palm.

[0111] Taking the air gesture input by the user as "two fingers open, three fingers together except two fingers" as an example, illustratively, after the AI ​​model determines the air gesture input by the user, it can report a first event to the system UI in the application layer. The first event can be used to indicate the display of a preset icon, and the preset icon is used to prompt the user that the electronic device enters the air mouse mode.

[0112] Taking the air gesture of "two fingers open and hover" input by the user as an example, illustratively, after the AI ​​model determines that the air gesture input by the user is, it can send a second event to the system UI in the application layer, and the second event can be used to indicate the display of the air mouse cursor.

[0113] It should be noted that the specific name of the above-mentioned AI model is not limited in the embodiments of the present application, and a functional module with any name can be set in the Sensing Hub to perform the above-mentioned functions.

[0114] After the air mouse cursor is displayed in the system UI, the AI ​​model can determine whether the user has input the air mouse cursor movement gesture (or the second air gesture) based on the image captured by the camera in real time, that is, whether the user has the intention to move the air mouse cursor. For example, taking the movement gesture of "index finger extended, four fingers except the index finger folded" as an example, the AI ​​model can continuously obtain N (N>1) frames of images collected in the recent time, and identify whether these N frames contain images of the user's palm. If these N frames contain the pattern of the user's palm, and the pattern of the user's palm in these N frames is "index finger extended, four fingers except the index finger folded", the AI ​​model can determine that the user has input the movement gesture at this time.

[0115] After the AI ​​model determines that the user has input a mobile gesture, the second air mouse module included in the Sensing Hub can detect the movement trajectory of the mobile gesture, and transmit the movement trajectory to the system UI through the IMS in the framework layer, so that the system UI moves the air mouse cursor along the movement trajectory of the mobile gesture.

[0116] In this way, by mapping the user's palm into a virtual wireless mouse, and displaying the air mouse cursor on the display screen of the electronic device through air gestures, and controlling the air mouse cursor to move along the moving trajectory of the air gestures, the purpose of continuously operating the electronic device when it is inconvenient for the user is achieved. Continuous interactive operations can be achieved without the electronic device having to remember a large number of air gestures, thereby improving the efficiency of human-computer interaction.

[0117] In addition, when the user no longer wants to use the air mouse mode of air gestures to interact with the electronic device, he can also input the end gesture of the air mouse mode (or the fourth air gesture) to the camera. For example, the end gesture can be a "palm flip gesture." Then, when the AI ​​model determines that the user has input the end gesture based on the image captured by the camera, the AI ​​model can report a third event to the system UI, and the third event can be used to indicate the end of the air mouse mode of the air gesture. Accordingly, after receiving the third event, the systemUI can hide the above-mentioned air mouse cursor in the current display interface, thereby prompting the user that he has exited the air mouse mode of the air gesture.

[0118] It should be noted that in the embodiment of the present application, by detecting air gestures in a low power island (LPI), it is possible to achieve the purpose of continuously operating the electronic device when it is inconvenient for the user while also reducing power consumption.

[0119] Exemplarily, LPI includes ADSP and on-chip memory (such as TCM). Detection of air gestures through LPI means that the electronic device calls instructions stored in on-chip memory through ADSP to detect air gestures. Since ADSP includes Sensing Hub, Sensing Hub can control the camera when the AP is dormant, thereby reducing power consumption.

[0120] The interactive method of air gestures provided in the embodiment of the present application is described in detail below in conjunction with the drawings of the specification. The interactive method of air gestures provided in the embodiment of the present application can be performed by an electronic device equipped with a camera, such as a mobile phone, a tablet computer, a laptop computer, etc., or by a chip, a chip system or a processor that can implement the interactive method of air gestures provided in the embodiment of the present application, or a logic module or software that can implement all or part of the functions of the electronic device. The embodiment of the present application does not impose specific restrictions on this. The following is a detailed description of the scheme provided in the embodiment of the present application with the execution subject being a mobile phone.

[0121] Figure 4 A schematic diagram of a flow chart of an air gesture provided in an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps.

[0122] S201. The mobile phone detects a preset gesture input by a user through a camera, and in response to the preset gesture, the mobile phone displays a preset icon on a first display interface.

[0123] For example, by turning on the AO function, the mobile phone can set the camera of the mobile phone to a normally open state (that is, the gesture camera is always in working state), and the camera can collect images within the shooting range at a certain working frequency. For example, the above camera can also be a 3D camera (also called a depth camera or a depth of field camera), and the mobile phone can obtain the distance between the object in the image or the user and the mobile phone through the image captured by the 3D camera.

[0124] For example, Figure 5 As shown, the preset gesture of the air gesture can be input within the shooting range of the camera of the mobile phone (such as the front camera). For example, the preset gesture can be "two fingers open, three fingers other than the two fingers are folded"; or, the preset gesture can be the palm hovering for less than a preset time (such as 2 seconds), etc., without restriction. Since the camera of the mobile phone is in the normally open state, the camera can capture multiple frames of images of the user's palm, and the mobile phone can recognize whether the user is currently inputting the above preset gesture based on the multiple frames of images captured by the camera. It should be noted that the preset gesture can also be other gestures, without restriction. In addition, the two fingers can include the index finger and the thumb; or, the two fingers can be other two fingers, without restriction.

[0125] The preset gesture is used to instruct the mobile phone to enter the air mouse mode of the air gesture. Figure 5 As shown, optionally, after the mobile phone enters the air mouse mode of the air gesture, the mobile phone can display a preset icon at the preset position 1 (or the second preset position) of the first display interface; or, the mobile phone can display a preset icon at the reference position of the first display interface. The reference position is determined by the mobile phone based on the reference point of the user's palm, and the reference point can be any point in the user's palm. For example, the reference point can be the center point of the user's palm; or the reference point can be the end point of the user's index finger, etc., without limitation.

[0126] Taking the case where the mobile phone displays a preset icon at the reference position of the first display interface as an example, for example, Figure 6 As shown, the corresponding relationship between each position on the display screen and each position on each frame image captured by the camera can be pre-established in the mobile phone. For example, the upper right corner of the mobile phone display screen can be used as the origin O1, and a first coordinate system can be established with O1 as the origin. Similarly, the upper right corner of each frame image captured by the camera can be used as the origin O2, and a second coordinate system can be established with O2 as the origin. There is a mapping relationship between each coordinate point in the first coordinate system and each coordinate point in the second coordinate system.

[0127] When the mobile phone detects the preset gesture input by the user, the mobile phone can determine the position of the user's palm in the preset gesture (i.e., point A1) as the starting position of the user's input preset gesture. Then, the mobile phone can determine the point P corresponding to the starting position A1 in the first coordinate system. In other words, the current position of the user's palm corresponds to point P in the display screen. Then, if Figure 5 As shown, taking the mobile phone displaying the desktop as an example, the mobile phone can display a preset icon at point P on the desktop, thereby prompting the user that the current location of the palm corresponds to point P on the first display interface. Optionally, the preset icon can also move on the desktop with the preset gesture input by the user.

[0128] For example, the first display interface can be any display interface of the mobile phone in a bright screen state (such as an unlocked state), such as the first display interface can be the desktop of the mobile phone; or the first display interface can be the display interface of an application, etc., without limitation. The embodiment of the present application takes the first display interface as the desktop of the mobile phone as an example for illustration.

[0129] Exemplarily, the above-mentioned preset icons can be patterns such as "small hand icon", "arrow", "small square", etc. The embodiment of the present application does not impose any restrictions on the display effects such as size, position, shape and color of the preset icons. Figure 5 The preset image is a “small hand icon” as an example for illustration.

[0130] In this way, after the mobile phone enters the air mouse mode of air gesture, the mobile phone can prompt the user that it is currently in air mouse mode by displaying a preset icon; in this way, the user can continue to input the next air gesture through the preset icon, which improves the efficiency of human-computer interaction and enhances the user experience.

[0131] Exemplary, combined Figure 3 ,like Figure 7 As shown, S201 can be implemented by the following steps. Taking the example that the application layer of the mobile phone includes the system UI and the Sensing Hub includes the first empty mouse detection module, S201 can specifically include steps ① to ⑥.

[0132] Step ①: The first empty mouse detection module obtains multiple frames of images captured by the camera of the mobile phone.

[0133] Optional, such as Figure 3 As shown, after the AO application of the mobile phone is started, the camera is initialized through the air mouse control module in the hardware abstraction layer. Initializing the camera means restoring the camera to the initial setting or state. For example, setting various parameters of the camera configuration to a predefined initial state.

[0134] Optionally, after the camera is initialized, the mobile phone calls the camera driver of the kernel layer to drive the camera to collect multiple frames of images. For example, the camera driver can drive the camera to collect multiple frames of images based on some pre-configured image parameters (such as resolution, size, etc.).

[0135] Exemplarily, the photosensitive element in the camera converts the received light signal into an electrical signal, and sends the obtained electrical signal to the ISP for processing, and finally obtains multiple frames of images.

[0136] Then, the camera calls the always sensing camera firmware (ASC FM) in the kernel layer to transmit the obtained multi-frame images to the first air mouse module.

[0137] Step ②: The first empty mouse detection module detects whether the user inputs a preset gesture.

[0138] Optionally, if the first air mouse detection module detects, based on multiple frames of images, that the user's palm has "two fingers open and three fingers other than the two fingers are folded together", it is determined that the user has input a preset gesture.

[0139] Exemplarily, if each of the above multiple frames of images contains the user's palm, and the user's palm is in a gesture of "two fingers open, three fingers other than the two fingers folded together", it is determined that the user has input a preset gesture.

[0140] Step ③: The first air mouse detection module sends a first event to the air mouse control module of the hardware abstraction layer.

[0141] The first event is used to indicate that the user has input a preset gesture.

[0142] Optionally, the first air mouse detection module may perform information exchange through a Qualcomm messaging interface (QMI) to transmit the air mouse event to the air mouse control module of the hardware abstraction layer.

[0143] Step ④: The air mouse control module of the hardware abstraction layer initializes the air mouse driver of the kernel layer and sends a command to the air mouse driver to indicate starting the air mouse mode.

[0144] Step ⑤: The air mouse driver responds to the command and enters the air mouse mode of air gesture.

[0145] Step ⑥: The air mouse control module of the hardware abstraction layer sends a first event to the system UI of the application layer to indicate that the user has input a preset gesture.

[0146] Correspondingly, the system UI displays a preset icon in response to the first event.

[0147] Optionally, the air mouse control module calls a preset interface to send a first event to the middleware, and the middleware encapsulates the air mouse event and sends it to the system UI through the IMS of the framework layer.

[0148] For example, the air mouse control module of the hardware abstraction layer can call the Android interface definition language (AIDL) to send air mouse events to the middleware (or smart middleware). Alternatively, the air mouse control module of the hardware abstraction layer can call the hardware abstraction layer interface definition language (HAL interface definition language, HIDL) to send air mouse events to the middleware, without limitation.

[0149] In the embodiment of the present application, since the first event sent by the air mouse control module to the system UI is a signal, and the signaling interaction involves IP ports, signaling formats, transmission methods, etc., after the air mouse control module sends the first event to the middleware, the middleware encapsulates the signaling corresponding to the first event and then sends it to the system UI of the application layer, which is beneficial to improving the reliability of the signaling interaction.

[0150] It should be noted that the embodiment of the present application does not impose any restrictions on the order of execution of steps ④, ⑤ and ⑥. The mobile phone can execute steps ④, ⑤ and ⑥ at the same time, or can execute them in the order of steps ④, ⑤ and ⑥ without restriction.

[0151] After the mobile phone detects the preset gesture input by the user through the camera, it can not only display the preset icon on the first display interface in response to the preset gesture, but also execute step S202 in response to the above preset gesture.

[0152] S202. In response to the above-mentioned preset gesture, the mobile phone starts to detect a first air gesture input by the user through the camera. In response to the first air gesture, the mobile phone displays an air mouse cursor on the first display interface.

[0153] In S202, after the mobile phone detects that the user has input a preset gesture, it indicates that the user wants the mobile phone to enter the air mouse mode of the air gesture. Then, the mobile phone can continue to use the camera to capture multiple frames of images and detect whether the user has input the first air gesture; if the mobile phone detects that the user has input the first air gesture, the mobile phone displays the air mouse cursor on the first display interface.

[0154] For example, the first air gesture may include "pinch two fingers together, and bring three fingers together except the two fingers together"; or "pinch two fingers together and then open"; or "open two fingers and hover" (such as hovering for more than 2 seconds); or "pinch five fingers together", without restriction. The two fingers may include the thumb and the index finger, without restriction.

[0155] Taking the first air gesture of "two-finger pinch, three fingers other than two fingers pinch together" as an example, for example, Figure 8 As shown, the first air gesture can be input within the shooting range of the mobile phone's camera. The camera can capture multiple frames of images of the user's palm, and the mobile phone can recognize whether the user is currently inputting the above-mentioned first air gesture based on the multiple frames of images captured by the camera.

[0156] like Figure 8 As shown, optionally, the mobile phone can display an empty mouse cursor at a preset position 2 (or first preset position) of the first display interface; or, the mobile phone can display an empty mouse cursor at a reference position of the first display interface. Exemplarily, the reference point can be the center point of the user's palm.

[0157] It should be noted that the specific implementation method of displaying the empty mouse cursor at the reference position of the first display interface by the mobile phone can refer to the above Figure 6 The embodiments shown are not described in detail here.

[0158] Exemplarily, the above-mentioned air mouse cursor can be a pattern such as an "arrow" or a "transparent circle". The embodiment of the present application does not impose any restrictions on the display effects such as the size, position, shape and color of the preset icon.

[0159] In this way, the mobile phone can display the air mouse cursor to prompt the user that the air mouse cursor can be moved through air gestures; in this way, the user can continue to input the next air gesture through the air mouse cursor, thereby improving the efficiency of human-computer interaction and enhancing the user experience.

[0160] Exemplary, combined Figure 7 As in the above embodiment, S202 may be specifically Figure 7 The specific implementation process of steps ① to ③ and step ⑥ shown in the figure can refer to the above embodiment and will not be repeated here. It can be understood that, combined with Figure 3 As shown, the camera sends the collected multiple frames of images to the first air mouse detection module in the Sensing Hub. After the first air mouse detection module determines that the user has input the first air gesture, it sends a second event to the air mouse control module of the hardware abstraction layer. The second event is used to indicate that the user has input the first air gesture. After receiving the second event, the air mouse control module sends the first event to the system UI. In response to the second event, the system UI displays the air mouse cursor on the first display interface.

[0161] It should be noted that, in the embodiment of the present application, S201 is an optional step. In actual implementation, the mobile phone may execute S201 to display a preset icon, and after displaying the preset icon, execute S202. Of course, the mobile phone may not execute S201, but directly execute S202, that is, directly display the air mouse cursor through the first air gesture. In this case, S202 may be specifically performed by Figure 7 It is implemented by steps ① to ⑥ shown.

[0162] S203. In response to the first air gesture, the mobile phone starts to detect the second air gesture input by the user through the camera, and in response to the second air gesture, the mobile phone moves the air mouse cursor along the moving track of the second air gesture.

[0163] Exemplarily, in the second air gesture, the palm of the user's palm is opposite to the touch screen of the mobile phone, that is, the palm of the user's palm is facing the front camera of the mobile phone. For example, the second air gesture includes "index finger extended". In addition, in the embodiment of the present application, there is no limitation on the posture of the four fingers other than the index finger. For example, the four fingers other than the index finger can be extended at the same time, or three fingers other than the index finger can be extended, or four fingers other than the index finger can be folded, etc., without restriction.

[0164] In the embodiment of the present application, the reason why the palm of the user is opposite to the touch screen of the mobile phone in the second air gesture is that the palm of the user is easier to identify in the multiple frames of images captured by the camera. By determining the position of the palm, the air mouse cursor can be controlled to move with the movement of the second air gesture.

[0165] Take the second air gesture of "index finger extended, four fingers other than the index finger brought together" as an example. Fig. 9 As shown, the second air gesture can be input within the shooting range of the mobile phone camera (e.g., the front camera). Since the camera of the mobile phone is always on, the camera can capture multiple frames of images of the user's palm, and the mobile phone can recognize whether the user is currently inputting the above-mentioned second air gesture based on the multiple frames of images captured by the camera.

[0166] When the mobile phone recognizes that the user inputs the second air gesture, Fig.10 As shown, the mobile phone can obtain N frames of images collected by the camera within the most recent period of time (such as 200ms) in real time. In addition, a two-dimensional rectangular coordinate system (or a spatial rectangular coordinate system) can be preset in the mobile phone. Fig.10 As shown in the figure, the x-axis and y-axis are established with the upper right corner of each frame image as the origin. The x-axis and y-axis can be in pixels, or can be set in distance units such as mm (millimeter) and cm (centimeter). The mobile phone can recognize the pattern of the user's palm in these N frames of images and the reference point C in the pattern. The reference point C can be any point in the pattern of the user's palm; for example, the reference point C can be the end point of the index finger of the user's palm, or the reference point C can be the center point of the user's palm.

[0167] Exemplarily, the mobile phone can determine the moving distance of the user's palm according to the coordinates of the reference points C1, C2, ..., C(n) in the above N frames of images. Then, the mobile phone moves the air mouse cursor along the moving trajectory of the second air gesture according to the moving distance of the user's palm. The coordinates of the reference point C can be called the moving position.

[0168] As another example, the mobile phone can determine the moving distance of the user's palm in each two adjacent frames of images according to the coordinates of the reference points C1, C2, ..., C(n) in the above N frames of images. Furthermore, the mobile phone can move the air mouse cursor along the moving trajectory of the second air gesture according to the moving distance of the user's palm in each two adjacent frames of images.

[0169] As another example, the mobile phone may move the air mouse cursor along the moving trajectory of the second air gesture based on the coordinates of the reference points C1, C2, C3, ..., C(n) in the above-mentioned N frames of images.

[0170] Optionally, after the mobile phone detects that the user has input the second air gesture, it can also record the position of the user's palm at this time. Fig.10The coordinates of the reference point C1 of the user's palm in the first frame image are used as the position of the user's palm, and this position can also be used as the starting position for the subsequent movement of the air mouse cursor.

[0171] Then, the mobile phone moves the air mouse cursor along the moving trajectory of the second air gesture according to the starting position and the positions of other reference points (such as the coordinates of reference points C2, C3, ..., C(n)).

[0172] like Fig.11 As shown, S203 can be implemented by the following steps. Taking the example of the mobile phone moving the empty mouse cursor based on the coordinates of the reference points C1, C2, ..., C(n) in the above N frames of images, S203 can specifically include steps a to e.

[0173] Step a: The second empty mouse detection module obtains N frames of images collected by the mobile phone through the camera.

[0174] For example, the mobile phone calls ASC FM through the camera to send the collected N frames of images to the second empty mouse detection module.

[0175] Step b: The second empty mouse detection module identifies the pattern of the user's palm in the N frames of images and the reference point C in the pattern, and determines the coordinates of the reference point C in each frame of the image.

[0176] For example, the second empty mouse detection module may determine the coordinates of the reference point C in each frame of the image based on a preset two-dimensional rectangular coordinate system.

[0177] Step c: The second air mouse detection module sends the coordinates of the reference point C in each frame of the image to the air mouse control module of the hardware abstraction layer.

[0178] Step d: The air mouse control module sends the coordinates of the reference point C in each frame image to the IMS of the framework layer.

[0179] For example, the air mouse control module directly sends the coordinates of the reference point C in each frame of the image to the IMS of the framework layer without calling the middleware for sending. In this way, the delay of coordinate data transmission can be reduced.

[0180] Step e: IMS controls the movement of the air mouse cursor based on the coordinates of the reference point C in each frame of the image.

[0181] Exemplarily, the moving distance of the air mouse cursor is proportional to the moving distance of the second air gesture input by the user.

[0182] S204. As the second air gesture moves, the mobile phone moves the air mouse cursor to the first position area of ​​the first display interface; in response to the third air gesture input by the user, the mobile phone triggers the operation of the air mouse cursor in the first position area to display the second display interface.

[0183] Exemplarily, the third air gesture includes pinching two fingers, and bringing three fingers together except the two fingers; or, the third air gesture includes pinching two fingers and then opening them; or, the third air gesture includes opening and hovering two fingers; or, the third air gesture includes pinching five fingers; or, opening five fingers; or, the third air gesture includes clicking along the z-axis direction of the spatial rectangular coordinate system; wherein the z-axis direction is perpendicular to the horizontal and vertical directions when the mobile phone is displayed.

[0184] For example, two fingers may include a thumb and an index finger; or, two fingers may include an index finger and a middle finger; or, two fingers may include a thumb and a middle finger, etc., without limitation.

[0185] It should be noted that in the embodiment of the present application, the first air gesture and the second air gesture are different, the first air gesture and the third air gesture are different, and the second air gesture and the third air gesture may be the same or different without limitation.

[0186] Optionally, in response to a third air gesture input by the user, the mobile phone triggers a sliding operation of the air mouse cursor in the first position area to display the second display interface.

[0187] For example, Fig.12 As shown in (1) in FIG. 1 , as the “index finger is extended and the other four fingers are brought together” gesture is moved, the mobile phone moves the empty mouse cursor to the top of the desktop for display. Fig.12 As shown in (2) in FIG. 1 , in response to the “two-finger open and hover” gesture input by the user, the mobile phone triggers the air mouse cursor to slide down at the top of the desktop, and the following is displayed: Fig.12 The drop-down bar interface shown in (2) in the figure.

[0188] Furthermore, if Fig.12 As shown in (3) in FIG, as the “index finger is extended, and the four fingers other than the index finger are brought together” gesture moves, the mobile phone moves the air mouse cursor to the “volume icon” for display. Fig.12 As shown in (4) in FIG. 1 , in response to the user's input of the "two-finger open and hover" gesture, the mobile phone triggers the air mouse cursor to slide down on the "volume icon", and the following is displayed: Fig.12 The interface shown in (4) in the figure. It can be seen that Fig.12 The volume bar of the "volume icon" shown in (4) is reduced, and accordingly, the media volume of the mobile phone is reduced.

[0189] Optionally, in response to a third air gesture input by the user, the mobile phone triggers a click operation of the air mouse cursor in the first position area to display the second display interface.

[0190] For example, Fig.13As shown in (1) in FIG. 1 , as the “index finger is extended, and the four fingers other than the index finger are brought together” gesture is moved, the mobile phone moves the empty mouse cursor to the “short video APP” icon for display. Fig.13 As shown in (2) in FIG. 1 , in response to the user input of the gesture of “clicking along the z-axis direction of the spatial rectangular coordinate system”, the mobile phone triggers the mouse cursor to click the “short video APP” icon, and the following is displayed: Fig.13 The interface shown in (2) in FIG. Fig.13 The interface shown in (2) is the video display interface of the "Short Video APP".

[0191] For example, after the user inputs the third air gesture, the mobile phone can collect multiple frames of images through the camera. After each frame of image is obtained, the mobile phone can identify the pattern of the user's palm in the image. Then, the mobile phone can determine the position of the user's palm in the image. For example, the mobile phone can use the coordinates of the end point of the index finger of the user's palm as the position of the user's palm in each frame of image. In this way, the mobile phone can determine the movement distance of the user's palm on the z-axis based on the coordinates of the end point of the index finger in each frame of image.

[0192] Exemplarily, when the movement distance of the user's palm in the z-axis direction is greater than a threshold, it indicates that the third air gesture input by the user is a click operation.

[0193] Exemplary, combined Figure 3 As shown, the camera sends the collected multiple frames of images to the second air mouse detection module in the Sensing Hub. After the second air mouse detection module determines the third air gesture input by the user, it generates an operation event (such as a sliding event, a click event, etc.). The second air mouse detection module sends the operation event to the air mouse control module of the hardware abstraction layer. After receiving the operation event, the air mouse control module sends the operation event to the IMS. In response to the operation event, the IMS controls the air mouse coordinates to perform the operation corresponding to the operation event.

[0194] For example, the IMS controls the empty mouse coordinates to perform a sliding operation corresponding to a sliding event. For another example, the IMS controls the empty mouse coordinates to perform a clicking operation corresponding to a clicking event.

[0195] In summary, by adopting the solution of the embodiment of the present application, the user's palm is mapped into a virtual wireless mouse, and the air mouse cursor is displayed on the display screen of the electronic device through air gestures, and the air mouse cursor is controlled to move along the moving trajectory of the air gestures, so as to achieve the purpose of continuously operating the electronic device when it is inconvenient for the user. Continuous interactive operations can be achieved without the electronic device memorizing a large number of air gestures, thereby improving the efficiency of human-computer interaction.

[0196] In some embodiments, the mobile phone can start a preset timer after detecting that the user inputs the third air gesture. Taking the 10s timer 1 as an example, after detecting that the user inputs the third air gesture, the mobile phone can start timer 1. When timer 1 does not exceed 10s, the mobile phone can continuously recognize and respond to the user's third air gesture according to the method in the above embodiment.

[0197] When timer 1 exceeds 10s, the mobile phone can automatically exit the air mouse mode of air gestures, and no longer continue to recognize and respond to air gestures performed by the user based on the image captured by the camera. At this time, if the user wants to continue to use the air mouse mode of air gestures to interact with the mobile phone, the user can input the preset gesture to the mobile phone again. Then, the mobile phone can continue to execute the interaction method of the air mouse mode of air gestures provided in the embodiment of the present application from the above S201.

[0198] In addition, when timer 1 times out, the mobile phone can hide the air mouse cursor originally displayed in the current display interface, thereby prompting the user that the mobile phone has exited the air mouse mode of the air gesture.

[0199] In other embodiments, when the user no longer wishes to interact with the mobile phone using the air mouse mode of air gestures, the mobile phone may also exit the air mouse mode in response to the user's operation.

[0200] For example, if the mobile phone does not detect the relevant pattern of the user's palm in the most recent consecutive multiple frames of images (i.e., exceeding the preset time length), it means that the air gesture input by the user has exceeded the shooting range of the camera. At this time, the mobile phone can determine that the air mouse mode of the air gesture has ended, and then exit the air mouse mode.

[0201] Alternatively, if the phone detects that there is no obvious change in the position of the user's palm in the most recent consecutive multiple frames of images, it means that the air gesture input by the user has paused. At this time, the phone can also determine that the air mouse mode of the air gesture has ended, and then exit the air mouse mode.

[0202] Alternatively, if the mobile phone detects a click operation of the user on the touch screen (i.e., display screen) of the mobile phone, in response to the click operation, the mobile phone can also determine that the air mouse mode of the air gesture has ended, and then exit the air mouse mode.

[0203] Alternatively, when the user no longer wants to use the air mouse mode of the air gesture to interact with the mobile phone, the user can also input an end gesture (or fourth air gesture) pre-set to exit the air mouse mode into the camera. Exemplarily, the end gesture can be a palm flip gesture; or, the end gesture can be a gesture of the user clenching a fist and hovering.

[0204] like Fig.14As shown in (1) in , if the mobile phone detects that the image collected in the recent period contains a pattern of the user making a fist, and the position of the user's fist has not changed in the image, the mobile phone can determine that the user has input an end gesture. Then, in response to the end gesture, Fig.14 As shown in (2), the mobile phone can hide the air mouse cursor in the current display interface, thereby prompting the user that the air mouse mode of the air gesture has been exited.

[0205] Exemplary, combined Figure 3 ,like Fig.15 As shown, the mobile phone can exit the air mouse mode of the air gesture by the following steps. For example, it can include steps i to vii.

[0206] Step i: The first empty mouse detection module obtains multiple frames of images captured by the camera of the mobile phone.

[0207] Step ii: The first empty mouse detection module detects whether the user inputs an end gesture.

[0208] Optionally, if the first empty mouse detection module detects that the images collected in the recent period contain a pattern of the user making a fist, and the position of the user making a fist does not change in the image, it can be determined that the user has input an end hand.

[0209] Step iii: The first air mouse detection module sends a third event to the air mouse control module of the hardware abstraction layer.

[0210] The third event is used to indicate that the user has input an end gesture.

[0211] Step IV: The air mouse control module of the hardware abstraction layer sends a command to the air mouse driver of the kernel layer to instruct to exit the air mouse mode.

[0212] Step V: The air mouse driver of the kernel layer responds to the instruction and exits the air mouse mode of the air gesture.

[0213] Step VI: The air mouse control module of the hardware abstraction layer sends the third event to the system UI of the application layer.

[0214] Step ⅶ: The system UI of the application layer hides the empty mouse cursor in response to the third event.

[0215] It should be noted that for the specific implementation process of step i to step ⅶ, please refer to step ① to step ⑥ in the above embodiment, which will not be repeated here.

[0216] It should be noted that the contents recorded in each embodiment of the embodiments of the present application can explain the technical solutions in other embodiments of the embodiments of the present application, and the technical features recorded in each embodiment can also be applied in other embodiments to form new solutions by combining the technical features in other embodiments. The present application only exemplarily lists several embodiments for illustration, and does not mean that the present application is limited to this.

[0217] The present application embodiment provides an electronic device, which may include a memory and one or more processors; the memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the functions or steps executed by the mobile phone in the above embodiment. The structure of the electronic device can refer to the above Figure 1 The structure of the electronic device 100 is shown.

[0218] The present application also provides a chip system for use in electronic devices. Fig.16 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 may be the processor of the above embodiment. Figure 1 The processor 110 is shown. The interface circuit 1102 may be, for example, an interface circuit between the processor and an external memory; or an interface circuit between the processor and an internal memory.

[0219] The processor 1101 and the interface circuit 1102 can be interconnected via lines. For example, the interface circuit 1102 can be used to receive signals from other devices (such as the memory of the electronic device 100). For another example, the interface circuit 1102 can be used to send signals to other devices (such as the processor 1101). Exemplarily, the interface circuit 1102 can read the instructions stored in the memory and send the instructions to the processor 1101. When the instructions are executed by the processor 1101, the electronic device can execute the various functions or steps executed by the mobile phone in the above embodiment. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiments of the present application.

[0220] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step executed by the mobile phone in the above method embodiment.

[0221] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0222] It should be noted that the terms "first" and "second" in the embodiments of the present application and the accompanying drawings are used to distinguish different objects rather than to describe a specific order. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0223] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0224] It should be understood that in the present application, "at least one (item)" refers to one or more. "Multiple" refers to two or more. "At least two (items)" refers to two or three and more than three. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple. “When” and “if” both mean that corresponding measures will be taken under certain objective circumstances. It does not limit the time, nor does it require any judgment when it is implemented, nor does it mean that there are other limitations.

[0225] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0226] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0231] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for interactive air gestures, It is characterized in that include: The electronic device displays a first display interface; In response to a first air gesture input by a user, the electronic device displays an air mouse cursor on the first display interface; In response to a second air gesture input by the user, the electronic device moves the air mouse cursor along a moving track of the second air gesture; As the second air gesture moves, the electronic device moves the air mouse cursor to a first position area of ​​the first display interface; In response to a third air gesture input by the user, the electronic device triggers the operation of the air mouse cursor in the first position area to display a second display interface.

2. The method according to claim 1, It is characterized in that In response to the first air gesture input by the user, the electronic device displays an air mouse cursor on the first display interface, including: In response to the first air gesture input by the user, the electronic device displays the air mouse cursor at a first preset position of the first display interface.

3. The method according to claim 1 or 2, It is characterized in that A rectangular coordinate system is pre-set in the electronic device; Wherein, in response to the second air gesture input by the user, the electronic device moves the air mouse cursor along the moving track of the second air gesture, including: In response to a second air gesture input by the user, the electronic device determines a starting position of the second air gesture mapped in the rectangular coordinate system; As the second air gesture moves, the electronic device determines at least one moving position of the second air gesture during the moving process; The electronic device moves the air mouse cursor along the moving trajectory of the second air gesture according to the starting position and the at least one moving position.

4. The method according to any one of claims 1 to 3, It is characterized in that Before the electronic device displays an air mouse cursor on the first display interface in response to the first air gesture input by the user, the method further includes: The electronic device detects a preset gesture input by a user, where the preset gesture is used to enter an air mouse mode of an air gesture; In response to the preset gesture input by the user, the electronic device displays a preset icon at a second preset position of the first display interface.

5. The method according to claim 4, It is characterized in that The method further comprises: If the electronic device fails to detect the air gesture for a preset period of time, the electronic device exits the air mouse mode; or In response to a user clicking the touch screen of the electronic device, the electronic device exits the air mouse mode; or, In response to a fourth air gesture input by the user, the electronic device exits the air mouse mode.

6. The method according to claim 5, It is characterized in that The method further comprises: The electronic device hides the empty mouse cursor.

7. The method according to any one of claims 1 to 6, It is characterized in that In the second air gesture, the palm of the user is opposite to the touch screen of the electronic device.

8. The method according to claim 7, It is characterized in that The second air gesture includes extending the index finger and bringing four fingers other than the index finger together.

9. The method according to any one of claims 1 to 8, It is characterized in that The first air gesture includes pinching two fingers together, and bringing three fingers other than the two fingers together; wherein the two fingers include a thumb and an index finger; or, The first air gesture includes pinching two fingers together and then opening them; or, The first air gesture includes opening and hovering with two fingers; or, The first air gesture includes pinching with five fingers.

10. The method according to any one of claims 4 to 9, It is characterized in that The first preset gesture includes spreading two fingers and bringing three fingers other than the two fingers together; wherein the two fingers include a thumb and an index finger.

11. The method according to claim 5, It is characterized in that The fourth air gesture is a palm flip gesture; or, The fourth air gesture is a fist hovering gesture.

12. The method according to any one of claims 1 to 11, It is characterized in that In response to the third air gesture input by the user, the electronic device triggers the operation of the air mouse cursor in the first position area to display the second display interface, including: In response to a third air gesture input by the user, the electronic device triggers a click operation of the air mouse cursor in the first position area; In response to the click operation, the electronic device displays the second display interface.

13. The method according to claim 12, It is characterized in that The third air gesture includes pinching two fingers together, and bringing three fingers other than the two fingers together; wherein the two fingers include the thumb and the index finger; or, The third air gesture includes pinching two fingers together and then opening them; or, The third air gesture includes opening and hovering with two fingers; or, The third air gesture includes pinching with five fingers; or, The third air gesture includes clicking along the z-axis direction of the spatial rectangular coordinate system; wherein the z-axis direction is perpendicular to the horizontal direction and the vertical direction of the electronic device when displayed.

14. An electronic device, It is characterized in that include: memory and one or more processors; The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 13.

15. A chip system, It is characterized in that The chip system includes: at least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; the at least one processor executes the instructions so that the electronic device executes the method as described in any one of claims 1-13.

16. A computer-readable storage medium, It is characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Images