Over-the-air operation method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an air operation method, electronic equipment and a storage medium, and relates to the technical field of terminals.The method comprises the steps that in response to first air operation of a user, an air mouse mode is started, and a cursor is displayed in a display interface of the electronic equipment; in response to a second air operation of the user on the cursor, executing a first preset response event corresponding to the second air operation; and in response to the air gesture operation of the user, executing a second preset response event corresponding to the air gesture operation. Global air operation can be provided for a user, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a method for air operation, an electronic device, and a storage medium. Background Art

[0002] Currently, intelligent electronic devices such as mobile phones can apply the always-on camera (AON) function, which can realize functions such as air gesture control of the screen. When the AON function is applied to a mobile phone, the front camera of the mobile phone is always on, and images can be collected in real time. The mobile phone performs gesture recognition through image analysis and can respond to user gestures to control the screen without the user touching the mobile phone, thus realizing the control of the mobile phone screen. For example, users can perform certain actions such as answering calls, browsing content, and taking screenshots through gestures such as air pressing, air grasping, and air sliding.

[0003] However, the current air gesture function has the characteristics of functional certainty and discontinuous operation, that is, only some certain functions can be realized through the air gesture function. When the user is inconvenient to touch the mobile phone screen, there are some experience breakpoints when using the air gesture function. For example, the current air gesture function does not support precise click operations, and operations such as volume adjustment, pause, and fast forward cannot be performed, and global air operation cannot be realized, affecting the user experience. Summary of the Invention

[0004] This application provides a method for air operation, an electronic device, and a storage medium to realize global air operation and improve the user experience.

[0005] In a first aspect, an embodiment of this application provides a method for air operation, which is applied to an electronic device. The method includes: in response to a first air operation of a user, starting an air mouse mode and displaying a cursor on the display interface of the electronic device; in response to a second air operation of the user on the cursor, executing a first preset response event corresponding to the second air operation; in response to an air gesture operation of the user, executing a second preset response event corresponding to the air gesture operation.

[0006] Among them, each time the user needs to start the air mouse mode, it needs to be started through the first air operation. After entering the air mouse mode, the electronic device will collect the user's gestures in real time through the camera and recognize the gestures related to the air mouse, so as to perform corresponding operations. In the embodiments of the present application, the first air operation is an operation on the air mouse. For example, keeping the fingers open and moving to move the cursor, or performing a two-finger pinch click to click an icon. The air gesture operation refers to an operation through the movement of the palm. For example, swiping left and right on the page by waving the wrist to the left and right respectively, and performing an air return by flipping the palm. In the embodiments of the present application, coarse-grained control can be performed through air gestures, and fine-grained control can be performed through the second air operation, such as click operations, etc., which can meet the user's air operation requirements in various scenarios, avoid experience breaks, and can realize global air operations, providing a better user experience.

[0007] In a possible implementation manner, in response to the user's second air operation on the cursor, execute the first preset response event corresponding to the second air operation, including: if the first preset response event is an event of moving the cursor, move the cursor according to the moving distance of the second air operation; if the first preset response event is a click event, trigger a click response on the position where the cursor is located; if the first preset response event is a first sliding event, perform a sliding operation on the object pointed to by the cursor.

[0008] For example, the user can move the cursor to the icon to be clicked through the second air operation, and then click the icon through the second air operation. For another example, if the user needs to adjust the volume, a sliding operation can be performed on the volume adjustment control to drag the volume bar to achieve the effect of adjusting the volume.

[0009] It can be seen that in the embodiments of the present application, the user can move the cursor, perform click operations or perform sliding operations on an object through the second air operation, which can make up for the discontinuity of the current air gesture operations and provide a more coherent air operation experience for the user.

[0010] In a possible implementation manner, in response to the user's air gesture operation, execute the second preset response event corresponding to the air gesture operation, including: if the second preset response event is a return event, return to the previous interface of the current display interface; if the second preset response event is a second sliding event, display the interface after sliding. For operations that do not require precise control, the user can choose to implement them through the air gesture function, which is more convenient to operate.

[0011] In a possible implementation, before starting the air mouse mode in response to the user's first air operation, the method further includes: if the electronic device is in the screen-off state, in response to the user's air gesture operation to light up the screen, light up the screen of the electronic device. For example, the air gesture operation can be an operation to light up the screen by hovering, that is, when the user hovers the palm in front of the screen, the electronic device can light up the screen. When the user is inconvenient to touch the screen, the screen can be conveniently lit up.

[0012] In a possible implementation, after executing the second preset response event corresponding to the air gesture operation in response to the user's air gesture operation, the method further includes: in response to not recognizing the user's second air operation on the cursor within a preset duration, turn off the air mouse mode. As an example, the preset duration can be 20 seconds. That is, when the user's use of the air mouse is not recognized within the preset duration, the air mouse mode can be turned off, and subsequent real-time recognition of the user's air operation on the air mouse is not required, which can reduce the power consumption of the electronic device.

[0013] In a possible implementation, the method further includes: receiving the user's operation to enable the air mouse function for the first time, and enabling the air mouse function; in response to the user's operation on the air mouse for a preset number of times before, display a notification card in the notification bar, where the notification card includes a first control for specifying the air gesture function; in response to the user's click operation on the area other than the first control in the notification card, display a play tips introduction page for the air gesture function; in response to the user's click operation on the first control, display a switch page for the specified air gesture function, where the switch page includes a switch control; in response to the user's click operation on the switch control, enable the specified air gesture function.

[0014] It can be understood that each displayed notification card includes a first control for a different specified air gesture function. For example, when the user's first air operation on the air mouse is recognized, the notification card is used to notify the user to enable the air gesture function of air return, and when the user's second air operation on the air mouse is recognized, the notification card is used to notify the user to enable the air gesture function of air sliding. By adopting this method, when the user enables the air mouse function for the first time, the user can be reminded to learn and enable the air gesture function, which is convenient for the user to learn the global air operation method and provides convenience for the user's subsequent air operations.

[0015] In a possible implementation, after receiving the operation of the user turning on the air mouse function for the first time and turning on the air mouse function, the method further includes: displaying a learning interface for air mouse operations, and showing an operation prompt message for the air mouse on the learning interface. The operation prompt message is used to prompt the user about the functions and usage methods of the air mouse operation at a distance; if it is recognized that the user performs an air mouse operation according to the operation prompt message, then play the animation effect of the response event corresponding to the operation prompt message on the learning interface; show the next operation prompt message for the air mouse on the learning interface, and return to the step of playing the animation effect of the response event corresponding to the operation prompt message on the learning interface if it is recognized that the user performs an air mouse operation according to the operation prompt message, until it is determined that the user has completed the learning of the air mouse operation, that is, it is determined that the user has performed an air mouse operation according to various operation prompt messages, then exit the learning interface. Alternatively, the user can also actively choose to exit the learning interface.

[0016] After the user turns on the air mouse function for the first time, the playing skills of the air mouse can be shown to the user, and the user can be guided to learn the usage method of the air mouse in an interactive form, so that the user can quickly learn how to use the air mouse, providing convenience for subsequent air mouse operations at a distance.

[0017] In a possible implementation, the first air mouse operation is a two-finger pinch operation.

[0018] In a possible implementation, when the second air mouse operation is an operation of moving while keeping the two fingers open, the first preset response event is an event of moving the cursor; when the second air mouse operation is a two-finger pinch operation, the first preset response event is a click event;

[0019] When the second air mouse operation is an operation of moving while keeping the two fingers pinched, the first preset response event is a first sliding event.

[0020] In a possible implementation, when the air gesture operation is a palm flipping operation, the second preset response event is a return event; when the air gesture operation is an operation of waving the wrist with the palm facing the screen and in a specified direction, the second preset response event is a second sliding event.

[0021] In a second aspect, an embodiment of the present application provides an electronic device, including: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the methods described in the first aspect and any possible implementation manner in the first aspect.

[0022] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processors are configured to call computer instructions to cause the electronic device to execute the methods described in the first aspect and any possible implementation manner of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer program, which, when running on an electronic device, causes the electronic device to execute the methods described in the first aspect and any possible implementation manner of the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code, which, when running on an electronic device, causes the electronic device to execute the methods described in the first aspect and any possible implementation manner of the first aspect.

[0025] It can be understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the methods provided in the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0028] Figure 2 It is a software structure block diagram of an electronic device provided in an embodiment of the present application;

[0029] Figure 3 It is a schematic diagram of a setting interface for an air gesture provided in an embodiment of the present application;

[0030] Figures 4A - 4F It is a schematic diagram of setting interfaces for various air gestures provided in an embodiment of the present application;

[0031] Figures 4G - 4J It is a schematic diagram of a playing skill for an air gesture provided in an embodiment of the present application;

[0032] Figures 5A - 5FSchematic diagram of interface changes in the air gesture scenario in the prior art;

[0033] Figures 6A - 6H Exemplary schematic diagram of the functions supported by the air mouse in the embodiments of the present application;

[0034] Figure 7A Schematic diagram of another air gesture setting interface provided by the embodiments of the present application;

[0035] Figure 7B Schematic diagram of a setting interface of an air mouse provided by the embodiments of the present application;

[0036] Figure 7C and Figure 7D Schematic diagram of the notification method of the play skills of air return provided by the embodiments of the present application;

[0037] Figure 7E Schematic diagram of the play skills interface of air operation provided by the embodiments of the present application;

[0038] Figure 7F Schematic diagram of the home page of the play skills provided by the embodiments of the present application;

[0039] Figure 7G and Figure 7H Schematic diagram of the user starting to use the air mouse through the starting gesture of the air mouse;

[0040] Figure 8 Flowchart of an air operation provided by the embodiments of the present application;

[0041] Figures 9A - 9G Schematic diagram of the interface change of Scenario 1 provided by the embodiments of the present application;

[0042] Figures 10A - 10D Schematic diagram of the interface change of Scenario 2 provided by the embodiments of the present application;

[0043] Figures 11A - 11D Schematic diagram of the interface change of Scenario 3 provided by the embodiments of the present application. Detailed implementation manners

[0044] To better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] It should be noted that in the present application, words such as "exemplarily" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the present application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Exactly speaking, the use of words such as "exemplarily" or "for example" is intended to present relevant concepts in a specific manner.

[0046] An embodiment of the present application provides a method for air operation. This method is applied to an electronic device, which can be a mobile phone, a tablet computer, a television, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or other electronic devices.

[0047] Figure 1 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 1 The illustrated electronic device may include a processor 110, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, a key 190, a motor 191, an indicator 192, a camera 1-N 193, a display screen 194, and a subscriber identification module (SIM) card interface 1-N 195, etc. Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0048] It can be understood that the structure schematically shown in the embodiment of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0049] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor, a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0050] Among them, the controller may be the nerve center and command center of the electronic device. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0051] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0052] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, etc.

[0053] The wireless communication function of the electronic device may be implemented by antenna 1, antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0054] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0055] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through Antenna 1, and perform processing such as filtering and amplifying on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 150 can be arranged in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be arranged in the same device.

[0056] In the application embodiments, the mobile communication module can also be referred to as a cellular module, and the two can be used interchangeably.

[0057] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to speakers, receivers, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor can be an independent device. In some other embodiments, the modulation and demodulation processor can be independent of the processor 110 and arranged in the same device as the mobile communication module 150 or other functional modules.

[0058] The wireless communication module 160 may provide solutions for wireless communications applied to an electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (WIFI) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation. In some embodiments, at least some functional modules of the wireless communication module 160 may be provided in the processor 110.

[0059] In some embodiments of the present application, the electronic device may establish a wireless connection with other electronic devices through the wireless communication module 160 (such as a Bluetooth module and a WLAN module) and the antenna 2 to achieve data transmission between the electronic device and other electronic devices.

[0060] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technologies.

[0061] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0062] The charging management module 140 is used to receive a charging input from a charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0063] The electronic device realizes the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0064] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than 1.

[0065] The electronic device can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor, etc.

[0066] The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0067] The digital signal processor is used to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0068] The electronic device can realize the audio function through the audio module 170, the speaker, the receiver, the microphone, the headphone jack, and the application processor, etc. Such as music playback, recording, etc.

[0069] The external memory interface 120 can be used to connect to an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device.

[0070] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as screen mirroring function, network sharing function, etc.). The data storage area can store the data created during the use of the electronic device (such as video data, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0071] The button 190 includes a power-on button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device can receive button inputs and generate key signal inputs related to the user settings and function controls of the electronic device.

[0072] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. The indicator 192 can be an indicator light and can be used to indicate the charging status, battery level changes, and can also be used to indicate messages, missed calls, notifications, etc.

[0073] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device. The electronic device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0074] The software system of the above-mentioned electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of this application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software system of the electronic device. As Figure 2 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers,

[0075] from top to bottom are the application layer, application framework layer, Android Runtime and system libraries, and kernel layer.

[0076] The application layer can include a series of application packages. As Figure 2 shown, the application packages can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0077] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. As Figure 2 shown, the application framework layer can include a window manager, content provider, view system, phone manager, resource manager, notification manager, and algorithm framework, etc.

[0078] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0079] The content provider is used to store and retrieve data and make this data accessible to applications. The data may include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0080] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build the display interface of an application. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0081] The phone manager is used to provide the communication functions of the terminal device. For example, the management of call states (including answering, hanging up, etc.).

[0082] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0083] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is complete, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application, or a notification that appears in the form of a dialogue window on the screen. For example, it prompts text information in the status bar, emits a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0084] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0085] The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0086] The system libraries can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing libraries (such as: OpenGL ES), 2D graphics engines (such as: SGL), etc.

[0087] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0088] The media library supports the playback and recording of various common audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0089] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0090] The 2D graphics engine is a drawing engine for 2D drawing.

[0091] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, a sensor driver, etc.

[0092] In the following, the embodiments of the present application use an electronic device with Figure 1 and Figure 2 the structure shown as an example to introduce in detail the air operation method provided by the embodiments of the present application.

[0093] For ease of understanding, first, the air gesture function involved in the embodiments of the present application is introduced.

[0094] The air gesture function means that the user makes an air gesture within the shooting range of the camera of the electronic device. The camera of the electronic device performs image acquisition. After the image including the user's gesture is acquired, the air gesture in the image is recognized, and then the electronic device executes a response corresponding to the air gesture.

[0095] As Figure 3 shown, Figure 3 it is the setting interface for air gestures. Figure 3 Exemplarily shown in it are the setting items for air screen sliding, air screenshot, air return, and air press. In Figure 3 these functions are all in the "turned off" state. The embodiments of the present application do not limit the setting items included in the setting interface for air gestures. For example, it may also include setting items such as hovering to turn on the screen. Optionally, Figure 3 other setting items that the user may need to set are also shown in it, such as the setting items for hovering to turn on the screen and intelligent perception, and are accompanied by a text prompt "Are you looking for other setting items?" If the user clicks on any setting item, it will jump to the corresponding setting interface for that setting item.

[0096] Figures 4A - 4F It is the setting interface for various air gestures, which can also be called the switch interface for air gestures, and the following will be described separately.

[0097] Among them, the air screen sliding supports four sliding directions, namely sliding up, sliding down, sliding left, and sliding right. Figure 4A and Figure 4BExemplary schematic diagrams of the setting interfaces for swiping left and right respectively. Figure 4A and Figure 4B both include the interface guide 401 for swiping the screen in the air. By swiping the interface left and right, the user can respectively open the setting interfaces for swiping up, swiping down, swiping left, and swiping right. The 4 dots included in the interface guide 401 respectively represent 4 setting interfaces. For example, in Figure 4A , the 3rd black dot is solid, indicating that the 3rd interface of the setting interface for swiping the screen in the air is the setting interface for swiping left. Another example, in Figure 4B , the 4th black dot is solid, indicating that the 4th interface of the setting interface for swiping the screen in the air is the setting interface for swiping right. It can be understood that by swiping the screen left in the interface of Figure 4A , the interface shown in Figure 4B can be opened.

[0098] Figure 4A and Figure 4B both include the gesture operation instructions for swiping the screen in the air. Figure 4A shows the gesture schematic diagram for swiping the screen left in the air and is accompanied by a text description: "At a distance of 20 - 40 cm from the screen, with the palm facing the screen, stay for a while. After the gesture icon appears above, wave the wrist to the left to swipe the screen to the left." Figure 4B shows the gesture schematic diagram for swiping the screen right in the air and is accompanied by a text description: "At a distance of 20 - 40 cm from the screen, with the palm facing the screen, stay for a while. After the gesture icon appears above, wave the wrist to the right to swipe the screen to the right."

[0099] And, in Figure 4A and Figure 4B , both include the switch 402 for swiping the screen in the air and the gesture operation guide. Figure 4A and Figure 4B The switches 402 for swiping the screen in the air in both are in the off state. The user can open the switch by clicking the switch 402 for swiping the screen in the air. Additionally, if the user clicks the gesture operation guide, they will enter the learning interface for swiping the screen in the air, where they can learn the function of swiping the screen in the air in an interactive manner.

[0100] Figure 4C Is an exemplary schematic diagram of the setting interface for returning in the air. Figure 4C includes the gesture operation instructions for returning in the air, specifically shows the gesture schematic diagram for returning in the air, and is accompanied by a text description: "At a distance of 20 - 40 cm from the screen, with the palm facing the screen, stay for a while. After the gesture icon appears above, flip the palm to return to the previous level." And, in Figure 4CIt also includes an air return switch and a gesture operation guide. The switch is in the off state. If the user clicks on the switch, the air return function can be activated. If the user clicks on the gesture operation guide, it will jump to the learning interface of the air return function, where the user can learn the air return function in an interactive manner.

[0101] Figure 4D It is an exemplary schematic diagram of the setting interface for hovering to turn on the screen. Figure 4D It includes a gesture operation description for hovering to turn on the screen, specifically showing a schematic diagram of the gesture for hovering to turn on the screen and accompanied by a text description: "When the device screen is off, keep your hand 20 - 40 cm away from the screen to turn on the screen." And, in Figure 4D It also includes a switch for hovering to turn on the screen. The switch is in the off state. If the user clicks on the switch, the hovering to turn on the screen function can be activated. Also, in the setting interface for hovering to turn on the screen, other setting items that the user may need to set are also shown, such as the setting item for turning on the screen, and accompanied by a text prompt "Are you looking for other setting items?" If the user clicks on the setting item for turning on the screen, it will jump to the setting interface for turning on the screen.

[0102] Figure 4E It is an exemplary schematic diagram of the setting interface for air pressing. Figure 4E It includes a gesture operation description for air pressing, specifically showing a schematic diagram of the gesture for air pressing and accompanied by a text description: "At a distance of 20 - 40 cm from the screen, with the palm facing the screen, stay for a while. After a gesture icon appears above, press forward to play or pause." That is, through the air pressing operation, the user can play and pause videos and music, and can also answer calls. And, in Figure 4E It also includes an air pressing switch and a gesture operation guide. The switch is in the off state. If the user clicks on the switch, the air pressing function can be activated. If the user clicks on the gesture operation guide, it will jump to the learning interface of the air pressing function, where the user can learn the air pressing function in an interactive manner.

[0103] Figure 4F It is an exemplary schematic diagram of the setting interface for air screenshot. Figure 4F It includes a gesture operation description for air screenshot, specifically showing a schematic diagram of the gesture for taking a screenshot and accompanied by a text description: "At a distance of 20 - 40 cm from the screen, with the palm facing the screen, stay for a while. After a gesture icon appears above, grab and make a fist to take a screenshot." And, in Figure 4F It also includes an air screenshot switch and a gesture operation guide. The switch is in the off state. If the user clicks on the switch, the air screenshot function can be activated. If the user clicks on the gesture operation guide, it will jump to the learning interface of the air screenshot function, that is, the playing tips interface, where the user can learn the air screenshot function in an interactive manner.

[0104] Among them, the playing skills of air gestures are as follows Figures 4G - 4J shown. Taking the example of swiping the screen to the left in the air, the learning interfaces of other air gestures are similar, and no further examples will be given here.

[0105] Such as Figure 4G shown, first, a prompt message "Start learning to swipe the screen in the air" will be displayed in the learning interface, and then it will jump to the Figure 4H shown learning interface. In this learning interface, a prompt message "At a distance of 20 - 40 cm from the screen, with your hand facing right, stay for a while until a hand icon appears" will be displayed, and a schematic diagram of the gesture with the hand facing right will be shown.

[0106] After recognizing the user's gesture with the hand facing right, as Figure 4I shown, a hand icon will be displayed in the learning interface, and a prompt message "Swing your wrist to the left and the screen will slide to the left accordingly" will be shown. After recognizing that the user operates according to this prompt message, the dynamic effect of the learning interface sliding to the right will be triggered, showing a new interface, and it will prompt the user that the operation is successful, as Figure 4J shown.

[0107] The following introduces the application of the air gesture function in the prior art in combination with specific application scenarios. When the user's hands are inconvenient to touch the screen, the screen can be operated through air gestures. Taking the example of the user eating crayfish, as Figures 5A - 5F shown, the user's control process is shown.

[0108] When the user starts to eat crayfish, assuming the mobile phone is in the screen-off state and the user hopes to watch videos while eating crayfish, then as Figure 5A shown, the user hovers the hand over the screen and lights up the screen through the hover-to-wake function.

[0109] After the screen is unlocked, as Figure 5B shown, the desktop of the mobile phone is displayed. The user can find the video application program they want to open, such as Figure 5B Douyin in. Since the air gesture function does not support precise clicking on applications, the user cannot open Douyin through air gestures and needs to click on Douyin by touching the screen to open Douyin. It can be seen that at this time, there is an experience breakpoint in the air gesture function, which will bring inconvenience to the user.

[0110] After entering the Douyin interface, as Figure 5C shown, if the user feels that the volume is too low and wants to turn up the volume, but the air gesture operation does not support volume adjustment, so the user cannot adjust the volume through air gestures. It can be seen that at this time, there is an experience breakpoint in the air gesture function, which will bring inconvenience to the user.

[0111] Subsequently, during the process of the user eating crayfish while watching videos, video switching can be performed by swiping the screen in the air. For example, from Figure 5C switch to Figure 5D , and then from Figure 5D switch to Figure 5E . The gesture direction when swiping the screen in the air is related to the settings of the video application. For example, if the video application switches videos by swiping the page left or right, the previous video can be switched to by waving the wrist to the left, and the next video can be switched to by waving the wrist to the right.

[0112] In Figure 5E , if the user sees a good video and wants to collect it, the collection button needs to be clicked. However, the air gesture function does not support precise clicking, so the user cannot collect through air gestures and instead needs to click the collection button by touching the screen. It can be seen that there is an experience breakpoint in the air gesture function at this time, which will bring inconvenience to the user.

[0113] Subsequently, the user can continue to swipe through videos using the air swipe function. If the user wants to drink a beverage and hopes to pause the video while drinking, as Figure 5F shown, the user can try to pause the video by pressing in the air. However, the current air pressing only supports start and pause operations on a simple video interface (such as the video interface of a traditional video player) and does not support start and pause on a complex video interface (such as a short video interface, like the Douyin interface). Therefore, when the user presses in the air, the pause may fail, and there will also be an experience breakpoint, bringing inconvenience to the user.

[0114] It can be seen that in the above process, the air gesture function cannot meet the user's needs, there is an experience breakpoint during the operation of air gestures, and global air operations cannot be achieved, resulting in a poor user experience.

[0115] To solve this problem, an air mouse function can be added in the embodiments of the present application, that is, through the combination of the air gesture function and the air mouse function, global air operations can be achieved, thereby improving the user experience.

[0116] For ease of understanding, the air mouse function involved in the embodiments of the present application is introduced.

[0117] An air mouse, abbreviated as an air mouse, is a non-contact human-computer interaction technology that can collect images or videos related to the user's gestures through the camera of an electronic device, and then detect information such as the user's gestures and positions, and map the detected data to the operation parameters of the cursor, so as to control the air mouse to execute corresponding response events.

[0118] The air mouse supports refined air control functions such as precise clicking, pinching to browse, and pinching to fast forward.

[0119] It should be noted that the pinching and two-finger related operations involved in the embodiments of the present application refer to the actions of two fingers, such as the thumb and the index finger.

[0120] As Figures 6A - 6H shown, Figures 6A - 6H FIG. is an exemplary diagram of the functions supported by the air mouse. When the electronic device has activated the air mouse function, the user can enter the air mouse mode, abbreviated as the air mouse mode, through an air pinching operation. And if no air operation of the user on the air mouse is recognized within a preset duration, the air mouse mode is exited. If the user still needs to use the air mouse, the user needs to enter the air mouse mode again through an air pinching operation.

[0121] As Figure 6A shown, after entering the air mouse mode, the cursor 601 can be displayed on the display interface. In the embodiments of the present application, the cursor 601 is taken as a circle as an example. In actual implementation, the cursor 601 can be other shapes supported by the air mouse, such as an arrow, etc. The embodiments of the present application do not limit this.

[0122] After entering the air mouse mode, the user can control the movement of the cursor through an air operation of spreading the two fingers and moving the two fingers. For example, in Figure 6B , if the user wants to open Douyin, an air operation of spreading the two fingers and moving the two fingers is performed, so as to move the cursor 601 to the Douyin icon. Then the two fingers are pinched together, and the click operation on the Douyin icon can be triggered, and then it jumps to Figure 6C the interface shown.

[0123] In Figure 6C the interface shown, if the user wants to collect a video, the cursor 601 can be moved to the collection button through an air operation of spreading the two fingers and moving the two fingers, and then a pinching operation is performed, and the click operation on the collection button can be triggered. The interface after the click operation is triggered is as Figure 6D shown.

[0124] In addition, in the air mouse mode, precise sliding operations are also supported.

[0125] For example, it supports the user to achieve precise browsing through an air operation. For example, in Figure 6E the display interface shown, if the user wants to control the screen to slide down to browse the subsequent content, the air operation of keeping the two fingers pinched and moving the two fingers can be used to control the screen to slide down, and the distance of the screen sliding is controlled by the distance that the two fingers move. As an example, the display interface after the screen slides down is as Figure 6F shown.

[0126] For another example, it also supports the user to adjust the volume through an air operation. For example, in Figure 6GOn the page shown, the user can control the volume adjustment by performing an air operation of moving two fingers while keeping them pinched. For example, if the two fingers move upward, the volume is increased; if the two fingers move downward, the volume is decreased.

[0127] For another example, it also supports the user to perform fast forward or rewind operations on the video through air operations. For example, in Figure 6H the video playback interface shown, during video playback, the user can control the fast forward or rewind of the video by moving two fingers while keeping them pinched. For example, if the two fingers move to the right, the video is fast forwarded; if the two fingers move to the left, the video is rewound. In Figure 6H the cursor 601 is moved from position 1 to position 2 by a gesture of pinching two fingers, triggering the video to fast forward at twice the speed (2.0X).

[0128] From the above introduction, it can be seen that air gestures support coarse-grained controls such as waking up the screen, swiping, and turning pages, and the air mouse supports precise clicking, and the air mouse supports fine-grained controls such as precise clicking and precise swiping. In the embodiments of the present application, both the air gesture function and the air mouse function are supported. The user can choose to use air gestures or control the mouse according to the specific scenario, and the two functions complement each other, which can provide the user with a global and natural and smooth air operation experience.

[0129] When the user first uses the air mouse, the air mouse switch can be turned on. As Figure 7A shown, Figure 7A is the setting interface for air gestures. An air mouse control is added on the basis of Figure 3 , and Figure 7A the switch of the air mouse in is in the "closed" state. After the user clicks the air mouse control, the air mouse setting interface can be entered. As Figure 7B shown, the setting interface includes the gesture operation instructions of the air mouse. Figure 7B shows a schematic diagram of the usage gesture of the air mouse, and is accompanied by a text description: "Quick start, perform air operations at any time. Naturally extend the thumb and index finger at a distance of 20-40 cm from the screen. Wait for the cursor to appear on the screen, and the air mouse starts". And Figure 7B also includes the switch and gesture operation guide of the air mouse. After the user clicks the switch, the air mouse function is turned on. If the user clicks the gesture operation guide, it jumps to the learning interface of the air mouse, that is, the play skills interface of the air mouse, and then the user can be guided to learn how to use the air mouse through an interactive method.

[0130] After the user first enables the air mouse function, the playing skills of the air mouse can be shown to the user. Specifically, it can jump to the learning interface of the air mouse. In this learning interface, various operation prompt messages for the air mouse will be shown in sequence. After recognizing that the user operates according to an operation prompt message, the next operation prompt message will be shown until the user completes the learning of various air operation methods of the air mouse.

[0131] For example, in the learning interface, the user is prompted to extend the thumb and index finger. After detecting the air operation of the user extending the thumb and index finger, the cursor is displayed in the learning interface, and the user is prompted that the operation is successful. Then, in the learning interface, the user is prompted that pinching with two fingers can quickly enter the air mouse mode. After detecting the pinching action of the user's two fingers, the user is prompted that the operation is successful. Then, in the learning interface, the user is prompted to click the button in the learning interface by pinching with two fingers. If the pinching action of the user's two fingers is detected, the dynamic effect corresponding to the click event of the button is played, and the user is prompted that the operation is successful. Then, in the learning interface, the user is prompted to move while keeping the two fingers pinched. When detecting the operation of the user moving while keeping the two fingers pinched, the screen is scrolled according to the moving direction and distance of the user's two fingers, and the user is prompted that the operation is successful. Then, the next operation prompt message is continued to be shown until the user completes the learning of various air operation methods of the air mouse.

[0132] After exiting the playing skills interface, the user can enter the air mouse mode by pinching with two fingers. It should be noted that when the air mouse switch is in the on state, if the user needs to use the air mouse, the user needs to first enter the air mouse mode by pinching with two fingers, and then the electronic device will further detect the air operation of the user's control of the air mouse. And if no air operation of the user on the air mouse is detected for a preset duration (such as 20 seconds), the air mouse mode will be exited. In this way, it can avoid misoperations when the user does not need the air mouse and can save the power consumption of the electronic device.

[0133] After first entering the air mouse mode, when the air operation of the user on the air mouse is recognized for the first preset number of times, a notification card can be popped up in the notification bar to notify the user to turn on common air gestures.

[0134] For example, when the air operation of the user on the air mouse is recognized for the first time, a notification card is popped up to remind the user to turn on the air return function. When the air operation of the user on the air mouse is recognized for the second time, a notification card is popped up to remind the user to turn on the air sliding function.

[0135] Taking the notification card that reminds the user to turn on the air return function as an example, such as Figure 7CAs shown, the notification card can be a notification for playing device skills. The title "Air Return" is displayed in the notification card, and the prompt message "When performing air operations, you can use quick gestures to conveniently return" is also displayed. Moreover, the notification card also includes a "Go to Enable" control. If the user performs a pull-down operation on the notification bar, it will jump to the interface as shown in Figure 7D i.e., Figure 7C the shown notification card is displayed in the notification interface.

[0136] Subsequently, if the user clicks the "Go to Enable" control in the notification card, it will jump to the air return setting interface as shown in Figure 4C Thus, the user can choose to turn on the air return function and can click the gesture operation guide control to learn how to use the air return function.

[0137] If the user clicks on the area other than the "Go to Enable" control in the notification card, it will jump to the function introduction page of the playing device skills for air operations as shown in Figure 7E This page includes the prompt message "When it is inconvenient to touch the screen, you can perform air operations, making it more convenient to operate the screen". Figure 7E Some air operation methods are exemplarily shown in , such as "1. At a distance of 20 - 40 cm from the screen, face the palm towards the screen, stay for a while, and after the gesture icon appears above, flip the palm to return to the previous level", and "2. At a distance of 20 - 40 cm from the screen, face the palm towards the screen, stay for a while, and after the gesture icon appears above, wave the wrist to the right to slide the screen to the right". In actual implementation, the air operation methods are not limited to this.

[0138] Optionally, the user can also actively open the home page of the playing device skills subsequently. As shown in Figure 7F the home page of the playing device skills includes an option for air operations. When the user clicks on this air operation option, it will jump to the function introduction page of the playing device skills for air operations as shown in Figure 7E In addition, Figure 7F other options for playing device skills are also shown, such as the YOYO suggestions that understand you, intelligent screenshot, intelligent interconnection, image creation, intelligent life enjoyment, practical skills, etc. Figure 7F These are only examples, and the options shown on the home page of the playing device skills in the embodiments of the present application are not limited.

[0139] It can be understood that the user can also actively select other air gesture functions required by themselves through the air gesture setting interface, such as the hover screen lighting function, etc.

[0140] It should be noted that, in order to provide a good user experience, each time the user needs to use the air mouse, the starting gesture can be executed first, and then the mobile phone recognizes the user's starting gesture. For example, the starting gesture can be a gesture of spreading two fingers apart. After recognizing the starting gesture, an icon of the detected user's starting gesture is displayed on the display interface, as Figure 7G shown. After the user sees this icon, subsequent air operations can be performed. For example, video fast-forwarding can be performed through the two-finger pinch and move operation, as Figure 7H shown. It should be noted that in other embodiments, the focus is mainly on the interface changes when the user performs air operations, and the recognition process of the starting gesture is omitted.

[0141] After the user enables the air mouse function and the required air gesture functions, global air operations can be achieved through air gestures and the air mouse, as Figure 8 shown. The air operation method provided by the embodiments of the present application includes the following steps:

[0142] S801. In response to the user's two-finger pinch operation, start the air mouse mode and display the cursor of the air mouse on the display interface of the electronic device.

[0143] S802. In response to the user's air operation on the air mouse, execute the response event corresponding to the air operation.

[0144] Among them, the air operation can be any air operation on the air mouse introduced in the above embodiments.

[0145] For example, if the air operation is that the user spreads two fingers apart and moves the two fingers, then control the cursor to move with the movement of the user's two fingers; for another example, if the air operation is a two-finger pinch action, then trigger a click response on the control at the cursor position. Of course, it can also be other air operations on the air mouse, and no further examples are given here.

[0146] S803. In response to the user's air gesture operation, execute the response event corresponding to the air gesture operation.

[0147] The air gesture operation is any air gesture operation introduced in the above embodiments.

[0148] For example, if the air gesture operation is a palm flip, then return to the previous interface; for another example, if the air gesture operation is a gesture of grasping and making a fist, then take a screenshot of the current display interface. Of course, it can also be other air gesture operations, and no further examples are given here.

[0149] It should be noted that the execution order between S802 and S803 is not limited in the embodiments of the present application, nor is the number of executions of S802 and S803 limited. Users can flexibly use the air gesture operation and the air operation of the air mouse according to their own needs, so as to achieve global air operation, avoid experience breakpoints, and improve the user experience.

[0150] The following introduces the air operation method provided by the embodiments of the present application in combination with three specific scenarios.

[0151] Scenario 1: The scenario where it is inconvenient for the user to touch the mobile phone screen when eating crayfish. The interface changes in this scenario are as Figures 9A - 9G shown.

[0152] When the user starts to eat crayfish, assuming the mobile phone is in the screen-off state and the user hopes to brush videos while eating crayfish, then as Figure 9A shown, the user hovers the hand over the screen and lights up the screen through the hover screen-on function of the air gesture.

[0153] After the screen is unlocked, as Figure 9B shown, the desktop of the mobile phone is displayed. Then, after recognizing the operation of the user pinching with two fingers, the air mouse mode is started. Then, the user moves the cursor to the Douyin icon by keeping the fingers open and moving, and then performs the operation of pinching with two fingers. The mobile phone responds to the operation of the user pinching with two fingers and triggers a click response to the Douyin icon, that is, opens Douyin and jumps to the Douyin interface.

[0154] After entering the Douyin interface, if the user feels that the volume is too low and wants to turn up the volume, the user can click the volume button on the side of the screen to call up the volume adjustment control, as Figure 9C shown. Then, by keeping the two fingers pinched and sliding upward, the increase of the volume is controlled. Or, the user can keep the two fingers pinched and slide downward at the top of the Douyin interface, so that the mobile phone opens the control center, as Figure 9G shown. There is a volume adjustment control in the control center. After recognizing that the user moves the cursor to the volume adjustment control, if it is recognized that the user keeps the two fingers pinched and slides upward, the volume is increased.

[0155] Subsequently, during the process of the user watching videos while eating crayfish, the video can be switched by swiping the screen in the air, that is, when the mobile phone recognizes that the user's palm hovers in front of the screen, it further detects the user's air gesture. If the gesture of waving the wrist upward is recognized, the next video is switched, as Figure 9D shown.

[0156] After switching to the next video, as Figure 9EAs shown in the figure, if the user wants to collect the video, the user keeps two fingers open and moves the cursor to the collection button. Then, if the user's two-finger pinching operation is recognized, the collection response to the collection button is triggered, that is, the collection button switches to the state after being clicked, for example, the collection button changes from white to yellow (in Figure 9E In the example, the collection button becomes solid black, and a reminder box "Collected successfully" is displayed in the interface. Optionally, the reminder box may also include a "Go and see" control. If the user clicks the "Go and see" control by pinching two fingers, the user can jump to the collection interface of Douyin.

[0157] If the user does not want to continue watching the video, he can perform a palm flip gesture. After the phone recognizes the user's palm flip, it can return to the desktop. Figure 9F shown.

[0158] Compared with the existing technology Figures 5A - 5F In the scenario shown, the embodiment of the present application supports air gestures. When the user needs to open Tik Tok, adjust the volume, or collect videos, it can be achieved through the function of the air mouse. That is, the user can perform air operations throughout the process without any breakpoints in the experience. This provides users with a more convenient air operation function and can enhance the user experience.

[0159] Scenario 2: The user replies to WeChat messages and answers calls while watching a video. The interface changes in this scenario are as follows: Figures 10A - 10D shown.

[0160] like Figure 10A As shown in the figure, when a user is watching a video, if a WeChat notification message pops up and it is inconvenient for the user to touch the screen with his hands, he can perform a two-finger pinch operation. In response to the two-finger pinch operation, the phone activates the air mouse mode. Then the user moves the cursor to the notification message by keeping the two fingers open and moving, and then performs a two-finger pinch operation. In response to the two-finger pinch operation, the phone triggers a click on the notification message and then jumps to the WeChat chat interface.

[0161] If the user needs to reply to a WeChat message by voice, such as Figure 10B As shown, the user moves the cursor to the "Press and Speak" control by keeping two fingers apart and moving them, and then pinches them together. The phone triggers a click on the "Press and Speak" control in response to the pinching operation, and then receives the current voice. The user can start speaking to reply to the message. If no sound is detected for a specified period of time, it is determined that the user has completed the voice reply, and the received voice is automatically sent to the other party.

[0162] After the user completes the message reply, if they want to continue watching the video, they can perform an air gesture of flipping the palm. The mobile phone responds to this air gesture of flipping the palm and returns to the previous interface, that is, the video interface that the user was watching before replying to the voice message, as Figure 10C shown.

[0163] Subsequently, if a call comes in while the user is watching the video, it will jump to the interface as shown in Figure 10D shown. If the user needs to answer the call, they can perform an air gesture of keeping the palm in front of the screen and pressing forward. The mobile phone responds to this air gesture and answers the call. Subsequently, the user can answer the incoming call.

[0164] It can be understood that if the user does not want to answer the call at this time, they can click on controls such as hang up or text message through the air mouse.

[0165] In the above process, while the user is watching the video, they can reply to WeChat messages and answer calls through air gestures and air operations on the air mouse. Throughout the process, there is no need to touch the screen with the hand. It can be seen that the air operation in the embodiment of the present application can achieve global free control in the social communication scenario, and the implementation is convenient, without experience breakpoints, and can improve the user experience of air operation.

[0166] Scenario 3: The user takes a screenshot and shares it while swiping through Xiaohongshu. The interface changes in this scenario are as shown in Figures 11A - 11D shown.

[0167] While the user is swiping through Xiaohongshu, as shown in Figure 11A shown, if the user needs to swipe up the interface but it is inconvenient to touch the screen, they can perform an operation of pinching with two fingers. The mobile phone responds to the operation of pinching with two fingers and activates the air mouse mode. Then the user controls the interface to swipe up by keeping the two fingers open and swiping up. The user can precisely control the distance that the interface swipes up by the distance that the fingers slide, so as to browse the content of interest in the interface. For example, the user controls the interface to slide, so that part of the content of the comment area is displayed in the interface, as shown in Figure 11B shown. If the user wants to take a screenshot of the currently displayed content and share it with friends at this time, they can take a screenshot through the air gesture of taking a screenshot in the air, that is, when the mobile phone recognizes that the user's palm hovers in front of the screen, it further detects the user's air gesture. If it recognizes the gesture of grasping and making a fist, it takes a screenshot and displays the captured picture in the lower left corner of the screen. Then when the mobile phone recognizes that the user moves the cursor to the captured picture and performs an operation of pinching with two fingers, it displays the operation interface of the captured picture.

[0168] If the user clicks on the share control in this operation interface through the air mouse, then as shown in Figure 11CAs shown, icons of shareable applications are displayed, such as WeChat, Moments, Bluetooth, etc. If the user moves while keeping two fingers apart, moves the cursor to the WeChat icon, and performs the operation of pinching two fingers together, then in response to the operation of pinching two fingers together, a click on the WeChat icon is triggered, and then through the click response, it jumps to, for example, Figure 11D the interface shown.

[0169] In Figure 11D , the user can move while keeping two fingers apart, move the cursor to the avatar of the friend they want to share with, and perform the operation of pinching two fingers together. Then, the mobile phone responds to the operation of pinching two fingers together, triggers a click operation on the avatar, and shares the captured screen to the WeChat account corresponding to the avatar, thus completing the sharing of the captured screen.

[0170] It can be seen that in the above sharing scenario of social information, the user can complete the global sharing operation through air operation, without touching the screen, there is no experience breakpoint, and it can improve the user experience of using the air operation function.

[0171] In specific implementation, the present application also provides a computer storage medium, including a computer program. When the computer program runs on an electronic device, the electronic device is enabled to execute some or all of the steps in the above embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (abbreviation: ROM) or a random access memory (abbreviation: RAM), etc.

[0172] In specific implementation, the embodiments of the present application also provide a computer program product. The computer program product contains computer program code. When the computer program code runs on an electronic device, the electronic device is enabled to execute some or all of the steps in the above method embodiments.

[0173] The embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system. The programmable system includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device, and at least one output device.

[0174] Program code can be applied to the input instructions to perform the various functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

[0175] The program code can be implemented in a high-level procedural language or an object-oriented programming language to communicate with the processing system. When needed, the program code can also be implemented in assembly language or machine language. In fact, the mechanisms described in this application are not limited to the scope of any particular programming language. In any case, the language can be a compiled language or an interpreted language.

[0176] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions can be distributed via a network or via other computer-readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, floppy disks, optical disks, optical discs, Compact Disc Read Only Memories (CD-ROMs), magneto-optical disks, Read Only Memories (ROMs), Random Access Memories (RAM), Erasable Programmable Read Only Memories (EPROMs), Electrically Erasable Programmable Read Only Memories (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) in electrical, optical, acoustic, or other forms via the Internet. Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0177] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or ordering may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. Additionally, the inclusion of a structural or method feature in a particular figure does not imply that such a feature is required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0178] It should be noted that each unit / module mentioned in the device embodiments of the present application is a logical unit / module. Physically, a logical unit / module may be a physical unit / module, may be a part of a physical unit / module, or may also be implemented as a combination of multiple physical units / module. The physical implementation manner of these logical units / modules themselves is not the most important. The combination of the functions implemented by these logical units / modules is the key to solving the technical problems raised in the present application. In addition, in order to highlight the innovative part of the present application, the above device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems raised in the present application. This does not mean that there are no other units / modules in the above device embodiments.

[0179] It should be noted that in the examples and the specification of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0180] Although the present application has been illustrated and described by referring to certain preferred embodiments of the present application, those of ordinary skill in the art should understand that various changes may be made in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for remote operation, characterized in that, Applied to an electronic device, the method includes: In response to a first air operation by the user, start the air mouse mode and display a cursor on the display interface of the electronic device; In response to a second air operation by the user on the cursor, execute a first preset response event corresponding to the second air operation; In response to an air gesture operation by the user, execute a second preset response event corresponding to the air gesture operation.

2. The method according to claim 1, wherein The step of "In response to a second air operation by the user on the cursor, execute a first preset response event corresponding to the second air operation" includes: If the first preset response event is an event of moving the cursor, move the cursor according to the moving distance of the second air operation; If the first preset response event is a click event, trigger a click response at the position where the cursor is located; If the first preset response event is a first swipe event, perform a swipe operation on the object pointed to by the cursor.

3. The method according to claim 1, wherein The step of "In response to an air gesture operation by the user, execute a second preset response event corresponding to the air gesture operation" includes: If the second preset response event is a return event, return to the previous-level interface of the current display interface; If the second preset response event is a second swipe event, display the swiped interface.

4. The method according to any one of claims 1 to 3, characterized in that, Before the step of "In response to a first air operation by the user, start the air mouse mode", the method further includes: If the electronic device is in the screen-off state, in response to an air gesture operation by the user to turn on the screen, turn on the screen of the electronic device.

5. The method according to any one of claims 1 to 3, characterized in that, After the step of "In response to an air gesture operation by the user, execute a second preset response event corresponding to the air gesture operation", the method further includes: In response to not recognizing a second air operation by the user on the cursor within a preset duration, turn off the air mouse mode.

6. The method according to claim 1, characterized in that, The method further includes: Receive an operation by the user to initially enable the air mouse function, and enable the air mouse function; In response to an operation by the user on the air mouse for a previous preset number of times, display a notification card in the notification bar, where the notification card includes a first control for specifying an air gesture function; In response to a click operation by the user on an area of the notification card other than the first control, display a play machine skills introduction page for the air gesture function; In response to a click operation by the user on the first control, display a switch page for the specified air gesture function, where the switch page includes a switch control; In response to a click operation by the user on the switch control, enable the specified air gesture function.

7. The method according to claim 6, characterized in that After the step of "Receive an operation by the user to initially enable the air mouse function, and enable the air mouse function", the method further includes: Display a learning interface for air mouse operations, and display an operation prompt message for the air mouse on the learning interface, where the operation prompt message is used to prompt the user about the functions and usage methods of air operations; If it is recognized that the user performs an air operation according to the operation prompt message, play an animation effect of a response event corresponding to the operation prompt message on the learning interface; Display the next operation prompt message for the air mouse on the learning interface, and return the step of playing the animation effect of the response event corresponding to the operation prompt message on the learning interface if it is recognized that the user performs an air operation according to the operation prompt message, until it is determined that the user has completed the learning of the air mouse operation, and then exit the learning interface.

8. The method according to claim 1, wherein The first air operation is a two-finger pinching operation.

9. The method according to claim 2, wherein When the second air operation is an operation of moving while keeping the two fingers spread apart, the first preset response event is the event of moving the cursor; When the second air operation is a two-finger pinching operation, the first preset response event is the click event; When the second air operation is an operation of moving while keeping the two fingers pinched, the first preset response event is the first sliding event.

10. The method according to claim 3, wherein When the air gesture operation is a palm flipping operation, the second preset response event is the return event; When the air gesture operation is an operation of waving the wrist with the palm facing the screen and in a specified direction, the second preset response event is the second sliding event.

11. An electronic device, characterized in that, Comprising: One or more processors and a memory; The memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, Including a computer program, when the computer program runs on an electronic device, causing the electronic device to execute the method according to any one of claims 1 to 10.

Images