Control method and device

Abstract

The embodiment of the invention provides a control method and device, and relates to the technical field of terminals.The method comprises the steps that electronic equipment can enter an air mouse function when responding to a first gesture that a user does not touch a display screen, an air mouse cursor is displayed, and the function of moving the cursor to any position in the display screen is achieved based on a second gesture that the user does not touch the display screen. And the function of opening the application at any position of the display screen is realized based on the third gesture that the user does not touch the display screen, so that the user can realize fine control on any position in the electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a control method and device. Background Art

[0002] With the rapid development of smart electronic devices such as mobile phones and the advancement of the intelligentization of electronic products, mobile terminals with human-computer interaction functions are increasingly widely used in people's lives and production. Users are also increasingly demanding the convenience of human-computer interaction. For example, electronic devices can instruct devices to execute different instructions based on different touch operations of users on the display screen, such as opening an application in response to a user's click operation on an application, or displaying other interfaces on the desktop in response to a user's sliding operation on the desktop.

[0003] However, when it is inconvenient for the user to touch the mobile phone, it is difficult for the electronic device to realize the above-mentioned control of opening applications and sliding pages. Summary of the invention

[0004] The embodiments of the present application provide a control method and device, so that an electronic device can flexibly control any position in a trigger interface of the electronic device according to a user's air gesture.

[0005] In a first aspect, an embodiment of the present application provides a control method, comprising: an electronic device displays a first interface, the first interface including an icon of a first application; in response to a first gesture of a user not touching the display screen, the electronic device displays a first cursor at a first position of the first interface, the first position being located in an area other than the icon of the first application; in response to a second gesture of the user not touching the display screen, the electronic device displays the first cursor at a second position of the first interface, and after the second gesture in response to the user not touching the display screen, the first cursor is not displayed at the first position, and the second position is located in an area where the icon of the first application is located; in response to a third gesture of the user not touching the display screen, the electronic device displays a second interface, the second interface being any interface in the first application, and the second interface including the first cursor.

[0006] The first gesture may be the air mouse gesture described in the embodiment of the present application, the second gesture may be the move gesture described in the embodiment of the present application, and the third gesture may be the press gesture and lift gesture described in the embodiment of the present application. The first interface may be Figure 5 In the interface shown in a in FIG. 1 , the first cursor may be the empty mouse cursor described in the embodiment of the present application, and the second interface may be Figure 5 The first application may be a short video application.

[0007] The electronic device can enter the air mouse function and display the air mouse cursor in response to a first gesture in which the user does not touch the display screen, and implement the function of moving the cursor to any position on the display screen based on a second gesture in which the user does not touch the display screen, and then implement the function of opening an application at any position on the display screen based on a third gesture in which the user does not touch the display screen, so that the electronic device can achieve refined control of any position in the electronic device.

[0008] In one possible implementation, before responding to a first gesture in which the user does not touch the display screen, the method further includes: in response to a fourth gesture in which the user does not touch the display screen, the electronic device displays a first identifier in the first interface, the first identifier being used to indicate that the electronic device is about to enter a first function, and after the first gesture in response to the user not touching the display screen, the first identifier is not displayed in the first interface.

[0009] The fourth gesture may be the wake-up gesture described in the embodiment of the present application, the first function may be the air mouse function described in the embodiment of the present application, and the first identifier may be Figure 5 The mark 500 shown in b in FIG.

[0010] When the electronic device detects the wake-up gesture, it can prompt the user through the first identifier that the first function is about to be entered, so that the user can be aware of the entry of the first function in time, thereby improving the user's experience of using the first function.

[0011] In one possible implementation, the third gesture includes a fifth gesture and a sixth gesture, and in response to the third gesture in which the user does not touch the display screen, the electronic device displays a second interface, including: in response to the fifth gesture in which the user does not touch the display screen, the electronic device shrinks and displays the icon of the first application, and the icon of the first application is in a selected state; in response to the sixth gesture in which the user does not touch the display screen, the electronic device displays the second interface.

[0012] The fifth gesture may be the pressing gesture described in the embodiment of the present application, and the sixth gesture may be the lifting gesture described in the embodiment of the present application.

[0013] The electronic device can simulate the click operation of the mouse on the electronic device through the fifth gesture and the sixth gesture in the air, so as to trigger any display content in the electronic device to improve the user's experience of using the first function.

[0014] In one possible implementation, the second interface includes a first video and a first progress bar, and the method further includes: in response to the second gesture of the user not touching the display screen, the electronic device displays the first cursor at the third position of the first progress bar; in response to the fifth gesture of the user moving without touching the display screen, the electronic device displays the first cursor at the fourth position of the first progress bar, and after the fifth gesture of moving in response to the user not touching the display screen, the first cursor is not displayed at the third position; in response to the sixth gesture of the user not touching the display screen, the electronic device plays at the fourth position of the first video.

[0015] The electronic device can drag the progress bar by moving the fifth gesture, so that the user can control the video content remotely through simple gestures.

[0016] In one possible implementation, after the electronic device displays the first cursor at the second position of the first interface, the method further includes: in response to the fifth gesture of moving the first cursor without contacting the display screen, the electronic device displays the first cursor and the icon of the first application at the fifth position, and after the fifth gesture of moving the first cursor without contacting the display screen, the first cursor and the icon of the first application are not displayed at the second position; in response to the sixth gesture of moving the first cursor without contacting the display screen, the electronic device displays the first cursor and the icon of the first application at the sixth position, and the sixth position is a position in the first interface where application icons are allowed to be displayed.

[0017] The electronic device can drag the icon by moving the fifth gesture, so that the user can conveniently move the position of any icon in the interface, thereby improving the user's experience of using the first function.

[0018] In one possible implementation, after the first gesture in response to the user not touching the display screen, the method further includes: in response to the electronic device not detecting the gesture of the user not touching the display screen within a first threshold, the electronic device canceling display of the first cursor.

[0019] When the electronic device does not detect a gesture within a time threshold, the first cursor may be cancelled to exit the first function.

[0020] In a possible implementation, after the response to the electronic device not detecting a gesture of the user not touching the display screen, the method further includes: the electronic device releasing an input device; the electronic device canceling display of the first cursor, including: the electronic device canceling display of the first cursor based on the releasing of the input device.

[0021] The electronic device can achieve the purpose of canceling the first cursor by releasing the input device.

[0022] In a possible implementation, after the first gesture in response to the user not touching the display screen, the method further includes: the electronic device registering the input device; the electronic device displaying a first cursor at a first position of the first interface, including: the electronic device displaying the first cursor at a first position of the first interface based on registering the input device.

[0023] The electronic device may register an input device so that the electronic device can determine that a device is connected to the device in response to the registered input device, and then present the position where the input device moves through a first cursor.

[0024] In a possible implementation, after the second gesture in response to the user not touching the display screen, the method further includes: the electronic device obtaining a first input event, the first input event including: the coordinates of the second position; the electronic device displays the first cursor at the second position of the first interface, including: the electronic device displays the first cursor at the second position of the first interface based on the first input event transmitted by the input device.

[0025] The electronic device can recognize the user's air operation by transmitting the coordinate information in the input event, and move the first cursor to the position of the coordinates.

[0026] In a possible implementation, the electronic device includes: an air mouse service management module, a miscellaneous MISC device management module, and a system user interface system UI. After the second gesture in response to the user not touching the display screen, the method also includes: the air mouse service management module generates a first instruction, the first instruction includes: the second position, and the movement type corresponding to the second gesture, the air mouse service management module sends the first instruction to the MISC device management module; in response to the first instruction, the MISC device management module writes the first input event to the input device; the electronic device, based on the first input event transmitted by the input device, includes: the input device sends the first input event to the system UI.

[0027] The electronic device can create and transmit input events through the MISC device, so that the electronic device can perform corresponding operations in response to the input events.

[0028] In one possible implementation, before the electronic device displays the first interface, the method further includes: in response to the operation of starting the electronic device, the MISC device management module registers the MISC device; in response to the first instruction, including: when the MISC device management module determines to register the MISC device, the MISC device management module responds to the first instruction.

[0029] The electronic device can respond to the subsequent first instruction through the MISC device registered at startup, and the first instruction can be the ioctl instruction described in the embodiment of the present application.

[0030] In a possible implementation, after the fifth gesture in response to the user not touching the display screen, the method further includes: the electronic device obtaining a second input event, the second input event including: a parameter for indicating that the fifth gesture is a press-down type; the electronic device reducing the display of the icon of the first application, including: the electronic device reducing the display of the icon of the first application based on the second input event transmitted by the input device.

[0031] The electronic device may generate a corresponding second input event in response to the fifth gesture, and the second input event may implement a pressing operation on the electronic device, simulating pressing of the electronic device by a mouse.

[0032] In a possible implementation, after the sixth gesture in response to the user not touching the display screen, the method further includes: the electronic device obtaining a third input event, the third input event including: a parameter for indicating that the sixth gesture is a press up type; the electronic device displays the second interface, including: the electronic device displays the second interface based on the third input event transmitted by the input device.

[0033] The electronic device may generate a corresponding third input event in response to the fifth gesture, and the third input event may implement a lifting operation on the electronic device, simulating lifting of the electronic device by a mouse.

[0034] In one possible implementation, in response to a first gesture in which the user does not touch the display screen, the method further includes: the electronic device displays a third interface, the third interface including a button for setting an air gesture; in response to an operation on the button for setting the air gesture, the electronic device displays a fourth interface, the fourth interface including: a button for setting a first function; in response to the button for setting the first function, the electronic device displays a fifth interface, the fifth interface including a button for turning on the first function and prompt information for how to use the first function; in response to the button for turning on the first function, the electronic device turns on the first function.

[0035] The third interface can be Figure 4 The fourth interface can be Figure 4 The fifth interface can be Figure 4 The interface shown in c (or d).

[0036] The electronic device can provide a path to open the first function, so that the user can open the first function as needed and view how to use the first function, thereby improving the user's experience in subsequent use of the first function.

[0037] In one possible implementation, the first gesture is a pinch gesture, and / or the second gesture is a single-finger stretch gesture, and / or the fourth gesture is a two-finger stretch gesture, and / or the fifth gesture is a two-finger pinch gesture, and / or the sixth gesture is a two-finger open gesture.

[0038] Based on the above simple air gesture operation, the electronic device can achieve flexible control of any position in the electronic device.

[0039] In a second aspect, an embodiment of the present application provides a control device, which includes a display unit and a processing unit. The display unit is used to process the steps of displaying data in the control device, and the processing unit is used to process the steps of processing data in the control device.

[0040] In a possible implementation, the control device may further include a storage unit, which may include one or more memories, and the memory may be a device in one or more devices or circuits for storing programs or data.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory being used to store code instructions; the processor being used to run the code instructions so that the electronic device executes the method described in the first aspect or any one of the implementations of the first aspect.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the method described in the first aspect or any one of the implementations of the first aspect.

[0043] In a fifth aspect, a computer program product includes a computer program, and when the computer program is executed, the computer executes the method described in the first aspect or any one of the implementations of the first aspect.

[0044] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of an interface for opening the air mouse function provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of an interface using the air mouse function provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of another interface using the air mouse function provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of another interface using the air mouse function provided in an embodiment of the present application;

[0052] Figure 8 A schematic diagram of another interface using the air mouse function provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of module interaction of a control method provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of module interaction of another control method provided in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of the structure of a control device provided in an embodiment of the present application;

[0056] Fig.12 A schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to clearly describe the technical solution of the embodiment of the present application, the terms involved in the embodiment of the present application are explained:

[0058] 1. Air gestures

[0059] Air gestures can be understood as gestures that do not touch the display screen. Users can control electronic devices by making air gestures through the front camera of the electronic device.

[0060] Typically, electronic devices can support functions such as taking screenshots, sliding the screen, answering calls, or pausing audio and video playback through air gestures.

[0061] For example, in response to a user's grasping gesture without contacting the display screen, the electronic device can take a screenshot of the currently displayed interface; or, in response to a user's swinging up or swinging down gesture without contacting the display screen, the electronic device can turn the pages of the displayed content.

[0062] 2. Airmouse

[0063] Air mouse can also be called air mouse. The air mouse described in the embodiments of the present application can be understood as a user simulating a mouse connected to an electronic device to control an electronic device through air gestures without touching the display screen. In the specific implementation of the air mouse, the user can simulate the mouse connected to the electronic device to single-click, double-click, move, long press, etc. through different air gestures.

[0064] For example, users can use the air gestures supported by the air mouse function to click, move, and drag any icon in the selected interface, allowing users to flexibly control electronic devices even if they do not touch the display.

[0065] 3. Electronic equipment

[0066] Electronic devices may be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices may be mobile phones with touch screens, smart TVs, wearable devices, tablet computers (Pads), computers with wireless transceiver functions, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms used by the electronic devices.

[0067] 4. Other nouns

[0068] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first value and the second value are only used to distinguish different values, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0069] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0071] For example, Figure 1 A schematic diagram of a scenario provided in an embodiment of the present application. Figure 1 In the corresponding embodiments, an electronic device is taken as a mobile phone as an example for illustration, and this example does not constitute a limitation on the embodiments of the present application.

[0072] Electronic devices display Figure 1 The interface shown in a in the figure may be one of the interfaces in the desktop, and the interface may include at least one application icon. For example, the interface may display one or more of the following icons: an icon of a calendar card, an icon of a daily news card, an icon of a music application, an icon of a calendar application, an icon of a document management application, an icon of a short video application, or an icon of an application in a dock. The icons of applications in the dock may include an icon of a clock application, an icon of a computer application, an icon of a gallery application, an icon of a memo application, etc.

[0073] In response to the user clicking the icon of the short video application, the electronic device can display the following Figure 1 The interface shown in b in the figure can be any interface in the short video application. Figure 1 The interface shown in b in FIG. 1 can display video content and one or more of the following: a search box for searching videos, a video progress bar, a button for viewing the homepage, a button for displaying comments, a button for collecting videos, a button for forwarding videos, etc.

[0074] The user can view the comments corresponding to the video by triggering the button for displaying comments, or the user can also view the video by dragging the video progress bar.

[0075] It can be understood that, in response to the user's triggering operation on the display screen, the electronic device can achieve flexible control of the electronic device based on the user's triggering operation on the display screen.

[0076] However, when it is inconvenient for users to touch the display screen in scenarios such as cooking, it is difficult for users to control electronic devices.

[0077] In possible implementations, electronic devices may support certain functions based on certain specific air gestures, such as taking a screenshot of the display screen through a grabbing gesture, and turning pages through an up-and-down gesture.

[0078] The above-mentioned specific air gestures are for controlling the entire interface. Although they can realize certain control functions, they cannot enable users to trigger any position on the display screen, and their ability to fine-tune the control of electronic devices is relatively poor.

[0079] In view of this, an embodiment of the present application provides a control method, whereby the electronic device can enter the air mouse function and display the air mouse cursor in response to a first gesture in which the user does not touch the display screen, and implement the function of moving the cursor to any position on the display screen based on a second gesture in which the user does not touch the display screen, and then implement the function of opening an application at any position on the display screen based on a third gesture in which the user does not touch the display screen, so that the electronic device can achieve refined control of any position in the electronic device.

[0080] Among them, the first gesture can be the entering air mouse gesture described in the embodiment of the present application, the second gesture can be the moving gesture described in the embodiment of the present application, and the third gesture can include the pressing gesture and the lifting gesture described in the embodiment of the present application.

[0081] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiments of the present application is introduced below. Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0082] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0083] The sensor module 180 may include one or more of the following: a touch sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor, etc. ( Figure 2), which is not specifically limited in the embodiments of the present application.

[0084] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0085] The processor 110 may include one or more processing units, and the processor 110 may implement the steps of detecting a user's air gestures in the control method provided in an embodiment of the present application, and controlling the device to execute corresponding instructions based on the user's air gestures.

[0086] The USB interface 130 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0087] The charging management module 140 is used to receive charging input from a charger. The power management module 141 is used to connect the charging management module 140 and the processor 110 .

[0088] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0089] The electronic device realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor.

[0090] The display screen 194 is used to display images and videos, etc. For example, the display screen 194 can display the desktop interface, short video application interface, and air mouse setting interface provided in the embodiments of the present application.

[0091] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0092] Camera 193 is used to capture static images or videos. For example, camera 193 can obtain the user's air gestures.

[0093] The external memory interface 120 may be used to connect an external memory card.

[0094] The internal memory 121 may be used to store computer executable program codes, where the executable program codes include instructions.

[0095] The electronic device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0096] The touch sensor can be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, or a "touch screen". The touch sensor can detect a series of trigger operations for turning on the air mouse function.

[0097] Buttons 190 include a power button, a volume button, and the like.

[0098] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture, etc., which will not be elaborated here.

[0099] For example, Figure 3 The software structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in Figure 1, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0100] In some embodiments, the Android system is divided into multiple layers, including, from top to bottom, the application (APP) layer, the application framework (FWK) layer, the hardware abstraction layer (HAL), the kernel layer, and the sensor hub, etc., which is not limited in the embodiments of the present application.

[0101] The application layer may include a series of application packages. The application layer may include one or more of the following: a system user interface (system UI) and a setting application, which is not limited in the embodiments of the present application.

[0102] The systemUI is used to execute corresponding control instructions in response to input events, such as controlling the electronic device to display the air mouse cursor, move the air mouse cursor, cancel the display of the moving cursor, etc. based on the user's air gestures.

[0103] The Settings app is used to enable or disable the air mouse function.

[0104] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer.

[0105] The application framework layer includes some predefined interfaces. The application framework layer may include: input FWK.

[0106] The input FWK is used to report input events, for example, reporting received input events from the FWK to the systemUI.

[0107] In a possible implementation, the FWK may also include a display synthesizer, a window manager, a content provider, a resource manager, a view system, or a notification manager, etc. ( Figure 3 The display composition system may be used to receive graphics data buffers from multiple sources, perform layer composition on them, and then send them to the layer processing module in the HAL.

[0108] The purpose of the hardware abstraction layer is to abstract the hardware. It can provide a unified interface for querying hardware devices for upper-layer applications, or it can also provide data storage services for upper-layer applications.

[0109] The hardware abstraction layer may include: intelligent perception service module and intelligent secure execution environment (HISEE).

[0110] The intelligent perception service module is used to realize the processing of intelligent perception services such as air mouse service, air gesture service, code scanning service, staring at the screen service, face orientation service, etc. in electronic devices.

[0111] The smart perception service module may include one or more of the following: a business management module, a distribution control module, a smart perception client, or a smart perception hardware abstraction layer Android definition language (AIDL).

[0112] The service management module can be used to manage the air mouse service, air gesture service, code scanning service, staring at the screen service, and face orientation service in electronic devices.

[0113] The business management module may include an air mouse business management module, which can be used to transmit the received data to a miscellaneous (MISC) device management module, or to transmit the received data to the application layer via the smart perception AIDL.

[0114] Intelligent perception AIDL is used to pass the received data to the application layer.

[0115] In the process of the air mouse service management module managing the air mouse service, in response to the user's operation of starting the air mouse function, the air mouse service management module can subscribe to data obtained based on the air mouse service.

[0116] For example, in response to the operation of turning on the air mouse function, the service management module sends the subscription instruction to the air mouse engine in the smart perception engine through the distribution control module, the smart perception client, HISEE, the Qualcomm messaging interface (QMI), and the smart perception interface. In response to the gesture operation of the user without touching the display screen, the air mouse engine can return the recognition result of the user in the air mouse service to the air mouse service management module through the smart perception engine, the smart perception interface, QMI, HISEE, the smart perception client, and the distribution control module.

[0117] HISEE is used to transmit data received from the smart perception client to QMI, or to send messages based on QMI transmission to the smart perception client.

[0118] The distribution control module enables the subscription instructions or gesture recognition results to be sent from the smart perception client to the air mouse business management module.

[0119] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware to make it work. The kernel layer may include one or more of the following: QMI, MISC device management module, input operation module or input core module.

[0120] QMI can also be the abbreviation of Qualcomm Modem Interface or Qualcomm MSM Interface. QMI is a functional interface for multi-processor inter-process communication. It is an interface for inter-core communication between application processor (AP) and multimedia accelerator (CP). It transfers data through a shared memory driver (SMD). SMD implements physical operations of shared memory by operating smem and provides multiple data channels for message transmission.

[0121] The MISC device management module is used to register MISC devices and input devices. After the MISC device registration is completed, the driver in the MISC management module can respond to the received input / output control (ioctl) instructions.

[0122] The input handle module and the input core module can both be understood as modules for transmitting input events.

[0123] The sensor hub may include: AON firmware (fimeware, FW), and an intelligent perception algorithm platform.

[0124] AON FW is used to acquire image data and transmit the acquired image data to the empty mouse algorithm module in the intelligent perception computing module for recognition.

[0125] The intelligent perception algorithm platform may include: an intelligent perception interface, an intelligent perception computing module and an intelligent perception engine. The intelligent perception computing module may include: an air mouse algorithm module, and the intelligent perception engine may include an air mouse engine.

[0126] The intelligent perception interface is used to realize data interaction with the kernel layer.

[0127] The AirMouse algorithm module is used to perform image recognition based on the acquired image data and return the recognition results to the AirMouse engine.

[0128] The air mouse engine is used to send the received recognition results to the smart perception interface, or to receive data sent by the smart perception interface.

[0129] The sensor hub runs in a CPU core with weaker computing power, such as a core with weaker computing power, which can also be called a small core. In the embodiment of the present application, the air mouse recognition algorithm can be set in the sensor hub. Compared with setting the air mouse recognition algorithm on the AP side, the power consumption of the sensor hub is lower, which can reduce the overall power consumption of the control method provided in the embodiment of the present application.

[0130] In a possible implementation, the electronic device may include a hardware layer, for example, the hardware layer may include one or more of the following: a camera, a liquid crystal display (LCD), a graphics processing unit (GPU), or a central processing unit (CPU) and other hardware ( Figure 3 not shown).

[0131] In the embodiments of the present application, no specific limitation is imposed on the software layers involved in the software architecture, the modules contained in the layers, and the functions of the modules.

[0132] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0133] Before using the air mouse function in an electronic device, the user can Figure 4 Turn on the air mouse function in Settings - Smart Assistant - Air Gestures as described in the previous section, and learn how to use the air mouse function. Figure 5-Figure 8 The air gesture described in realizes the air mouse service.

[0134] Figure 4 A schematic diagram of an interface for opening the air mouse function provided in an embodiment of the present application.

[0135] In response to the user opening the smart assistant in the settings application, the electronic device displays the following Figure 4 The interface shown in a in the figure may be a smart assistant setting interface. The interface may display one or more of the following: a button for setting YOYO suggestions, a button for setting negative one screen display, a button for setting smart vision, a button for setting smart flow, a button for setting smart voice, a button 401 for setting air gestures, a button for setting eye tracking, etc.

[0136] In response to the user's triggering operation on button 401, the electronic device displays the following Figure 4 The interface shown in b in the figure may be an air gesture setting interface. The interface may display one or more of the following: a button 402 for setting an air mouse, a button for setting an air sliding screen, a button for setting an air screenshot, a button for setting an air return, etc.

[0137] It is understandable that the air mouse can realize the user's global operation of the electronic device, that is, the user can use the air mouse to trigger any position on the display screen. Air sliding screen, air screenshot and air return can be understood as a way to control electronic devices based on quick gestures.

[0138] In response to the user's triggering operation on button 402, the electronic device sequentially displays Figure 4 The interface shown in c and Figure 4 The interface shown in d in the figure shows the usage of the air mouse in a carousel. Figure 4 The interface shown in c in FIG. 4 may be an air mouse setting interface, which may display: an icon for indicating entering the air mouse function, text information corresponding to the icon, and a button 403 for turning on the air mouse function.

[0139] like Figure 4 The interface shown in c in the figure can display electronic devices and air gestures, and text information is used to prompt the user to start the air mouse. For example, the text information can be displayed as: Quick start, air operation at any time, naturally extend your thumb and index finger at a distance of 20-40 cm from the screen, wait for the cursor to appear on the screen, and the air mouse will start. Among them, the specific process of starting the air mouse can be seen in Figure 5 The corresponding embodiments.

[0140] like Figure 4 In the interface shown in d, the text information displayed in the interface can be used to prompt the user to move the cursor. The text information can be displayed as: the cursor moves freely, the cursor movement is controlled in the air, and the operation can be performed by lightly touching the thumb and index finger.

[0141] It is understood that the electronic device may take turns to display Figure 4 The interface shown in c and Figure 4 The interface shown in d allows users to fully learn how to trigger and use the air mouse function.

[0142] In response to user Figure 4 The trigger operation of the button 403 shown in c in FIG. 1 is performed, and the electronic device displays the following Figure 4 The interface shown in e in FIG. 4 may display a prompt box 404. The prompt box 404 is used to prompt the user to learn the gestures in the air mouse. For example, the prompt box 404 may display one or more of the following: please follow the gesture guide, a prompt message to learn how to operate, a no more prompt button, a start learning button, and a cancel button.

[0143] Further, in response to the user's triggering operation on the start learning button, the electronic device may display a gesture guidance interface ( Figure 4 (not shown), the user can learn the air mouse function based on the gesture guidance interface.

[0144] Based on this, electronic devices can not only be based on Figure 4 The corresponding embodiment enables the air mouse function, and can also realize the learning of the method of using the air mouse function to improve the user experience of using the air mouse function.

[0145] In electronic devices based Figure 4 In the corresponding embodiment, when the air mouse function is turned on, the electronic device can be based on Figure 5 The corresponding embodiment enters the air mouse function and then executes the air mouse service.

[0146] Figure 5 This is a schematic diagram of an interface using the air mouse function provided in an embodiment of the present application. The electronic device can be based on Figure 5 The corresponding embodiment uses the air mouse function to open any application in the interface.

[0147] Electronic devices display Figure 5 The interface shown in a in FIG. 5 may include an icon 501 of a short video application. The content displayed in the interface may be Figure 1 The interface shown in a is similar and will not be described here.

[0148] In response to a wake-up gesture by a user without touching the display screen, the electronic device displays a Figure 5 The interface shown in b in the figure may display a wake-up gesture mark 500, which is used to indicate that the electronic device is about to enter the air mouse function. The wake-up gesture can refer to the description in S904, which will not be repeated here.

[0149] In response to the user entering the empty mouse gesture without touching the display screen, the electronic device displays the following Figure 5 In the interface shown in c, the interface may not display the logo 500, and the interface may display the air mouse cursor (or simply referred to as cursor) 502. For entering the air mouse gesture, please refer to the description in S907, which will not be repeated here.

[0150] In a possible implementation, the gesture for entering the air mouse can be different from the wake-up gesture, that is, the user can enter the air mouse through two different gestures. Alternatively, the gesture for entering the air mouse can also be the same as the wake-up gesture, that is, the user can enter the air mouse when the continuous display time of a gesture reaches a certain time threshold.

[0151] It is understandable that when an electronic device displays an air mouse cursor, the user can implement air mouse services based on air gestures and perform refined control of the electronic device.

[0152] For example, in response to a user's moving gesture without contacting the display screen, the electronic device moves the position where the empty mouse cursor 502 is displayed. For example, the empty mouse cursor can be Figure 5 Move to the blank position in the interface shown in c Figure 5 The position of the icon 501 in the interface shown in d in FIG. The moving gesture can refer to the description in S1001, which will not be repeated here.

[0153] When the empty mouse cursor is located at the position of icon 501, in response to the user's pressing gesture without touching the display screen, the electronic device displays the following Figure 5 In the interface shown in e, the cursor 502 can continue to be displayed in the interface. In the interface, the icon 501 is in a selected state, and the icon 501 in the selected state can be displayed in a reduced size and / or in grayscale. The press gesture can refer to the description in S1006, which will not be repeated here.

[0154] In response to a user's lift gesture without contacting the display screen, the electronic device may display a Figure 5The interface shown in f in FIG. 5 may be any interface in a short video application, such as the interface in which video content may be displayed, the video content may be in a playing state, and the interface may continue to display a cursor 502. The lifting gesture may refer to the description in S1011.

[0155] It is understandable that the pressing gesture and the lifting gesture can simulate the mouse clicking operation on the display screen to achieve functions such as entering a certain page.

[0156] Based on this, the electronic device can quickly enter the air mouse function and display the air mouse cursor in the interface through the wake-up gesture and the air mouse gesture. In addition, the movement of the air mouse cursor can simulate the movement of the mouse on the electronic device to achieve access to any position on the interface, and then the user can determine the moving position through the continuously displayed air mouse cursor, so as to improve the user's experience of using the air mouse function.

[0157] In electronic devices based Figure 5 In the case where the interface shown in c in FIG. 1 shows an empty mouse cursor, the electronic device can control the interface switching in response to the user's mobile gesture without touching the display screen (see Figure 6 ), or the electronic device may move the selected content in response to a user moving the press gesture without contacting the display screen (see Figure 7 corresponding embodiments).

[0158] Figure 6 Another interface diagram of using the air mouse function provided in an embodiment of the present application, the electronic device can be based on Figure 6 In the corresponding embodiment, the air mouse function is used to control the interface switching.

[0159] Electronic devices display Figure 6 The interface shown in a in the figure can be Figure 5 The interface shown in c is similar and will not be repeated here.

[0160] In response to the user's moving gesture without contacting the display screen, the electronic device may move the position of the cursor 502. For example, as the moving gesture continues to move to the right, the electronic device may move the cursor 502 from Figure 6 The interface shown in a moves to Figure 6 On the right side of the interface shown in b.

[0161] like Figure 6 The interface shown in b in FIG. 1 may be another interface in the desktop, which may include one or more of the following: an icon of a health application, an icon of a weather application, an icon of a browser application, an icon of a settings application, etc.

[0162] It is understandable that the electronic device can switch the interface based on the user's movement gesture. For example, in response to the user's movement gesture of moving to the left without contacting the display screen, the electronic device can also display the negative one screen interface.

[0163] In a possible implementation, when the electronic device does not detect any gesture within the first threshold, the electronic device may display the following Figure 6 The interface shown in c in FIG. 1 does not include a cursor 502 .

[0164] In a possible implementation, the electronic device may also exit the air mouse function in response to other gesture operations by the user without touching the display screen. The gesture for exiting the air mouse function may be the same as or different from the gesture for entering the air mouse function, and the specific implementation of exiting the air mouse function is not limited in the embodiments of the present application.

[0165] Figure 7 A schematic diagram of another interface using the air mouse function provided in an embodiment of the present application, wherein the electronic device can be based on Figure 7 The corresponding embodiment is based on the application icon in the air mouse function moving interface.

[0166] Electronic devices display Figure 7 In the interface shown in a, the cursor in the interface is located in icon 501, and icon 501 is in a selected state. The content displayed in the interface can be seen in Figure 5 The interface shown in e will not be described here.

[0167] In response to the user moving the pressing gesture without touching the display screen (or understanding that the pressing gesture is maintained and the position of the pressing gesture is moved), the electronic device displays the following Figure 7 The interface shown in b.

[0168] exist Figure 7 The interface shown in a changes to Figure 7 During the process of the interface shown in b in FIG. 1 , icon 501 moves from position 701 to position 702, and an indicator frame 703 may be displayed around position 702, and indicator frame 703 is used to indicate the position where icon 501 moves when icon 501 is released. Since position 702 is not a position preset in the electronic device for displaying application icons, in response to the user's lift gesture without touching the display screen, the electronic device may move icon 501 to the indicator frame 703, and then display the following. Figure 7 The interface shown in c.

[0169] It is understood that electronic devices can be based on Figure 7 The corresponding embodiment realizes the refined control of moving the icon to the specified position, and the use scenario of the move and press gesture may not be limited to Figure 7 The scene described in .

[0170] In possible implementations, users can also achieve more precise control by moving and pressing gestures in game scenes, photo-editing scenes, etc., so as to improve the user experience of using the air mouse function.

[0171] In electronic devices based Figure 5 When the interface shown in f in FIG. 1 displays video content, the electronic device can control the video to fast forward in response to a press gesture without the user touching the display screen, or a move gesture + a move press gesture.

[0172] Figure 8 This is another interface diagram of using the air mouse function provided in an embodiment of the present application. When the electronic device opens a short video application, the electronic device can Figure 8 The corresponding embodiment uses the air mouse function to control the fast forward of the video in the short video application.

[0173] In one implementation, in response to a press gesture in which the user does not contact the display screen lasting longer than a second threshold, the electronic device fast-forwards the video.

[0174] Electronic devices display Figure 8 The interface shown in a in the figure can be found in Figure 5 In response to the user not touching the display screen for a pressing gesture that lasts for a second threshold, the electronic device can fast forward the video content and display the following: Figure 8 The interface shown in b in FIG. Figure 8 As shown in a in the figure, the position of the cursor can be a preset area in the short video application, such as the preset area can be the area above the user A's avatar, and the preset area is an area that allows fast-forwarding video content by long pressing.

[0175] like Figure 8 The interface shown in b in the figure may display a fast forward mark 801. The fast forward mark may be displayed as two triangles pointing to the right, and fast forward prompt information may be displayed around the fast forward mark, such as the fast forward prompt information may be 2X fast forward.

[0176] In response to the user's lift gesture, the electronic device can restore the video playback speed, that is, the video can be played normally at 1x speed.

[0177] In another implementation, in response to a move gesture by the user that does not touch the display screen, and a move and press gesture, the electronic device fast-forwards the video.

[0178] Electronic devices display Figure 8 In response to the user's moving gesture without touching the display screen, the electronic device displays the following Figure 8In the interface shown in d in the figure, the cursor can be located at the video progress bar. In response to the user's operation of moving and pressing the gesture without touching the display screen, the electronic device moves the video progress bar, and then displays the video content corresponding to the video progress bar after moving.

[0179] like Figure 8 In the interface shown in e in FIG. 8 , the cursor can be moved from position 801 of the video progress bar to position 802 of the video progress bar. The interface can also display prompt information corresponding to the video progress, and the prompt information corresponding to the video progress can be displayed as 01:30 / 03:00.

[0180] Furthermore, in response to a lifting gesture by the user without contacting the display screen, the electronic device can display the corresponding video content after the video progress bar is moved.

[0181] It is understandable that the electronic device can fast forward the video content to the user's desired position based on the user's press gesture, or multiple gestures such as move gesture + move press gesture, so as to increase the user's flexibility in controlling video playback based on the air mouse function.

[0182] In order to achieve Figure 5-Figure 8 The empty mouse function described in the present application embodiment is based on Fig. 9 The invention provides an example to illustrate the method of starting the air mouse function and controlling the electronic device based on the air mouse function.

[0183] Fig. 9 A schematic diagram of module interaction of a control method provided in an embodiment of the present application. Fig. 9 In a corresponding embodiment, the electronic device may include: an air mouse engine, a MISC device management module, an air mouse service management module, an input FWK, and a system UI.

[0184] like Fig. 9 As shown, the control method may include the following steps:

[0185] S901. In response to a device startup operation, the MISC device management module registers the MISC device.

[0186] In response to the device startup operation, the system notifies the kernel of the electronic device to initialize by calling module init(). For example, the MISC device management module in the kernel layer can be initialized in response to module init().

[0187] During the initialization of the MISC device management module, the MISC device management module registers the platform driver by calling platform_driver_register(), completes the driver registration by calling hiaon_airmouse_probe(), registers the MISC device by calling misc_regist(), and opens the air mouse device node dev / hiaon_airmouse.

[0188] When the node dev / hiaon_airmouse is enabled, the MISC device management module can register the input device based on the ioctl instruction received later, and then when the MISC device management module receives the message, it can index the input device under the MISC device to realize the process of transmitting input events and other data to other modules. If the node dev / hiaon_airmouse is not enabled, the MISC device management module cannot respond to the received ioctl instruction.

[0189] Exemplarily, the operation of starting the device may be a power-on operation or a restart operation, which is not limited in the embodiments of the present application.

[0190] S902: In response to the user's operation of enabling the air mouse function, the air mouse service management module sends a subscription instruction to the air mouse engine.

[0191] The subscription instruction may include parameters such as the identifier of the air mouse service.

[0192] For the operation of enabling the air mouse function, please refer to Figure 4 The corresponding embodiments.

[0193] For example, in response to the user's operation of turning on the air mouse function, the setting application can transmit the message used to turn on the air mouse function to the air mouse service management module in the HAL intelligent perception service module through the application layer. The air mouse service management module can generate a subscription instruction, and the subscription instruction is sent to the intelligent perception engine by the distribution control module, the intelligent perception client, HISEE, QMI, and the intelligent perception interface. The intelligent perception engine notifies the air mouse engine to subscribe to the air mouse service.

[0194] After the air mouse engine subscribes to the air mouse service, the air mouse engine is equivalent to establishing a path between the air mouse engine, AON FW, and the air mouse algorithm module. For example, when the air mouse engine confirms the subscription to the air mouse service, the air mouse engine can control AONFW to send the output image data to the air mouse algorithm module. After the air mouse algorithm module recognizes the gesture recognition result based on the image data, the recognized gesture recognition result can be sent to the air mouse engine, and the air mouse engine returns the gesture recognition result to the air mouse service management module through the smart perception interface, QMI, HISEE, smart perception client, and distribution control module.

[0195] S903. The airmouse engine returns a response message to the airmouse service management module.

[0196] The response message is used to instruct the air mouse engine to complete the subscription to the air mouse service. For example, the air mouse engine can return the response message from the smart perception interface, QMI, HISEE, smart perception client, and distribution control module to the air mouse service management module.

[0197] S904: In response to the user's wake-up gesture without touching the display screen, the air mouse engine may send the recognition result of the wake-up gesture to the air mouse service management module.

[0198] The wake-up gesture can be a two-finger extended gesture, such as a gesture in which the thumb and index finger are vertical and the other fingers are held, or a gesture in which both the thumb and index finger are pointing upward and the other fingers are held, etc. This is not limited in the embodiments of the present application.

[0199] For example, in response to a user's wake-up gesture without touching the display screen, the AON FW can send the collected image data of the wake-up gesture to the air mouse algorithm module. The air mouse algorithm module performs image recognition on the image data of the wake-up gesture and returns the recognition result of the wake-up gesture to the air mouse service management module via the air mouse engine, the smart perception interface, QMI, HISEE, the smart perception client, and the distribution control module.

[0200] S905. The air mouse service management module sends the recognition result of the wake-up gesture to the system UI.

[0201] The air mouse service management module can return the recognition result of the wake-up gesture to the systemUI of the application layer via the intelligent perception AIDL.

[0202] It can be understood that since the air mouse has not really entered at this time, or it can also be understood that the air mouse cursor has not been displayed in the display interface, the air mouse service management module can send the received data to the systemUI via AIDL and binder calls. When the electronic device enters the air mouse, or it can also be understood that the air mouse cursor is displayed in the interface, the air mouse service management module can send the received data to the systemUI via the MISC device management module and the related input module.

[0203] S906. The systemUI calls the display to display a wake-up gesture logo.

[0204] The wake-up gesture logo may display a reduced wake-up gesture, and the wake-up gesture logo is used to indicate that the electronic device is about to enter the air mouse function. For example, the wake-up gesture logo may be Figure 5 The mark 500 shown in b in FIG.

[0205] It is understandable that after the electronic device receives the wake-up gesture, it can enter the air mouse function based on further triggering by the user (see steps shown in S907-911).

[0206] S907. In response to the user entering air mouse gesture without touching the display screen, the air mouse engine may send a message indicating entering air mouse to the air mouse service management module.

[0207] The air mouse gesture (or pinch gesture) may be a light touch gesture between the thumb and index finger, or may be other gestures, which is not limited in the embodiments of the present application.

[0208] For example, in response to a user entering an air mouse gesture without touching the display screen, the AON FW can send the collected image data of the entering air mouse gesture to the air mouse algorithm module. The air mouse algorithm module performs image recognition on the image data of the entering air mouse gesture, and sends the recognition result of the entering air mouse gesture to the air mouse engine. The air mouse engine can generate a message enable_hiaon_airmouse() for indicating the entering air mouse when determining that the entering air mouse gesture is received, and then return the message for indicating the entering air mouse to the air mouse service management module via the smart perception interface, QMI, HISEE, smart perception client, and distribution control module.

[0209] S908. The air mouse service management module sends the first ioctl instruction to the MISC device management module.

[0210] The first ioctl instruction may be an enable instruction hiaon_airmouse_ioctl_enable(), and the first ioctl instruction is used to instruct the MISC device management module to register an input device.

[0211] In a possible implementation, the air mouse service management module may also pass the first ioctl instruction to the systemUI through the AIDL interface. In the process of passing the first ioctl to the systemUI, other modules in the software architecture may implement the process of converting the first ioctl instruction into an input event, and then uploading the input event to the systemUI, so that the systemUI displays the air mouse cursor. In this scenario, the electronic device may not execute the steps shown in S908-S911.

[0212] It can be understood that the ioctl instructions provided in the embodiments of the present application can all be passed to the systemUI through the AIDL interface, which will not be repeated later.

[0213] S909. The MISC device management module registers the input device.

[0214] In response to the first ioctl instruction, the MISC device management module instructs the internal driver module to register the input device of the air mouse service by calling the instruction hiaon_airmouse_input_register(). After registering the input device, the MISC device management module can respond to the ioctl instruction to realize the construction and transmission of input events in the air mouse service.

[0215] S910. MISC device management sends a message indicating completion of input device registration to systemUI.

[0216] For example, MISC device management may send a message indicating completion of input device registration to the system UI via the input core and input handle in the kernel, and the input FWK in the FWK.

[0217] S911, system UI cancels the display of the wake-up gesture icon and calls the display to display the empty mouse cursor.

[0218] The empty mouse cursor can be seen Figure 5 The cursor 502 in the interface shown in c.

[0219] It is understandable that when the systemUI displays an empty mouse cursor, the electronic device can implement different control strategies such as movement of the electronic device and click triggering based on other gestures of the user (see Fig.10 corresponding embodiments).

[0220] In a possible implementation, the electronic device may also cancel displaying the air mouse cursor when it determines that no gesture has been detected for a first threshold (or an exit air mouse gesture is detected), which can be understood as exiting the air mouse service (see steps S912-S916).

[0221] The first threshold may be a time length of 30 seconds or the like. The exit air mouse gesture may be any gesture different from the entry air mouse gesture, which is not limited in the embodiments of the present application.

[0222] S912: When the time period during which no user gesture is detected reaches a first threshold, the air mouse engine may send a message for instructing to exit the air mouse to the air mouse service management module.

[0223] For example, when the air mouse engine does not receive the gesture recognition result sent by the air mouse algorithm module within the first threshold, it can generate a message instructing the exit of the air mouse, and then return the message instructing the exit of the air mouse to the air mouse business management module via the smart perception interface, QMI, HISEE, smart perception client, and distribution control module.

[0224] S913. The air mouse service management module sends the second ioctl instruction to the MISC device management module.

[0225] The second ioctl instruction may be a disabling instruction hiaon_airmouse_ioctl_disable(), and the second ioctl instruction is used to instruct the MISC device management module to release the input device.

[0226] S914. The MISC device management module releases the input device.

[0227] In response to the second ioctl instruction, the MISC device management module instructs the internal driver module to release the input device under the air mouse service by calling the instruction hiaon_airmouse_input_free(). After releasing the input device, the MISC device management module will not respond to the ioctl instruction.

[0228] S915. The MISC device management module sends a message for instructing to release the input device to the systemUI.

[0229] For example, MISC device management may send a message for instructing to release the input device to the system UI via the input core and input handle in the kernel, and the input FWK in the FWK.

[0230] S916. SystemUI cancels the display of the empty mouse cursor.

[0231] Based on this, the electronic device can display the air mouse cursor in the interface based on the user's wake-up gesture and entry air mouse gesture, or it can exit the air mouse when it is determined that no gesture is detected, thereby achieving flexible entry and exit of the air mouse.

[0232] exist Fig. 9 In a corresponding embodiment, after S911 and before S912, in response to other gesture operations of the user without touching the display screen, the user can achieve refined control over the electronic device.

[0233] For example, in response to a moving gesture of a user without touching the display screen, the electronic device can control the movement of the air mouse cursor (see steps S1001-S1005); or, in response to a pressing gesture of a user without touching the display screen, the electronic device can select a certain content (see steps S1006-S1010); or, in response to a pressing and lifting gesture of a user without touching the display screen (see steps S1006-S1015), the electronic device can realize functions such as entering a certain page, which is not limited in the embodiments of the present application.

[0234] For example, Fig.10 A schematic diagram of module interaction of another control method provided in an embodiment of the present application, Fig.10 The module information contained in can be used with Fig. 9 The same is true in , so I will not repeat it here.

[0235] like Fig.10 As shown, the control method may include the following steps:

[0236] S1001. In response to a movement gesture of a user that does not touch the display screen, the air mouse engine may send a recognition result of the movement gesture to the air mouse service management module.

[0237] The moving gesture may be a gesture with a single finger pointing upwards and other fingers holding, or the moving gesture may be the same as the wake-up gesture, etc. This is not limited in the embodiments of the present application.

[0238] The recognition result of the moving gesture may include: the type of the moving gesture (type), such as type may be movement (motion), and the coordinates of the moving gesture, such as coordinates (coordinate) may be x, y, and x, y may be understood as the coordinates of the end position of the moving gesture.

[0239] It is understandable that the path for the air mouse engine to send the recognition result of the mobile gesture to the air mouse service management module can refer to the data transmission path described in S904, which will not be repeated here.

[0240] S1002. The air mouse service management module sends the third ioctl instruction to the MISC device management module.

[0241] In response to the recognition result of the moving gesture, the air mouse service management module generates a third ioctl instruction. The third ioctl instruction may be an instruction hiaon_airmouse_ioctl_event() for transmitting air mouse events. The third ioctl instruction may carry parameters such as type: motion and coordinate (x, y).

[0242] S1003. The MISC device management module generates a first input event.

[0243] In response to the third ioctl instruction, the MISC device management module writes a first input event into the input device according to the parameters carried in the third ioctl instruction. The first input event may carry: REL_X=x, REL_Y=y.

[0244] S1004. The MISC device management module sends the first input event to the system UI.

[0245] The path for the MISC device management module to send the first input event to the system UI can refer to the data transmission path described in S910, which will not be described in detail here.

[0246] S1005. systemUI controls the movement of the air mouse cursor.

[0247] The systemUI may move the air mouse cursor to the position indicated by the coordinates based on the coordinates in the first input event.

[0248] S1006. In response to a pressing gesture by the user without touching the display screen, the air mouse engine may send a recognition result of the pressing gesture to the air mouse service management module.

[0249] The pressing gesture may be the pinching gesture described in S907 .

[0250] The recognition result of the press gesture may include parameters such as type: action, and press (down) flag.

[0251] The path for the air mouse engine to send the recognition result of the press gesture to the air mouse service management module can refer to the data transmission path described in S904, which will not be repeated here.

[0252] S1007. The air mouse service management module sends the fourth ioctl instruction to the MISC device management module.

[0253] In response to the recognition result of the moving gesture, the air mouse service management module generates a fourth ioctl instruction. The fourth ioctl instruction may be an instruction hiaon_airmouse_ioctl_event() for transmitting an air mouse event. The fourth ioctl instruction may carry parameters such as type:action and a down flag.

[0254] S1008. The MISC device management module generates a second input event.

[0255] In response to the fourth ioctl instruction, the MISC device management module writes a second input event to the input device according to the parameters carried in the fourth ioctl instruction. The second input event may carry: BIN_LEFT=down.

[0256] S1009. The MISC device management module sends the second input event to the system UI.

[0257] The path for the MISC device management module to send the second input event to the system UI can refer to the data transmission path described in S910, which will not be described in detail here.

[0258] S1010, systemUI controls the empty mouse cursor to be pressed.

[0259] Pressing the empty mouse cursor is equivalent to the user's finger pressing the display screen, or the user controlling the mouse connected to the electronic device to press the display screen.

[0260] For example, combined with Figure 5 The corresponding embodiment illustrates the steps performed by the electronic device in response to the press gesture. Figure 5 In the interface shown in d in FIG. 1 , when the cursor 502 is located at the position of the icon 501, in response to the user's pressing gesture without touching the display screen, the electronic device can control the cursor 502 to press down, so that Figure 5 In the interface shown in e, icon 501 is in a selected state, which is equivalent to the user pressing icon 501 with his finger.

[0261] S1011. In response to a lift gesture by the user without touching the display screen, the air mouse engine may send a recognition result of the lift gesture to the air mouse service management module.

[0262] The lifting gesture may be the two-finger open gesture described in S907, that is, the thumb and the index finger may be kept at a distance and the other fingers may be held.

[0263] The recognition result of the raise gesture may include parameters such as type: action and an up flag.

[0264] The path for the air mouse engine to send the recognition result of the lift gesture to the air mouse service management module can refer to the data transmission path described in S904, which will not be repeated here.

[0265] It is understandable that the lift gesture is usually used together with the press gesture, that is, the user can simulate a single click of the mouse on the display screen by pressing the gesture and lifting the gesture. In a possible implementation, in response to the user's operation of pressing the gesture twice and lifting the gesture, a double click of the mouse on the display screen can also be simulated.

[0266] S1012. The air mouse service management module sends the fifth ioctl instruction to the MISC device management module.

[0267] In response to the recognition result of the moving gesture, the air mouse service management module generates a fifth ioctl instruction. The fifth ioctl instruction may be an instruction hiaon_airmouse_ioctl_event() for transmitting an air mouse event. The fifth ioctl instruction may carry parameters such as type:action and up flag.

[0268] S1013. The MISC device management module generates a third input event.

[0269] In response to the fifth ioctl instruction, the MISC device management module writes a third input event to the input device according to the parameters carried in the fifth ioctl instruction. The third input event may carry: BIN_LEFT=up.

[0270] S1014. The MISC device management module sends the third input event to the system UI.

[0271] The path for the MISC device management module to send the third input event to the system UI can refer to the data transmission path described in S910, which will not be described in detail here.

[0272] S1015. systemUI controls the empty mouse cursor to be raised.

[0273] Pressing the empty mouse cursor is equivalent to the user pressing the display screen with a finger and then lifting the finger, or the user controlling the mouse connected to the electronic device pressing the display screen and then releasing the mouse.

[0274] For example, combined with Figure 5 The corresponding embodiment illustrates the steps performed by the electronic device in response to the lift gesture. Figure 5In the interface shown in e in FIG. 1 , when the electronic device selects icon 501, in response to the user's lifting gesture without touching the display screen, the electronic device can open icon 501 and then display the following Figure 5 The interface shown in f is equivalent to the user clicking icon 501 or the mouse clicking icon 501.

[0275] Based on this, the electronic device can not only access any location in the interface based on a moving gesture, but also trigger any button in the interface based on a pressing gesture and a lifting gesture, thereby achieving refined control over the electronic device.

[0276] Combined with the above Figure 4-Figure 10 , the method provided in the embodiment of the present application is described, and the device for executing the above method provided in the embodiment of the present application is described below. Fig.11 As shown, Fig.11 A schematic diagram of the structure of a control device provided in an embodiment of the present application. The control device may be an electronic device in an embodiment of the present application, or a chip or chip system in an electronic device.

[0277] like Fig.11 As shown, the control device 1100 can be used in a communication device, a circuit, a hardware component or a chip, and the control device 1100 includes: a display unit 1101 and a processing unit 1102. The display unit 1101 is used to support the display step executed by the control method; the processing unit 1102 is used to support the control device 1100 to execute the information processing step.

[0278] In a possible implementation, the control device 1100 may further include a communication unit 1103, and the communication unit 1103 is used to support the control device 1100 to perform steps such as receiving or sending a message.

[0279] The control devices described in the embodiments of the present application may include Fig.11 The units described in the corresponding embodiments.

[0280] Specifically, the processing unit 1102 and the display unit 1101 may be integrated together, and the processing unit 1102 and the display unit 1101 may communicate with each other.

[0281] In a possible implementation, the control device 1100 may further include: a storage unit 1104. The storage unit 1104 may include one or more memories, and the memory may be a device in one or more devices or circuits for storing programs or data.

[0282] The storage unit 1104 may exist independently and be connected to the processing unit 1102 via a communication bus. The storage unit 1104 may also be integrated with the processing unit 1102.

[0283] Taking the control device 1100 as an example, which can be a chip or chip system of an electronic device in an embodiment of the present application, the storage unit 1104 can store computer-executable instructions of the method of the electronic device, so that the processing unit 1102 executes the method of the electronic device in the above embodiment. The storage unit 1104 can be a register, a cache, or a random access memory (RAM), etc. The storage unit 1104 can be integrated with the processing unit 1102. The storage unit 1104 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 1104 can be independent of the processing unit 1102.

[0284] In a possible implementation, the control device 1100 may further include: a communication unit 1103. The communication unit 1103 is used to support the control device 1100 to interact with other devices. Exemplarily, when the control device 1100 is an electronic device, the communication unit 1103 may be a communication interface or an interface circuit. When the control device 1100 is a chip or a chip system in an electronic device, the communication unit 1103 may be a communication interface. For example, the communication interface may be an input / output interface, a pin or a circuit, etc.

[0285] The device of this embodiment can be used to execute the steps executed in the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0286] Fig.12 A schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application.

[0287] The electronic device includes a processor 1201, a communication circuit 1204 and at least one communication interface ( Fig.12 The communication interface 1203 is used as an example for explanation).

[0288] The processor 1201 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0289] Communications link 1204 may include circuitry to transmit information between the above-described components.

[0290] The communication interface 1203 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0291] Possibly, the electronic device may further include a memory 1202 .

[0292] The memory 1202 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a communication line 1204. The memory may also be integrated with the processor.

[0293] The memory 1202 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1201. The processor 1201 is used to execute the computer-executable instructions stored in the memory 1202, thereby implementing the method provided by the embodiment of the present application.

[0294] Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not specifically limit this.

[0295] In a specific implementation, as an embodiment, the processor 1201 may include one or more CPUs, such as Fig.12 CPU0 and CPU1 in.

[0296] In a specific implementation, as an embodiment, the electronic device may include multiple processors, such as Fig.121 and 1205. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0297] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0298] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium can include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (digital versatiledisc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0299] The embodiments of the present application also provide a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0300] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0301] The above combinations should also be included in the scope of computer-readable media. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

[0302] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A control method, characterized in that: The method comprises: The electronic device displays a first interface, wherein the first interface includes an icon of a first application; In response to a first gesture of the user without touching the display screen, the electronic device displays a first cursor at a first position of the first interface, the first position being located in an area outside an icon of the first application; In response to a second gesture in which the user does not touch the display screen, the electronic device displays the first cursor at a second position of the first interface, and after the second gesture in response to the user not touching the display screen, the first cursor is not displayed at the first position, and the second position is located in an area where an icon of the first application is located; In response to a third gesture of the user without touching the display screen, the electronic device displays a second interface, where the second interface is any interface in the first application, and the second interface includes the first cursor.

2. The method according to claim 1, characterized in that In response to the first gesture of the user not touching the display screen, the method further includes: In response to the fourth gesture of the user without touching the display screen, the electronic device displays a first identifier in the first interface, and the first identifier is used to indicate that the electronic device is about to enter a first function. After the first gesture in response to the user without touching the display screen, the first identifier is not displayed in the first interface.

3. The method according to claim 1 or 2, characterized in that: The third gesture includes a fifth gesture and a sixth gesture, and in response to the third gesture in which the user does not touch the display screen, the electronic device displays a second interface, including: In response to the fifth gesture in which the user does not touch the display screen, the electronic device reduces and displays the icon of the first application, and the icon of the first application is in a selected state; In response to the sixth gesture in which the user does not touch the display screen, the electronic device displays the second interface.

4. The method according to claim 3, characterized in that The second interface includes a first video and a first progress bar, and the method further includes: In response to the second gesture of the user not touching the display screen, the electronic device displays the first cursor at a third position at the first progress bar; In response to the fifth gesture of the user moving without contacting the display screen, the electronic device displays the first cursor at the fourth position of the first progress bar, and after the fifth gesture of the user moving without contacting the display screen, the first cursor is not displayed at the third position; In response to the sixth gesture of the user not contacting the display screen, the electronic device plays at a fourth position of the first video.

5. The method according to claim 3, characterized in that: After the electronic device displays the first cursor at the second position of the first interface, the method further includes: In response to the fifth gesture of the user moving without contacting the display screen, the electronic device displays the first cursor and the icon of the first application at the fifth position, and after the fifth gesture of the user moving without contacting the display screen, the first cursor and the icon of the first application are not displayed at the second position; In response to the sixth gesture in which the user does not touch the display screen, the electronic device displays the first cursor and the icon of the first application at a sixth position, and the sixth position is a position in the first interface where application icons are allowed to be displayed.

6. The method according to any one of claims 3 to 5, characterized in that: After the first gesture in response to the user not contacting the display screen, the method further includes: In response to the electronic device not detecting a gesture of the user not contacting the display screen within a first threshold, the electronic device cancels display of the first cursor.

7. The method according to claim 6, characterized in that After the electronic device does not detect the user's gesture of not contacting the display screen in response to the electronic device, the method further includes: The electronic device releases the input device; The electronic device cancels display of the first cursor, comprising: the electronic device cancels display of the first cursor based on releasing the input device.

8. The method according to claim 7, characterized in that After the first gesture in response to the user not contacting the display screen, the method further includes: The electronic device registers the input device; The electronic device displays a first cursor at a first position of the first interface, comprising: the electronic device displays the first cursor at a first position of the first interface based on registering the input device.

9. The method according to claim 7 or 8, characterized in that: After the second gesture in response to the user not contacting the display screen, the method further includes: The electronic device acquires a first input event, where the first input event includes: coordinates of the second position; The electronic device displays the first cursor at a second position of the first interface, comprising: the electronic device displays the first cursor at the second position of the first interface based on the first input event transmitted by the input device.

10. The method according to claim 9, characterized in that The electronic device includes: an air mouse service management module, a miscellaneous MISC device management module, and a system user interface system UI. After the second gesture in response to the user not touching the display screen, the method further includes: The air mouse service management module generates a first instruction, wherein the first instruction includes: the second position, and the movement type corresponding to the second gesture, The air mouse service management module sends the first instruction to the MISC equipment management module; In response to the first instruction, the MISC device management module writes the first input event into the input device; The electronic device, based on the first input event transmitted by the input device, includes: the input device sending the first input event to the system UI.

11. The method according to claim 10, characterized in that Before the electronic device displays the first interface, the method further includes: In response to starting the operation of the electronic device, the MISC device management module registers the MISC device; Responding to the first instruction includes: when the MISC device management module determines to register the MISC device, the MISC device management module responds to the first instruction.

12. The method according to claim 7 or 8, characterized in that: After the fifth gesture in response to the user not contacting the display screen, the method further includes: The electronic device acquires a second input event, wherein the second input event includes: a parameter for indicating that the fifth gesture is a press-down type; The electronic device reducing and displaying the icon of the first application includes: the electronic device reducing and displaying the icon of the first application based on the second input event transmitted by the input device.

13. The method according to claim 7 or 8, characterized in that: After the sixth gesture in response to the user not contacting the display screen, the method further includes: The electronic device acquires a third input event, wherein the third input event includes: a parameter for indicating that the sixth gesture is a press-up type; The electronic device displays the second interface, including: the electronic device displays the second interface based on the third input event transmitted by the input device.

14. The method according to any one of claims 1 to 13, characterized in that: In response to the first gesture of the user not touching the display screen, the method further includes: The electronic device displays a third interface, wherein the third interface includes a button for setting an air gesture; In response to the operation on the button for setting the air gesture, the electronic device displays a fourth interface, wherein the fourth interface includes: a button for setting the first function; In response to the button for setting the first function, the electronic device displays a fifth interface, wherein the fifth interface includes a button for starting the first function and prompt information for prompting how to use the first function; In response to the button for starting the first function, the electronic device starts the first function.

15. The method according to any one of claims 3 to 14, characterized in that: The first gesture is a pinch gesture, and / or the second gesture is a single-finger stretch gesture, and / or the fourth gesture is a two-finger stretch gesture, and / or the fifth gesture is a two-finger pinch gesture, and / or the sixth gesture is a two-finger open gesture.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device executes the method according to any one of claims 1 to 15.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer is caused to perform the method according to any one of claims 1 to 15.

Images