An operation control method and an electronic device

Abstract

Embodiments of the present application apply to the technical field of terminal, and provide an operation control method and an electronic device. The electronic device acquires a first image frame. Then, in the case that the electronic device detects that the first image frame includes a hand, if an eye movement cursor exists in a display interface, the electronic device replaces a display of a finger mouse at a first position of the eye movement cursor, wherein the first position is used to represent a gaze point position of a user's eye on the display interface. Then, the electronic device can control the finger mouse to perform a target operation corresponding to a first preset gesture. In the present application, a hand-eye linkage scene can be implemented, and the execution accuracy of the electronic device is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an operation control method and electronic device. Background Technology

[0002] With the continuous development of electronic devices, they are equipped with more and more functions (such as air gestures and eye tracking) for users to use. Currently, eye tracking, as a non-contact human-computer interaction function, can achieve human-computer interaction by analyzing the position of the user's eye gaze point.

[0003] However, related eye-tracking technologies cannot accurately analyze the position of the user's eye gaze point during the use of electronic devices. This may lead to incorrect operation of the electronic device due to position tracking errors, affecting the user experience. Summary of the Invention

[0004] This application provides an operation control method and an electronic device for realizing hand-eye coordination scenarios and improving the execution accuracy of the electronic device.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, an operation control method is provided. In this method, an electronic device acquires a first image frame, which is the image frame currently captured by the electronic device. The electronic device then performs hand detection on the first image frame to obtain a hand detection result, which indicates whether the first image frame includes a hand. Subsequently, if the hand detection result indicates that the first image frame includes a hand, the electronic device determines whether an eye-tracking cursor exists on the display interface. If an eye-tracking cursor exists, the electronic device can replace the displayed finger mouse cursor with a cursor at a first position, where the first position represents the user's gaze point on the display interface. The electronic device then controls the finger mouse to perform a target operation corresponding to a first preset gesture, where the first preset gesture is obtained based on gesture recognition of a hand included in a target image frame. The target image frame includes the first image frame and / or a second image frame, where the second image frame is the image frame captured after the electronic device displays the finger mouse cursor.

[0007] In this application, after the user's hand is included in the first image frame captured by the electronic device, if an eye-tracking cursor is displayed on the electronic device's screen, the electronic device replaces the eye-tracking cursor with a finger mouse. In other words, the electronic device switches from eye-tracking control to gesture control, meaning it can perform corresponding operations based on the user's hand gestures. This enables hand-eye coordination, improving the execution accuracy of the electronic device and reducing errors caused by minimal changes in the user's eye movements. Furthermore, it simplifies the user's operation process; the electronic device only needs to capture an image frame containing the user's hand, eliminating the need for special gestures and reducing user time. This improves the efficiency of the electronic device and ultimately enhances the user experience.

[0008] In one possible implementation of the first aspect, the method further includes: if the hand detection result indicates that the first image frame does not include a hand, the electronic device continues to acquire the first image frame.

[0009] In this application, if the hand detection result indicates that the first image frame does not include a hand, that is, the hand detection result does not include the hand detection box, it means that the user is not likely to trigger the finger mouse service at present. Therefore, the electronic device can continue to collect the first image frame to further detect whether the user's hand is included in the next image frame. In this way, real-time detection of the user's hand can be achieved, reducing the occurrence of situations where the finger mouse service cannot be triggered in time due to the electronic device missing image frames, improving the execution efficiency of the electronic device, and thus improving the user experience.

[0010] In one possible implementation of the first aspect, the method further includes: if the display interface does not have an eye-tracking cursor, the electronic device can perform gesture recognition on the hand included in the first image frame to obtain a second target gesture. Then, if the second target gesture is a second preset gesture, the electronic device displays a finger mouse on the display interface at a preset position. Subsequently, the electronic device controls the finger mouse to perform the target operation corresponding to the second preset gesture.

[0011] In this application, if the display interface does not have an eye-tracking cursor, it means that the electronic device has not received a notification message or the user has not enabled the eye-tracking function. Therefore, the electronic device can add a finger mouse at a preset position on the display interface. This can clearly prompt the user that the finger mouse service has been enabled, avoid the phone from performing incorrect operations due to the user accidentally triggering the finger mouse service, and thus improve the accuracy of operation.

[0012] In one possible implementation of the first aspect, the method further includes: if the second target gesture is not a second preset gesture, the electronic device continues to acquire the first image frame.

[0013] In this application, if the second target gesture is not the second preset gesture, it means that the user does not intend to perform a gesture control operation. Therefore, the electronic device can continue to collect the first image frame to further detect whether the user's hand is included in the next image frame. In this way, real-time detection of the user's hand can be achieved, reducing the occurrence of situations where the finger mouse service cannot be triggered in time due to the electronic device missing image frames, improving the execution efficiency of the electronic device, and thus improving the user experience.

[0014] In one possible implementation of the first aspect, the method further includes: upon receiving a notification message, the electronic device acquires a third image frame. Then, the electronic device performs eye-tracking recognition on the third image frame to obtain eye-tracking recognition data; wherein the eye-tracking recognition data includes the coordinates of the user's gaze point. Then, the electronic device displays an eye-tracking cursor on a display interface according to the gaze point coordinates.

[0015] In this application, after receiving a notification message, the electronic device can display an eye-tracking cursor on the display interface based on the coordinates of the gaze point corresponding to the user's gaze point in the third image frame. In other words, the display interface will display the eye-tracking cursor at the same time as displaying the message notification, so as to enable the eye-tracking function. In this way, the user can interact with the electronic device without contact, thus improving the convenience of human-computer interaction.

[0016] In one possible implementation of the first aspect, the process of performing eye-tracking recognition on the third image frame may specifically include: the electronic device performing face detection on the third image frame according to a face detection model to obtain a face detection result, wherein the face detection result is used to indicate whether the third image frame includes a face. Then, if the face detection result indicates that the third image frame includes a face, the electronic device may perform eye-tracking recognition on the third image frame according to an eye-tracking recognition algorithm to obtain eye-tracking recognition data.

[0017] In this application, since the eye-tracking recognition algorithm has higher detection accuracy than the face detection model, it requires more computing power. Therefore, in order to reduce unnecessary resource waste, the electronic device can perform face detection on the third image frame before performing eye-tracking recognition on the third image frame. This can avoid wasting computing resources.

[0018] In one possible implementation of the first aspect, the method further includes: when the face detection result indicates that the third image frame does not include a face, the electronic device does not perform eye-tracking recognition on the third image frame.

[0019] In this application, if the face detection result indicates that the third image frame does not contain a face, it means that the user is not looking at the electronic device and the possibility that the user is looking at the notification message is small. Therefore, the electronic device does not need to perform eye-tracking recognition on the third image frame. This can reduce unnecessary waste of resources and improve the utilization of computing resources.

[0020] In one possible implementation of the first aspect, the process of controlling the finger mouse to perform a target operation may specifically include: the electronic device performing gesture recognition on the hand included in the first image frame to obtain a third target gesture. Then, if the third target gesture is a first preset gesture, the electronic device controls the finger mouse to perform the target operation corresponding to the first preset gesture.

[0021] In this application, after replacing the eye-tracking cursor with a finger cursor, the electronic device can directly perform gesture recognition on the hand in the first image frame to obtain a third target gesture. If the third target gesture is a first preset gesture, it means that the user wants to perform the target operation corresponding to the third target gesture. Therefore, the electronic device can directly control the finger mouse to perform the target operation corresponding to the first preset gesture. In this way, the execution efficiency of the mobile phone can be improved.

[0022] In one possible implementation of the first aspect, the process of controlling the finger mouse to perform a target operation may specifically include: the electronic device acquiring a second image frame. Then, if the electronic device detects that the second image frame includes a hand, it performs gesture recognition on the hand included in the second image frame to obtain a first target gesture. Then, if the first target gesture is a first preset gesture, the electronic device controls the finger mouse to perform the target operation corresponding to the first preset gesture.

[0023] In this application, if the gesture included in the second image frame is a first preset gesture, the electronic device can control the finger mouse to execute the target operation corresponding to the first preset gesture. Thus, eye-tracking cursor movement can be achieved through eye gaze, and corresponding operations can be executed through hand gestures, providing convenience and improving the user experience. Furthermore, after determining that the first target gesture is the first preset gesture, the electronic device promptly executes the control operation corresponding to the first target gesture, improving the execution accuracy of the mobile phone.

[0024] In one possible implementation of the first aspect, the method further includes: when the second image frame does not include a hand, or when the first target gesture is not a first preset gesture, the electronic device does not perform gesture control operations.

[0025] In this application, if the second image frame does not include a hand, or if the gesture included in the second image frame is not the first preset gesture, the electronic device may not perform gesture control operation. In this way, the occurrence of erroneous operation by the electronic device can be reduced, the execution accuracy of the electronic device can be improved, and the user experience can be enhanced.

[0026] In one possible implementation of the first aspect, the method further includes: the electronic device continuously acquiring a preset number of first image frames. Then, the electronic device can perform hand detection on each of the first image frames to determine whether the user's hand is included in each first image frame. Then, if the user's hand is included in all of the preset number of first image frames, the electronic device determines whether an eye-tracking cursor exists on the display interface. Then, if an eye-tracking cursor exists on the display interface, the eye-tracking cursor is replaced with a finger mouse.

[0027] In this application, if the user's hand is included in all of the preset number of first image frames, it means that the user wants to trigger the finger mouse service. Therefore, the mobile phone can further determine whether there is an eye-tracking cursor on the display interface. In this way, the occurrence of incorrect finger mouse display caused by the hand accidentally appearing in the shooting range of the front camera can be reduced, thereby improving the display accuracy of the finger mouse.

[0028] Secondly, this application provides an electronic device, which includes a camera, a memory, and one or more processors; the camera, the memory, and the processor are coupled; the camera is used to acquire images, the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method described above.

[0029] Thirdly, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0030] Fourthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0031] Fifthly, a chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to execute the method described above.

[0032] The beneficial effects that the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect can achieve can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, and will not be repeated here. Attached Figure Description

[0033] Figure 1 A schematic diagram of an interface for displaying the pop-up click process provided in an embodiment of this application;

[0034] Figure 2 A schematic diagram illustrating a scenario where a user's eyes are focused on a screen, provided as an embodiment of this application;

[0035] Figure 3 A schematic diagram of a cursor movement interface provided for an embodiment of this application;

[0036] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram illustrating a front-facing camera of a mobile phone and the camera itself, provided as an embodiment of this application.

[0038] Figure 6 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0039] Figure 7 A flowchart of an operation control method provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram illustrating the settings of a finger mouse function provided in an embodiment of this application;

[0041] Figure 9 A schematic diagram illustrating a mobile phone performing hand detection, provided as an embodiment of this application;

[0042] Figure 10 A flowchart illustrating another operation control method provided in this application embodiment;

[0043] Figure 11 A schematic diagram of a mobile phone receiving notification messages provided in an embodiment of this application;

[0044] Figure 12 This is a schematic diagram illustrating the setup of an eye-tracking function provided in an embodiment of this application;

[0045] Figure 13 A schematic diagram illustrating eye-tracking recognition on a mobile phone, provided as an embodiment of this application;

[0046] Figure 14This is a schematic diagram illustrating face detection on a mobile phone, as provided in an embodiment of this application.

[0047] Figure 15 This application provides a schematic diagram of a mobile phone interface for displaying an eye-tracking cursor.

[0048] Figure 16 This application provides an example of an interface diagram for enhancing the text of controls.

[0049] Figure 17 A schematic diagram of an interface that replaces an eye-tracking cursor with a finger mouse, provided for an embodiment of this application;

[0050] Figure 18 A schematic diagram illustrating a scenario for triggering a click operation, provided as an embodiment of this application;

[0051] Figure 19 A schematic diagram illustrating a mobile phone entering a reply interface, provided as an embodiment of this application;

[0052] Figure 20 A flowchart illustrating an operation control process provided in an embodiment of this application;

[0053] Figure 21 This is a schematic diagram of a mouse interface for moving a finger, provided as an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0055] In some embodiments, when a user uses an electronic device and the device receives a notification message (such as an SMS), the device typically displays the notification message as a pop-up at the top of the screen to remind the user to check it. If the user wants to view or reply to the notification message, they can manually click the pop-up to access the detailed interface. However, this process is relatively complex, wastes the user's time, and results in a poor user experience. Furthermore, due to the limited display area of ​​the pop-up, if the notification message is long, the limited display area may prevent the complete message from being shown, requiring the user to click to view the specific message content, making the process cumbersome.

[0056] For example, such as Figure 1 As shown, if a phone receives an SMS notification while a short video is being displayed on the screen, the notification message can be displayed as a pop-up at the top of the screen. This is understandable. Figure 1In interface (a), area T is a pop-up window. Pop-up window T displays the SMS content, contact, SMS receipt time, reply control, and "mark as read" control. Subsequently, if the user clicks any location within pop-up window T other than the reply control and the "mark as read" control, it means the user wants to view the notification message. Therefore, the phone can display the detailed interface of the notification message, which is... Figure 1 The (b) interface allows the user to view detailed content. If the user clicks the reply control, it means the user wants to reply to the notification message. Therefore, the phone can display the reply interface for the notification message so that the user can reply to the SMS message. If the user clicks the mark-as-read control, it means the user does not want to view the notification message. Therefore, the phone can cancel the display of the pop-up T so that the phone screen can continue to display the short video content.

[0057] In one implementation, to simplify user operations and enhance the user experience, electronic devices can employ eye-tracking technology. By analyzing changes in the user's eyes, this allows users to control the device without touching the screen. It's understood that when a user's eyes look at different areas or specific locations on the screen, subtle changes occur in their eye movements. Therefore, the electronic device can extract these eye features and track these changes in real time to predict the user's browsing needs. Based on these browsing needs, the electronic device can then perform corresponding actions, thereby achieving the goal of controlling the device through the user's eyes.

[0058] For example, such as Figure 2 As shown, user Y is browsing e-book content on the screen of phone S, and user Y's gaze is located in region Z. If user Y wants to browse the e-book content at the top of the screen, user Y can adjust their gaze (e.g., move their gaze upwards) to bring their gaze to region Z1. The phone can then predict user browsing needs by tracking changes in the user's eyes.

[0059] In some embodiments, when an electronic device receives a notification message, it can capture a current image frame using its front-facing camera and identify that frame to determine the user's gaze point. If the electronic device then detects that the user's gaze point is within the pop-up window area, it indicates that the user wants to view the notification message. Therefore, the electronic device can automatically expand the notification message for the user to view, reducing the user's operational steps and improving the user experience.

[0060] For example, such as Figure 3As shown, when a mobile phone receives a text message, it can display the message content in a pop-up window, specifically in pop-up window T on interface (a) (e.g., "When are you free tomorrow? Want to go shopping...?"). A cursor is displayed at position Q, indicating the user's current gaze. If the cursor moves from position Q to position Q1, it means the user wants to view the message content. Therefore, the phone can further display the detailed message content, specifically in pop-up window T1 on interface (b) (e.g., "When are you free tomorrow? Want to go shopping and eat out!"). This allows the user to read the complete message content, improving the phone's operational efficiency.

[0061] However, because the changes in a user's eyes are subtle when browsing the display interface of an electronic device, it is difficult for the device to accurately capture these changes. Consequently, the accuracy of eye-tracking operations performed by electronic devices is relatively low. If an electronic device cannot accurately detect changes in a user's eyes, it cannot predict the user's operational needs in a timely manner, thus affecting the user experience.

[0062] Therefore, to improve the execution accuracy of electronic devices, this application provides an operation control method. In this method, the electronic device acquires a first image frame, which is the image frame currently captured by the electronic device. Then, if the electronic device detects that the first image frame includes a user's hand, and if an eye-tracking cursor exists on the display interface, the electronic device replaces the eye-tracking cursor with a finger mouse. Subsequently, the electronic device controls the finger mouse to execute the target operation corresponding to the user's hand.

[0063] In this embodiment, if the user's hand is included in the image frame captured by the electronic device, and an eye-tracking cursor is displayed on the electronic device's screen, the electronic device replaces the eye-tracking cursor with a finger mouse. In other words, the electronic device switches from eye-tracking control to gesture control, meaning it can perform corresponding operations based on the user's hand gestures. This enables hand-eye coordination, improving the execution accuracy of the electronic device and reducing errors caused by minimal changes in the user's eye movements. It also simplifies the user's operation process; the electronic device only needs to capture an image frame containing the user's hand, eliminating the need for special gestures and reducing user time. This improves the efficiency of the electronic device and ultimately enhances the user experience.

[0064] In some examples, the electronic device in this application embodiment may be a mobile phone, tablet computer, smartwatch, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., which includes a camera. This application embodiment does not impose any special restrictions on the specific form of the electronic device.

[0065] For example, Figure 4 A schematic diagram of the structure of electronic device 200 is shown. For example... Figure 4 As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0066] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0067] Processor 210 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0068] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0069] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0070] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0072] The charging management module 211 receives charging input from the charger. While charging the battery 213, the charging management module 211 can also supply power to the electronic device through the power management module 212.

[0073] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 240, wireless communication module 250, modem processor, and baseband processor.

[0074] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0075] The mobile communication module 240 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 200. The modem processor may include a modulator and a demodulator.

[0076] The wireless communication module 250 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0077] Electronic device 200 implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0078] The display screen (or screen) 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0079] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0080] The ISP (Image Signal Processor) is used to process data fed back from the camera 293. For example, when an electronic device takes a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (or image sensor). The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 293. In some embodiments, the camera 293 includes a shutter. The shutter is a device in the camera used to control the duration of light exposure to the photosensitive element.

[0081] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0082] In some embodiments, camera 293 may include a lens, which is an optical component for generating images.

[0083] For example, the aforementioned N cameras 293 may include one or more front-facing cameras and one or more rear-facing cameras. See, for example, [link to relevant documentation]. Figure 5 Taking the aforementioned electronic device 200 as an example, which is a mobile phone. Figure 5 The interface (a) shows a front-facing camera, such as front-facing camera 20. Figure 5 The interface in (b) shows three rear cameras, such as rear cameras 21, 22, and 23. Of course, the number of cameras in the above-mentioned mobile phone includes, but is not limited to, the number described in the above embodiments.

[0084] Among them, the above N cameras 293 may include one or more of the following cameras: main camera, telephoto camera, wide-angle camera, ultra-wide-angle camera, macro camera, fisheye camera, infrared camera, depth camera and monochrome camera.

[0085] In this embodiment, the aforementioned front-facing camera is a camera with an always-on camera (AON) function. Specifically, when the electronic device uses the AON function, the front-facing camera of the electronic device is always on, and can capture images in real time. The electronic device performs gesture recognition through image analysis and can respond to user gestures to control the screen, enabling the user to control the screen of the electronic device without touching it.

[0086] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 200 selects a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0087] The external storage interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0088] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of electronic device 200 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0089] Electronic device 200 can implement audio functions through audio module 270 and application processor, such as music playback and recording. The audio module 270 may include a speaker, receiver, microphone, and headphone jack.

[0090] Buttons 290 include a power button, volume buttons, etc. Indicator 292 may be an indicator light.

[0091] The sensor module 280 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0092] The gyroscope sensor can be used to determine the motion attitude of the electronic device 200. In some embodiments, the angular velocity of the electronic device 200 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor.

[0093] An accelerometer can detect the magnitude of acceleration of an electronic device 200 in various directions (typically three axes). When the electronic device 200 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device, and is applied to applications such as screen orientation switching and pedometers.

[0094] The software system of the aforementioned electronic device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 200.

[0095] Figure 6This is a software structure block diagram of an electronic device 200 according to an embodiment of this application. The solutions of each embodiment of this application will be discussed based on the following technical architecture. For ease of explanation, only schematic block diagrams are used to illustrate business logic relationships, without strictly expressing the specific location of each business within the technical architecture. Furthermore, the naming of each module in the software architecture diagram is illustrative; this application does not limit the naming of each module in the software architecture diagram. In actual implementation, the specific naming of the modules can be determined according to actual needs.

[0096] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system includes four layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), and kernel.

[0097] The application layer may include a series of application packages. For example, the application layer may include applications such as camera and smart sensing (applications can be simply referred to as apps), and this application embodiment does not impose any limitations on this. As another example, the application layer may also include a system user interface (UI), which can provide a basic display interface for the electronic device, such as a status bar at the top of the screen, a navigation bar at the bottom of the screen, a quick settings bar in a pull-down interface, a notification bar, a lock screen interface, a volume adjustment dialog box, and a screenshot display interface, etc.

[0098] The intelligent sensing application provided in this application supports various services. For example, the services supported by the intelligent sensing application may include finger mouse service and eye tracking service, as well as intelligent barcode scanning service, gaze-based screen-on service, intelligent always-on display service, etc. These services can be collectively referred to as intelligent sensing services.

[0099] Optionally, the implementation of these services supported by the smart sensing application relies on the AON camera (front-facing camera) of the electronic device being always on to collect images in real time and obtain data related to the smart sensing service. When the smart sensing application detects service-related data, it sends the data to the smart sensing algorithm platform. The platform analyzes the collected images and determines whether to execute the relevant service of the smart sensing application, or which specific service to execute, based on the analysis results.

[0100] The aforementioned air gesture service refers to a service where electronic devices recognize user gestures and respond according to preset strategies corresponding to those gestures, thus enabling human-computer interaction. This user gesture is understood to be an action performed by the user's hand within a preset distance of the electronic device; for example, this preset distance could be 20 centimeters. When the electronic device is in screen-on mode, it can support the recognition of various preset air gestures. Different gestures can correspond to different preset strategies, such as: an air OK gesture → finger mouse, an air up / down gesture → page turning.

[0101] In one example, taking the air OK gesture as an example, the user changes from an extended palm to an OK gesture, or from a clenched fist to an OK gesture. The default operation for this gesture is to display a finger mouse on the electronic device's display screen. When the electronic device's screen is on, the electronic device automatically displays the finger mouse when it detects the air OK gesture through the AON camera (front-facing camera), and performs the corresponding operation based on the user's subsequent gestures. In this way, human-computer interaction can be completed without the user touching the electronic device. The air OK gesture can also be called an air finger mouse gesture. It can be understood that in actual implementation, the air finger mouse gesture can also be other gestures, such as the V gesture, a pinch gesture, etc.

[0102] In another example, taking the air gestures up / down, when the finger changes from a closed, extended state to a downward-curved state (upward swipe gesture), the preset control method is to flip up a page or swipe the screen upward. Similarly, when the finger changes from a closed, curved state to an extended state (downward swipe gesture), the preset control method is to flip down a page or swipe the screen downward. When the electronic device is in screen-on mode, it automatically performs an upward page flip when it detects an upward swipe gesture via the AON camera (front-facing camera), and automatically performs a downward page flip when it detects a downward swipe gesture, completing human-computer interaction without the user touching the electronic device. The air gestures up / down can also be called air swipe gestures. Taking the upward or downward swipe gesture as an example, it can be understood that in actual implementation, air swipe gestures can also be left or right swipe gestures.

[0103] Furthermore, in this embodiment of the application, the air gesture service also supports user-defined operations, such as the user canceling "air press gesture → answer a call" and resetting it to "air press gesture → jump to the payment code interface".

[0104] For example, when an electronic device is on and displaying its home screen, a user can trigger a predefined quick service via an air-press gesture, such as quickly jumping from the home screen to the payment code interface. In this scenario, the air-press gesture can quickly bring up the payment interface, so this scenario can be called a smart payment scenario. For example, the user can also reset the gesture to "air-press gesture → jump to the scan interface" or "air-press gesture → jump to the transit code interface," etc.

[0105] The aforementioned eye-tracking service refers to an electronic device identifying changes in a user's eyes, determining the user's gaze point based on these changes, and responding according to a preset strategy based on the location and duration of the gaze point, thus achieving human-computer interaction. For example, when the electronic device is in a screen-on state, if a notification message is received, the electronic device can display an eye-tracking cursor on the current interface based on the user's gaze point. Subsequently, if the eye-tracking cursor is located in the area where the notification message is located, and the gaze duration reaches a preset duration, the electronic device can display the complete content of the notification message.

[0106] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 6 As shown, the application framework layer can include AON service, camera service, intelligent sensing service, input framework, etc.

[0107] The AON service is used for real-time image acquisition. The camera service is used for image acquisition; specifically, when the camera application on an electronic device is enabled, the device's camera can perform corresponding operations based on the user's touch input. For example, if the user clicks the shooting control in the camera application, the device's camera can perform a shooting operation to acquire an image.

[0108] The intelligent sensing service processes eye-tracking and recognition requests sent by applications and returns the processing results to the applications to achieve eye-tracking and recognition functionality. Simultaneously, the intelligent sensing service also processes gesture recognition requests sent by applications and returns the processing results to the applications to achieve gesture recognition functionality.

[0109] The Input Framework is a GUI toolkit designed for Java that can include graphical user interface devices such as text boxes, buttons, dividers, and tables. It can be responsible for registering smart sensing service fences, invoking smart sensing services when there are notification messages, and processing registration results accordingly.

[0110] The Hardware Abstraction Layer (HAL) is a wrapper around Linux kernel drivers, providing interfaces to higher-level systems. It hides the hardware interface details of a specific platform, thus providing a virtual hardware platform for the operating system. In this embodiment, the HAL includes the Camera HAL, the Smart Sensing HAL, and the Android Interface Definition Language (AIDL).

[0111] The Camera HAL is the core software framework of the Camera, which can include sensor nodes and image front end (IFE) nodes. Sensor nodes and IFE nodes are components (nodes) in the image data and control command transmission path (also known as the transmission pipeline) created by the Camera HAL.

[0112] The Smart Sensing HAL is the core software application of Smart Sensing. The Smart Sensing Control Module includes the Smart Sensing Client Application (CA), which runs in the REE environment.

[0113] The kernel layer is the layer between hardware and software. At a minimum, the kernel layer includes the Qualcomm communication interface (QMI), an air mouse detection module, an eye-tracking module, a display driver, a camera driver, and sensor drivers.

[0114] The Qualcomm communication interface mentioned above is a multiprocessor inter-process communication interface provided by Qualcomm, used for calling functions, reading data, etc. The camera driver is the driver layer of the camera device, primarily responsible for interacting with the camera hardware.

[0115] Understandable, Figure 6 The layers in the illustrated structure and the components contained in each layer do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the structure may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0116] Based on the electronic devices described above, embodiments of this application provide an operation control method. This method can be applied to intelligent sensing scenarios (such as hand-eye coordination scenarios). The following description uses a mobile phone as the electronic device and a hand-eye coordination scenario as an example to illustrate the method of this application embodiment. Specifically, as... Figure 7 As shown, the operation control method may include S701 to S710.

[0117] S701, the mobile phone captures the first image frame.

[0118] The first image frame mentioned above is the image frame currently captured by the mobile phone, that is, the image frame within the field of view that the front-facing camera of the mobile phone can capture. The front-facing camera is a camera with AON function, that is, the front-facing camera can capture the first image frame in real time.

[0119] For example, the front-facing camera of a mobile phone can capture the first image frame frame by frame, or it can capture images at preset intervals of frames. For instance, the phone can capture one first image frame every two frames, or it can capture images at preset time intervals, such as every 2ms. It is understood that the preset number of frames and time intervals can be set according to actual needs and are not specifically limited.

[0120] In some embodiments, the first image frame can be used to identify the user's hand to enable the mobile phone to perform gesture control operations. However, to protect user privacy, it is necessary to determine whether the user has enabled the finger mouse function (or finger mouse service) of the smart sensing application before the mobile phone captures the first image frame. Only when the finger mouse function in the smart sensing application is enabled, that is, when the mobile phone receives the user's activation command for the finger mouse function in the smart sensing application, can the mobile phone capture the first image frame for further hand detection.

[0121] For example, such as Figure 8 As shown, when the user clicks the settings controls on the initial screen, the phone displays the settings interface, which includes settings for WLAN, Bluetooth, mobile network, desktop and wallpaper, smart assistant, and smart sensing. Subsequently, when the user clicks the smart sensing settings item within the settings interface, the phone displays the smart sensing settings interface. This smart sensing settings interface includes a smart gaze settings bar, an air gesture settings bar, and other settings bars. The smart gaze settings bar includes keeping the screen on when looking at the screen, reducing volume when looking at the screen, and eye tracking. The air gesture settings bar includes air swipe, air screenshot, air press, and finger mouse. Other settings bars include smart always-on display and smart landscape / portrait mode.

[0122] Afterwards, if any sensing function in the Smart Sensing settings interface is clicked, the phone can control whether the clicked sensing function is turned on or off. For example, please see... Figure 8 If the user clicks the "Finger Mouse" option in the air gesture settings, it means the user wants to enable the finger mouse function, and therefore the phone can enable the finger mouse function. For another example, please see again... Figure 8If the user clicks the "Air Screenshot" function in the air gesture settings, it means that the user wants to turn off the air screenshot function. Therefore, the phone can turn off the air screenshot function.

[0123] It is understandable that the above Figure 8 The interface shown for setting up the sensing function of the smart sensing application is only an example. Users can also control the sensing function to be turned on or off in other ways. For example, the phone can directly display the air gesture option so that the user can agree to control the air gesture function to be turned on or off. That is to say, if the air gesture option is clicked by the user, all the functions included in the air gesture option will be turned on or off.

[0124] In some embodiments, the personal information used in the technical solutions of this application is limited to information for which individual consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes authorization of relevant user information before users use the function.

[0125] S702, if the mobile phone detects that the user's hand is not included in the first image frame, the mobile phone continues to collect the first image frame.

[0126] Specifically, after acquiring the first image frame, the phone can perform hand detection on that frame to obtain a hand detection result. This result indicates whether the first image frame contains the user's hand. It can be understood that if the hand detection result includes a hand detection box, it means the first image frame contains a hand, indicating the user intends to trigger the finger-mouse service. Therefore, the phone can further determine if an eye-tracking cursor is present on the display. If the hand detection result does not include a hand detection box, it means the first image frame does not contain a hand, indicating the user is not currently likely to trigger the finger-mouse service. Therefore, the phone can continue acquiring the first image frame to further detect whether it contains the user's hand.

[0127] In some embodiments, the mobile phone can capture multiple consecutive first image frames. Then, the mobile phone can perform hand detection on each of the first image frames to determine whether the user's hand is included in each first image frame. Subsequently, if at least one of the multiple first image frames does not include the user's hand, or if the acquisition time corresponding to the multiple first image frames is less than a first preset time, the mobile phone can continue capturing first image frames. This reduces the occurrence of incorrect finger / mouse display caused by a hand mistakenly appearing in the shooting field of view, thereby improving the display accuracy of the finger / mouse.

[0128] The hand detection result may also include the confidence level of the hand detection box. That is, if the first image frame includes a hand, the hand detection result may include the hand detection box and its confidence level. The confidence level of the hand detection box indicates the degree of confidence of the hand detection box. The higher the confidence level of the hand detection box, the more accurate the hand detection result, that is, the more precise the hand detection box.

[0129] Specifically, the mobile phone can perform hand detection on the first image frame using a hand detection model to obtain hand detection results. In other words, the mobile phone can input the first image frame into the hand detection model to obtain hand detection results. This hand detection model can take the first image sample as input to the hand detection model to be trained. Through learning and prediction by the hand detection model to be trained, it outputs a detection box for the hand in the first image sample. Based on the output hand detection box and the ground truth bounding boxes of the hand in the first image sample, the parameters of the hand detection model to be trained are adjusted until a trained hand detection model is obtained.

[0130] For example, such as Figure 9 As shown, after the front-facing camera 32 of the mobile phone captures an image, the phone can input the image into a trained hand detection model to perform hand detection and obtain the hand detection result. It can be seen that the image includes the user's hand (such as a hand gesture with the palm outstretched). Therefore, after the phone inputs the image containing the user's hand into the hand detection model, the obtained hand detection result includes the hand detection box T and the confidence score of 0.87 corresponding to the hand detection box T. The hand detection box is used to indicate the position information of the user's hand. For example, the phone can use the detection box to mark the user's hand in the target image frame, and the position information of the user's hand is represented by the coordinates corresponding to the four vertices of the detection box.

[0131] It is understood that the hand detection model in this application embodiment is not a hand detection model for a specific user. In other words, the hand detection model cannot reflect the personal information of a specific user.

[0132] In some embodiments, if the hand detection result indicates that the first image frame does not include a hand, it means that the user is unlikely to trigger the finger mouse service. Therefore, the mobile phone can continue to collect the first image frame to further detect whether the first image frame contains the user's hand.

[0133] S703, when the mobile phone detects that the user's hand is included in the first image frame, the mobile phone determines whether there is an eye-tracking cursor on the display screen.

[0134] The aforementioned display interface is the interface displayed by the mobile phone when the first image frame is captured. The eye-tracking cursor is used to represent the user's gaze point position (or gaze point coordinates) on the display interface.

[0135] In some embodiments, if the hand detection result indicates that the first image frame includes a hand, it means that the user has triggered the air mouse function. Therefore, the mobile phone can further determine whether there is an eye-tracking cursor on the display interface. If there is an eye-tracking cursor on the display interface, it means that the mobile phone is performing an eye-tracking service, and therefore, the mobile phone can execute S704. If there is no eye-tracking cursor on the display interface, it means that the mobile phone is not currently performing an eye-tracking service, and therefore, the mobile phone can execute S709 to further perform gesture recognition on the user's hand included in the first image frame. The air mouse function is an air mouse function, which refers to the function of moving icons (such as a mouse, cursor, etc.) on the display interface remotely by moving the user's hand or control device in three-dimensional space.

[0136] It can be understood that the user's hand included in the first image frame is the front of the hand, that is, the mobile phone performs gesture recognition based on the feature points of the front of the hand.

[0137] In some embodiments, the control device can be a mouse (such as a wireless mouse) connected to a mobile phone, or a device capable of instructing a finger mouse to perform control operations (such as a cursor pen), etc., and the specific device is not limited. For example, if the object controlled by the finger mouse is a wireless mouse, the user needs to connect the wireless mouse to the mobile phone via Bluetooth before triggering the finger mouse function. Afterwards, once the wireless mouse and mobile phone are successfully connected, the user can wake up the wireless mouse by moving it. Then, the mobile phone can execute the target operation corresponding to the operation instruction of the wireless mouse.

[0138] In one implementation, such as Figure 10 As shown, the process of displaying the eye-tracking cursor on the mobile phone can specifically include S711 to S714:

[0139] S711, the phone receives notification messages.

[0140] The aforementioned notification message refers to a notification message sent by any application on the mobile phone. For example, the notification message can be an SMS notification or a video recommendation notification from a video application.

[0141] It's understandable that after receiving a notification message, the phone can display it in the notification bar on top of the main screen. For example, the phone can display the notification message as a pop-up at the top of the current screen. The main screen refers to the interface displayed before the phone received the notification message. The notification bar can also display at least one of the following: the application name corresponding to the notification message, the application icon, the time the notification message was received, and a user control. For example, the user control could be a "reply" control or a "mark as read" control, etc., without specific limitations.

[0142] For example, such as Figure 11 As shown, the phone displays its main interface, which includes icons for multiple applications. If the phone receives a notification message, it can display the message as a pop-up at the top of the main interface; specifically, pop-up A displays the SMS notification. Pop-up A can be understood as a notification bar, which displays the application name (SMS), the time of receipt (just now), the sender of the SMS notification (sister), the SMS content (When are you free tomorrow? Want to go shopping…), a "Reply" control, and a "Mark as Read" control.

[0143] S712, the mobile phone captures a third image frame, wherein the third image frame is an image frame captured when the mobile phone receives a notification message.

[0144] Specifically, after receiving a notification message, the phone can trigger the front-facing camera to capture a third image frame. This third image frame is the image frame captured when the phone receives the notification message, which is the image frame within the field of view that the phone's front-facing camera can capture.

[0145] In some embodiments, the aforementioned third image frame can be used to identify the user's face so that the mobile phone can perform eye-tracking operations. However, to protect the user's privacy, it is necessary to determine whether the user has enabled the eye-tracking function (or eye-tracking service) of the smart perception application before the mobile phone collects the third image frame. Only when the eye-tracking function in the smart perception application is enabled, that is, when the mobile phone receives the user's activation command for the eye-tracking function in the smart perception application, can the mobile phone collect the third image frame for further face detection.

[0146] For example, such as Figure 12 As shown, if any sensing function in the Smart Sensing settings interface is clicked, the phone can control whether the clicked sensing function is turned on or off. For example, if the user clicks the eye-tracking function in the Smart Gaze settings, it means that the user wants to enable the eye-tracking function. Therefore, the phone can display the details interface of the eye-tracking function so that the user can further select the sub-sensing functions they want to enable.

[0147] The details interface for the aforementioned eye-tracking function includes sub-perception functions (gaze-based notification, gaze-based notification to details), function instructions (at a distance of 20-50 cm from the screen, with your eyes facing the screen, gaze to expand the notification; pause briefly to access details), and function prompts (please ensure your eyes and face are unobstructed; when the gaze-focused notification is detected, the corresponding action will be performed for you. This function is not currently supported in landscape mode.)

[0148] Subsequently, if any sub-sensory function in the details screen of the aforementioned eye-tracking function is clicked, the phone can control whether the clicked sub-sensory function is turned on or off. For example, please see... Figure 12 If a user clicks the "Enter Details" button while looking at a notification, it means the user wants to enable the "Enter Details" feature. Therefore, the phone can enable this feature. Afterward, when the phone receives a notification, it can determine whether to allow access to the details screen based on the user's eye gaze. For example, please see again... Figure 12 If the user clicks on the gaze-expand message notification, it means that the user wants to turn off the gaze-expand message notification function. Therefore, the phone can turn off the gaze-expand message notification function.

[0149] It is understandable that the above Figure 12 The interface shown for setting up the sensing functions of a smart sensing application is just an example. Users can also control the sensing functions to be turned on or off in other ways. For example, the phone can directly display an eye-tracking option so that the user can agree to control the eye-tracking function to be turned on or off. That is, if the eye-tracking option is clicked by the user, all the sub-sensing functions included in that eye-tracking option will be turned on or off.

[0150] S713, the mobile phone performs eye movement recognition on the third image frame to obtain eye movement recognition data, which includes the coordinates of the user's gaze point.

[0151] In some embodiments, after acquiring a third image frame, the mobile phone can perform eye-tracking recognition on that third image frame to obtain the coordinates of the user's gaze point. These gaze point coordinates indicate the coordinate position of the user's gaze point on the display interface. In this embodiment, the mobile phone calculates the gaze point coordinates using the top-left corner of the display interface as the origin. In other embodiments, the mobile phone can also calculate the gaze point coordinates using other corners of the display interface (such as the top-right corner) or the center point of the display interface as the origin.

[0152] Specifically, the mobile phone can perform eye movement recognition on the aforementioned third image frame using an eye movement recognition algorithm to obtain eye movement recognition data. In other words, the mobile phone can input the third image frame into the eye movement recognition algorithm to obtain eye movement recognition data. This eye movement recognition algorithm is a biometric technology that performs eye movement recognition based on the user's eye feature information.

[0153] For example, such as Figure 13 As shown, after the front-facing camera 32 of the mobile phone captures the third image frame, the mobile phone can input the third image frame into the eye-tracking recognition algorithm for eye-tracking recognition to obtain eye-tracking recognition data. It can be seen that the third image frame includes a face, that is, the third image frame includes the user's eyes. Therefore, after the mobile phone inputs the third image frame into the eye-tracking recognition algorithm, the obtained eye-tracking recognition data includes the coordinates (x1, y1) of the user's gaze point.

[0154] The aforementioned eye-tracking data may also include the duration of the user's gaze. This gaze duration refers to the length of time the user's eyes remain at the aforementioned gaze point.

[0155] In some embodiments, the mobile phone can continuously capture multiple third image frames. Then, the mobile phone can perform eye-tracking recognition on each of the third image frames to obtain the coordinates of the user's gaze point in each third image frame. Subsequently, only if the coordinates of the user's gaze point are identical in each third image frame, and the acquisition duration (or the user's gaze duration) corresponding to the multiple third image frames is greater than a second preset duration, can the mobile phone display the eye-tracking cursor on the display interface according to the user's gaze point coordinates. This reduces the occurrence of incorrect eye-tracking cursor display due to the user's inadvertent gaze at the screen, thereby improving the display accuracy of the eye-tracking cursor.

[0156] In one implementation, after acquiring a third image frame, the mobile phone can perform face detection on that frame to obtain a face detection result. This result indicates whether the third image frame contains a face. It can be understood that if the face detection result includes a face detection bounding box, it means the third image frame contains a face, indicating a possibility that the user is looking at the notification message. Therefore, the mobile phone can perform eye-tracking on the third image frame to obtain eye-tracking data. If the face detection result does not include a face detection bounding box, it means the third image frame does not contain a face, indicating the user is not looking at the electronic device. The possibility of the user looking at the notification message is low, therefore, the mobile phone does not need to further perform eye-tracking on the third image frame. This reduces unnecessary resource waste and improves the utilization of computing resources.

[0157] In some embodiments, the face detection result may further include the confidence score of the face detection bounding box. That is, if the third image frame includes a face, the face detection result may include the face detection bounding box and its confidence score. The confidence score of the face detection bounding box refers to a confidence score of the output result. For example, if the third image frame captured by the phone is unclear, the phone's detection and output results will be inaccurate, resulting in a lower confidence score when outputting the face detection bounding box. In other words, the confidence score of the face detection bounding box indicates the degree of confidence of the face detection bounding box. The higher the confidence score, the more accurate the face detection result, meaning the face detection bounding box is more precise. For example, the phone may output the confidence score in numerical form, such as a confidence score of 0.88.

[0158] Specifically, the mobile phone can perform face detection on the third image frame using a face detection model to obtain the face detection result. In other words, the mobile phone can input the third image frame into the face detection model to obtain the face detection result. This face detection model can take the second image sample as input to the face detection model to be trained. Through learning and prediction by the face detection model to be trained, it outputs detection boxes of faces in the second image sample. Based on the output face detection boxes and the ground truth annotation boxes of faces in the second image sample, the parameters of the face detection model to be trained are adjusted until a well-trained face detection model is obtained.

[0159] The second image sample can be the same as or different from the first image sample. If the second image sample is the same as the first image sample, then the second image sample and the first image sample can contain both a face and a hand.

[0160] For example, such as Figure 14 As shown, after the front-facing camera 42 of the mobile phone captures image 43, the mobile phone can input image 43 into a trained face detection model to perform face detection and obtain the face detection result. It can be seen that image 43 includes a face; therefore, after the mobile phone inputs image 43 containing a face into the face detection model, the obtained face detection result includes a face detection bounding box Y and a confidence score of 0.97 corresponding to the face detection bounding box Y. The face detection bounding box is used to indicate the location information of the user's face. For example, the mobile phone can use the face detection bounding box to mark the face in the third image frame, and the coordinates corresponding to the four vertex corners of the face detection bounding box represent the location information of the face.

[0161] It is understood that the face detection model in this application embodiment is not a face detection model for a specific user. In other words, the face detection model cannot reflect the personal information of a specific user.

[0162] S714, the mobile phone displays the eye-tracking cursor on the display interface according to the above gaze point coordinates.

[0163] Specifically, after obtaining the coordinates of the user's gaze point, the mobile phone can display an eye-tracking cursor at the gaze point coordinates on the display screen.

[0164] For example, such as Figure 15 As shown, after receiving a notification message via SMS, the mobile phone can display it on the interface (such as...). Figure 15 On the main interface shown, a text message notification is displayed in pop-up window A. Afterwards, the phone can display an eye-tracking cursor T on the interface displaying the text message notification, based on the coordinates of the user's gaze point.

[0165] It is understandable that the above Figure 15 The shape and color of the eye cursor shown are just examples; phones can also be configured to display eye cursors of other shapes and colors on the screen. For example, a phone can display a triangular eye cursor. Or, for another example, a phone can display a light gray eye cursor.

[0166] In some embodiments, after determining the coordinates of the gaze point, the mobile phone may not display the eye cursor. That is, the mobile phone does not need to replace the eye cursor with a finger mouse. In other words, after the mobile phone detects that the user's hand is included in the first image frame and determines the coordinates of the gaze point, it can directly execute S705 to further perform gesture recognition on the user's hand in the second image frame. In this way, the occurrence of the user's visual experience being affected by the movement of the eye cursor can be reduced, and the user's user experience can be improved.

[0167] For example, if the aforementioned gaze point coordinates are located on any control in the notification bar, it indicates a possibility that the user might click on that control. Therefore, the phone can darken the text of that control to help the user determine whether it is the control they want to click. For example, please see... Figure 16 After receiving a text message notification, the phone can display the notification in the message notification bar B on the main screen. This notification bar displays the application name (SMS), the time of receipt (just now), the sender of the message (sister), the message content (When are you free tomorrow? Want to go shopping…), a "Reply" control, and a "Mark as Read" control. If the user's gaze is focused on the "Mark as Read" control in the notification bar, it indicates a potential click on the control; therefore, the phone can bold the "Mark as Read" text.

[0168] In some embodiments, the execution order of the above-described process of displaying notification messages and the process of displaying eye-tracking cursor is not limited. For example, the mobile phone may execute the above-described process of displaying notification messages first and then execute the process of displaying eye-tracking cursor, or the mobile phone may execute the above-described process of displaying notification messages and the process of displaying eye-tracking cursor simultaneously. The specific order is not limited.

[0169] It is understandable that the process of displaying the eye-tracking cursor, namely processes S711 to S714, can be executed before step S701. Specifically, when the mobile phone receives a notification message, it displays the eye-tracking cursor on the display interface based on the coordinates of the user's gaze point. Afterward, the mobile phone can capture a first image frame to further determine whether the eye-tracking cursor needs to be replaced based on whether the first image frame contains the user's hand.

[0170] In the S704, the phone replaces the aforementioned eye-tracking cursor with a finger mouse.

[0171] Specifically, after confirming the presence of an eye-tracking cursor on the displayed interface, the phone can directly replace the eye-tracking cursor with a finger mouse. In other words, the phone replaces the finger mouse with the cursor at the initial position of the eye-tracking cursor; that is, the current position of the eye-tracking cursor becomes the initial position of the finger mouse. This initial position represents the user's gaze point on the displayed interface. This not only achieves seamless integration between the eye-tracking cursor and the finger mouse, eliminating the need for the user to manually move the finger mouse to the notification bar, thus providing convenience, but also clearly indicates to the user that the finger mouse service is enabled, reducing the likelihood of erroneous operations due to accidental activation of the finger mouse service and improving the user experience.

[0172] The display effects of the finger mouse and eye cursor can be the same or different, without any specific limitation. For example, the eye cursor can be circular, and the finger mouse can be hand-shaped. Another example is that the eye cursor is dark gray, and the finger mouse is light gray.

[0173] For example, such as Figure 17 As shown, after receiving a notification message from an SMS, the phone can display an eye-tracking cursor M based on the coordinates of the user's gaze point. Then, if the phone captures an image frame containing the user's hand, it indicates that the user wants to trigger the finger mouse service. Therefore, the phone can replace the eye-tracking cursor M with a finger mouse cursor N. The user can then control the finger mouse by changing gestures, causing the phone to perform corresponding operations.

[0174] In some embodiments, the phone can only display a finger mouse at a preset position on the display interface if it determines that the first image frame includes a hand and the gesture corresponding to that hand is a preset gesture; that is, the phone enables the air mouse function. Afterward, the phone can control the finger mouse to perform corresponding operations based on the user's hand gestures, resulting in low execution efficiency. However, in this embodiment, as long as the first image frame includes a hand and the display interface has an eye-tracking cursor, the phone can replace the eye-tracking cursor with a finger mouse, thus enabling the air mouse function. In other words, the phone does not need to perform gesture recognition on the hand included in the first image frame, nor does it need to determine whether the recognized gesture is a specific gesture. Therefore, the air mouse function can be used directly, thereby improving the phone's execution efficiency.

[0175] In one implementation, if the finger mouse and the eye cursor are the same, the phone can directly move the eye cursor by detecting the user's hand after determining that the first image frame includes the hand, without having to replace the eye cursor with the finger mouse. This can reduce unnecessary resource waste.

[0176] In another implementation, if the finger mouse and the eye cursor are different, the phone can directly convert the eye cursor to a finger mouse after determining that the hand is included in the first image frame. Alternatively, the phone can first delete the eye cursor. Then, the phone can display the finger mouse according to the aforementioned gaze coordinates.

[0177] In another implementation, if the phone does not display the eye-tracking cursor, then after determining that the first image frame includes a hand, the phone will not display the finger mouse. In other words, after determining that the first image frame includes a hand, the phone can directly move the finger mouse by detecting the user's hand. This can reduce the impact on the user's visual experience caused by the movement of the eye-tracking cursor and improve the user's experience.

[0178] Specifically, if the coordinates of the aforementioned finger and mouse cursor are located at any control in the notification bar, it indicates that the user may have clicked that control. Therefore, the phone can darken the text of that control to help the user determine whether it is the control they want to click, thus reducing the occurrence of incorrect operations by the phone.

[0179] In some embodiments, if the first image frame is an image frame captured by the mobile phone when the user touches their eyes or hair, it means that the user does not want to trigger the finger mouse service. Therefore, even if the mobile phone detects an image frame with the user's hand, there is no need to replace the eye-tracking cursor with the finger mouse.

[0180] In one implementation, after replacing the eye-tracking cursor with a finger cursor, the phone can directly perform gesture recognition on the hand in the first image frame to obtain a third target gesture. Then, if the third target gesture is a first preset gesture, the phone can directly execute S707, thus improving the phone's execution efficiency. For example, let's take the first preset gesture as the OK gesture, and the target operation corresponding to the OK gesture as a click operation. If the gesture included in the first image frame is the OK gesture, and the position corresponding to the finger mouse is a "Mark as Read" control, it means the user wants to click the "Mark as Read" control. Therefore, the phone can perform a click operation on the "Mark as Read" control, deleting the display of the message notification bar; that is, the phone can hide the notification bar. Figure 15 The message notification bar A shown is shown.

[0181] If the third target gesture is not the first preset gesture, the phone can execute S705 to further determine whether the phone can perform gesture control operations.

[0182] S705, the mobile phone captures a second image frame, wherein the second image frame is the image frame captured after the mobile phone displays the finger mouse.

[0183] Specifically, after the phone displays a finger mouse cursor, it can trigger the front-facing camera to continuously capture a second image frame. This second image frame is the image frame captured after the phone displays the finger mouse cursor.

[0184] For example, the front-facing camera of a mobile phone can capture the second image frame frame by frame, or it can capture images at preset intervals of frames. For instance, the phone can capture one second image frame every two frames, or it can capture images at preset time intervals, such as one second image frame every 2ms. The preset number of frames and the preset time interval can be set according to actual needs and are not specifically limited.

[0185] S706, the mobile phone determines whether the gesture included in the second image frame is a first preset gesture.

[0186] Specifically, after the phone captures the second image frame, it can first perform hand detection on the second image frame to determine whether the second image frame includes the user's hand. Then, if the second image frame includes the user's hand, the phone continues to perform gesture recognition on the hand in the second image frame to obtain the first target gesture. This first target gesture is the gesture made by the user's hand in the second image frame.

[0187] In some embodiments, the mobile phone can capture multiple consecutive third image frames. Then, the mobile phone can perform gesture recognition on the hand in each of the third image frames to determine whether the gesture included in each third image frame is a first preset gesture. Subsequently, if the gestures included in the multiple third image frames are all the first preset gestures, and the acquisition duration corresponding to the multiple third image frames is greater than a third preset duration, the mobile phone can control a finger mouse to execute the target operation corresponding to the preset gesture. This avoids the mobile phone performing erroneous operations and improves the execution accuracy of the mobile phone.

[0188] Specifically, the mobile phone can perform gesture recognition on the hand in the second image frame based on the gesture recognition model to obtain the first target gesture. The gesture recognition model is a model that analyzes gestures based on the shape and movement trajectory of the hand in the image.

[0189] Then, the phone can determine whether the first target gesture is a first preset gesture. If the first target gesture is a first preset gesture, it means the user wants the phone to perform the operation corresponding to the preset gesture. Therefore, the phone can execute S707. If the first target gesture is not a first preset gesture, it means the user does not want to perform a gesture control operation. Therefore, the phone can execute S708. It can be understood that the first preset gesture can be a default gesture pre-configured by the phone, or a gesture pre-configured by the user according to their own habits, etc., without specific limitations. For example, a preset gesture with the palm closed corresponds to pressing the mouse button, and a preset gesture with the palm open corresponds to releasing the mouse button. Another example is a preset gesture with the index finger raised, which corresponds to moving the mouse button.

[0190] For example, such as Figure 18 As shown, Figure 18 Interface (a) shows the scene corresponding to the image frame of the user's closed hand captured by the front-facing camera 52 of the mobile phone. Figure 18Interface (b) shows the scene corresponding to the user's unfolded image frame captured by the phone's front-facing camera 52. Specifically, after capturing an image frame of the user's closed hand, the phone can perform gesture recognition on the hand in the image frame based on a gesture recognition model to obtain target gesture 1, which corresponds to the user's action of pressing the mouse. Then, after capturing an image frame of the user's unfolded hand, the phone can perform gesture recognition on the hand in the image frame based on a gesture recognition model to obtain target gesture 2, which corresponds to the user's action of releasing the mouse. The phone can then determine whether there is a gesture identical to target gesture 1 and target gesture 2 in the first preset gestures. If there is a gesture identical to target gesture 1 and target gesture 2 in the first preset gestures, the phone can execute the operation corresponding to target gesture 1 and target gesture 2; that is, the phone can execute a click operation on the "Reply" control.

[0191] In one implementation, the number of the aforementioned first preset gestures can be multiple. That is, the mobile phone can determine whether the aforementioned first target gesture is any of the first preset gestures. If the target gesture is any of the first preset gestures, the mobile phone can execute S707. If the first target gesture is not any of the first preset gestures, the mobile phone can execute S708.

[0192] S707, the mobile phone controls the finger mouse to execute the target operation corresponding to the first preset gesture mentioned above.

[0193] Specifically, after determining that the gesture included in the second image frame is a first preset gesture, the mobile phone can control the finger mouse to execute the target operation corresponding to the first preset gesture. In this way, eye-tracking cursor movement can be achieved through eye gaze, and corresponding operations can be executed through hand gestures, providing convenience and improving the user experience. Furthermore, after determining that the first target gesture is the first preset gesture, the mobile phone promptly executes the control operation corresponding to the first target gesture, improving the phone's execution accuracy.

[0194] For example, such as Figure 19 As shown, after determining that the image frame captured by the front-facing camera 52 includes a preset gesture of the user's outstretched hand, the phone can determine that the user has triggered a click operation. Therefore, the finger mouse on the phone can click on the "Reply" control. Subsequently, in response to this click operation, the phone can display a reply interface. This reply interface displays the receiving time (8:01 AM today), the sender of the SMS notification (sister), and the detailed SMS content (When are you free tomorrow? Want to go shopping and have dinner!).

[0195] S708, the phone does not perform gesture control operations.

[0196] Specifically, once it is determined that the gesture included in the second image frame is not a preset gesture, the phone may not perform the gesture control operation. This reduces the occurrence of incorrect operations by the phone, improves the phone's execution accuracy, and thus enhances the user experience.

[0197] S709, the mobile phone determines whether the gesture included in the first image frame is a second preset gesture.

[0198] In some embodiments, after determining that there is no eye-tracking cursor on the display interface, the mobile phone can perform gesture recognition on the user's hand included in the first image frame according to the gesture recognition model to obtain a second target gesture. The second target gesture is the gesture made by the user's hand in the first image frame.

[0199] Next, the phone can determine whether the second target gesture is a second preset gesture. If the second target gesture is a second preset gesture, it means the user wants the phone to perform the operation corresponding to the second preset gesture. Therefore, the phone can execute S710. If the second target gesture is not a second preset gesture, it means the user does not want to perform a gesture control operation. Therefore, the phone can return to execute S701 to continue acquiring the first image frame. It can be understood that the second preset gesture can be a default gesture pre-configured by the phone, or a gesture pre-configured by the user according to their own habits, etc., and there is no specific limitation. The second preset gesture can be the same as or different from the first preset gesture.

[0200] The S710 displays a finger mouse cursor in a preset position.

[0201] The aforementioned preset position can be the default position pre-configured by the phone, or a position pre-configured by the user according to their own habits, etc., without any specific limitation. For example, the preset position is the center position of the display interface, where the center position is the intersection of the two diagonals of the display interface. Another example is the position corresponding to the bottom left corner of the display interface.

[0202] In some embodiments, after determining that the gesture included in the first image frame is a second preset gesture, the mobile phone can display a finger mouse at a preset position on the display interface. Then, the mobile phone can directly control the finger mouse to perform the target operation corresponding to the second preset gesture. This improves the execution efficiency of gesture control operations.

[0203] In other embodiments, after displaying the finger mouse, the user can control the finger mouse by changing gestures to make the phone perform corresponding operations. That is, after the phone displays the finger mouse, it can return to executing the above-described S705. In this way, corresponding operations can be performed through the user's hand gestures, providing convenience and improving the user experience.

[0204] In one implementation, upon acquiring a first image frame, the phone can determine whether it can perform gesture control operations based on whether the first image frame includes a hand and whether an eye-tracking cursor is present on the display screen. The following will combine the above... Figure 4 The structure shown and Figure 20 The operation control flow shown details how the mobile phone performs the target operation.

[0205] S201, The sensor center in the mobile phone acquires the first image frame.

[0206] The aforementioned sensor hub is an independent subsystem used to connect to and process data from various sensor devices. For example, the sensor hub can receive image frames emitted by a camera sensor.

[0207] In some embodiments, after the front-facing camera of the mobile phone captures a first image frame, it can send the first image frame to the sensor center. It can be understood that the front-facing camera is one of the camera sensors described above.

[0208] S202, The sensor center performs hand detection on the first image frame and obtains the hand detection result.

[0209] The hand detection results are used to indicate whether the user's hand is included in the first image frame.

[0210] In some embodiments, after acquiring a first image frame, the sensor center can perform hand detection on the first image frame according to a hand detection model to obtain a hand detection result. If the hand detection result indicates that the first image frame includes the user's hand, that is, the hand detection result includes a hand detection box, it means that the user has the potential to trigger the finger mouse service. Therefore, the sensor center in the mobile phone can execute S203. If the hand detection result indicates that the first image frame does not include the user's hand, that is, the hand detection result does not include a hand detection box, it means that the user does not have the potential to trigger the finger mouse service. Therefore, the sensor center in the mobile phone can continue to acquire the first image frame to wait for the user to trigger the finger mouse service.

[0211] S203. If the hand detection result indicates that the user's hand is included in the first image frame, the sensor center sends a display icon indication to the hardware abstraction layer.

[0212] Specifically, after determining that the user's hand is included in the first image frame, the sensor center in the mobile phone can send a display icon indication to the hardware abstraction layer. This display icon indication means that the system user interface can display finger and mouse operations.

[0213] In this embodiment, as long as the sensor center determines that the first image frame includes the user's hand, the sensor center can send a display icon instruction to the hardware abstraction layer. There is no need to perform gesture recognition on the user's hand, nor is it necessary to send a display icon instruction to the hardware abstraction layer after determining that the recognized gesture is a preset gesture. In this way, while ensuring the accuracy of the operation performed by the mobile phone, the user's operation process is simplified, the execution efficiency of the mobile phone is improved, convenience is provided to the user, and the user experience is enhanced.

[0214] S204. Upon receiving the display icon instruction, the hardware abstraction layer sends a start icon instruction to the intelligent sensing service.

[0215] It is understandable that, due to the large number of algorithms in the hand detection model, meaning that the hand detection model is relatively complex, and the low latency requirement when activating the air mouse function, the mobile phone needs to send the icon indication carrying the first image frame to the intelligent sensing service so that the intelligent sensing service can further determine whether the first image frame contains the user's hand, thereby improving the accuracy of hand detection.

[0216] S205. Upon receiving a display icon instruction, the intelligent sensing service sends a start icon instruction to the system user interface.

[0217] S206. The hardware abstraction layer determines whether an eye-tracking cursor exists on the display interface.

[0218] In some embodiments, after the system user interface receives a display icon instruction, the hardware abstraction layer can determine whether an eye-tracking cursor exists on the display interface. If an eye-tracking cursor exists on the display interface, it indicates that the phone is performing an eye-tracking service. Therefore, the phone can execute S207 to replace the eye-tracking cursor with a finger mouse. If an eye-tracking cursor does not exist on the display interface, it indicates that the phone is not currently performing an eye-tracking service. Therefore, the phone can execute S210 to further display a finger mouse.

[0219] S207. When an eye-tracking cursor is present on the display interface, the hardware abstraction layer sends the coordinates of the user's eye gaze point to the driver layer or kernel layer.

[0220] Specifically, after determining that an eye-tracking cursor exists on the display interface, the hardware abstraction layer can send the coordinates of the user's gaze point to the driver layer or kernel layer. These gaze point coordinates indicate the coordinate position of the user's gaze point on the display interface.

[0221] In some embodiments, the gaze coordinates can be obtained by the sensor hub performing eye-tracking recognition on a third image frame using an eye-tracking algorithm. This third image frame is an image frame captured when the mobile phone receives a notification message. After obtaining the gaze coordinates, the sensor hub can send them to the hardware abstraction layer. The hardware abstraction layer can then send the received gaze coordinates to the driver layer or the kernel layer.

[0222] In one example, the hardware abstraction layer can first send the gaze coordinates to the driver layer. Then, upon receiving the gaze coordinates, the driver layer sends them to the kernel layer. In another example, the hardware abstraction layer can directly send the gaze coordinates to the kernel layer.

[0223] S208: Upon receiving the coordinates of the user's gaze point, the driver layer or kernel layer sends the coordinates of the user's gaze point to the input frame layer.

[0224] S209. Upon receiving the coordinates of the user's gaze point, the input frame layer adds a finger mouse according to the coordinates of the user's gaze point.

[0225] Specifically, after receiving the aforementioned gaze coordinates, the input frame layer can add a finger mouse based on these coordinates, allowing the finger mouse to be displayed at the gaze coordinates on the display interface. It's understandable that if an eye-tracking cursor exists on the display interface, it means an eye-tracking cursor is already displayed at the gaze coordinates. Therefore, to avoid distracting the user's vision due to multiple icons (eye-tracking cursor and finger mouse) on the display interface, if the input frame layer wants to display a finger mouse, it needs to first remove the existing eye-tracking cursor from the display interface to improve the user's visual experience.

[0226] S210. When there is no eye cursor on the display interface, the hardware abstraction layer sends an air mouse instruction to the driver layer or the kernel layer.

[0227] Specifically, after determining that there is no eye-tracking cursor on the display interface, the hardware abstraction layer can send an air mouse instruction to the driver layer or the kernel layer. This air mouse instruction indicates that the input framework layer can add finger mouse operations to the display interface.

[0228] S211. Upon receiving an empty mouse instruction, the driver layer or kernel layer sends an empty mouse instruction to the input frame layer.

[0229] S212. When the input frame layer receives an air mouse instruction, it adds a finger mouse at a preset position.

[0230] Specifically, after receiving the aforementioned air mouse instruction, the input frame layer can add a finger mouse at a preset position, allowing the finger mouse to be displayed at the preset location on the screen. It's understood that if the screen doesn't display an eye-tracking cursor, it means the phone hasn't received a notification message or the user hasn't enabled eye tracking. Therefore, by adding a finger mouse at the preset position on the screen, the input frame layer clearly indicates to the user that the finger mouse service is enabled, preventing erroneous operations caused by the user accidentally triggering the finger mouse service and thus improving the accuracy of the operation.

[0231] It is understood that S207-S209 and S210-S212 are parallel schemes. That is, if there is an eye-tracking cursor on the display interface, then S207-S209 is executed, and there is no need to execute steps S210-S212; if there is no eye-tracking cursor on the display interface, then S210-S212 is executed, and there is no need to execute steps S207-S209.

[0232] S213, The sensor center acquires the second image frame.

[0233] The second image frame mentioned above is an image frame captured by the front-facing camera of the mobile phone after the system user interface displays a finger mouse.

[0234] Specifically, after the phone displays a finger mouse cursor, it can trigger the front-facing camera to continuously capture a second image frame. This front-facing camera is an AON (Action On) camera, meaning it can capture the second image frame in real time.

[0235] In some embodiments, after the front-facing camera of the mobile phone captures a second image frame, it can send the second image frame to the sensor center.

[0236] S214. The sensor center determines the target coordinates of the finger mouse based on the position of the user's hand in at least two second image frames and the corresponding position coordinates when the finger mouse is added.

[0237] Specifically, after receiving at least two second image frames, the sensor center can perform position recognition on the user's hand in each second image frame to obtain the user's hand movement data. This hand movement data can include the direction and length of the user's hand movement.

[0238] Subsequently, the sensor center can determine the target coordinates of the finger mouse based on the aforementioned user hand movement data and the position coordinates corresponding to the addition of the finger mouse. These target coordinates are the position coordinates of the finger mouse on the display interface after the user's hand has moved.

[0239] For example, such as Figure 21 As shown, if a user makes an upward hand gesture, the sensor center in the phone can determine the user's hand movement data based on the hand's position. Then, the sensor center can determine the target coordinates N2 of the finger-mouse based on this movement data and the finger-mouse position coordinates N1. In other words, the finger-mouse on the phone's display will move from position coordinate N1 to target coordinate N2.

[0240] S215, The hardware abstraction layer receives the target coordinates sent by the sensor center.

[0241] Specifically, after determining the target coordinates of the finger mouse, the sensor center can send the target coordinates to the hardware abstraction layer, thus providing a basis for the subsequent input frame layer to move the finger mouse.

[0242] In some embodiments, the sensor hub can send the target coordinates of the finger mouse to the hardware abstraction layer via the QMI interface.

[0243] S216. Upon receiving the target coordinates, the hardware abstraction layer sends the target coordinates to the driver layer or the kernel layer.

[0244] S217. Upon receiving the target coordinates, the driver layer or kernel layer sends the target coordinates to the input frame layer.

[0245] S218. Upon receiving the target coordinates, the input frame layer moves the aforementioned finger mouse according to the target coordinates.

[0246] Specifically, after receiving the target coordinates, the input frame layer can move the finger mouse according to the target coordinates so that the finger mouse can be displayed at the target coordinates on the display interface.

[0247] S219. The sensor center performs gesture recognition on the user's hand in the second image frame to obtain the target gesture.

[0248] Specifically, after receiving the second image frame, the sensor center can perform gesture recognition on the user's hand in the second image frame according to the gesture recognition model to obtain the target gesture (or the first target gesture).

[0249] S220, the hardware abstraction layer receives the target gesture sent by the sensor center.

[0250] Specifically, after determining the target gesture, the sensor center can send the target gesture to the hardware abstraction layer, thus providing a basis for the subsequent input framework layer to display the operation interface.

[0251] S221. The hardware abstraction layer determines whether the above target gesture is a preset gesture.

[0252] In some embodiments, after receiving the target gesture, the hardware abstraction layer can determine whether the target gesture is a preset gesture (or a first preset gesture). If the target gesture is a preset gesture, it means that the user wants the phone to perform an operation corresponding to the preset gesture. Therefore, the phone can execute S222. In this way, the phone can promptly execute the control operation corresponding to the target gesture after determining that the target gesture is a preset gesture, thereby improving the execution accuracy of the phone. If the target gesture is not a preset gesture, it means that the user does not want to perform a gesture control operation. Therefore, the phone can choose not to perform a gesture control operation. This reduces the occurrence of erroneous operations by the phone, improves the execution accuracy of the phone, and thus enhances the user experience.

[0253] S222. When the target gesture is a preset gesture, the hardware abstraction layer sends an operation instruction to the driver layer or the kernel layer.

[0254] Specifically, after determining that the target gesture is a preset gesture, the hardware abstraction layer can send an operation instruction to the driver layer or the kernel layer. This operation instruction indicates that the input framework layer can draw the interface after executing the target operation.

[0255] It is understandable that, due to the high latency requirements when a mobile phone performs gesture control operations, the phone does not need to send the operation instructions carrying the second image frame to the intelligent sensing service, thus improving the phone's execution efficiency.

[0256] S223. Upon receiving an operation instruction, the driver layer or kernel layer sends an operation instruction to the input framework layer.

[0257] S224. Upon receiving an operation instruction, the input frame layer draws the operation interface according to the instruction.

[0258] Specifically, after receiving the above operation instructions, the input framework layer can draw the operation interface according to the operation instructions so that the mobile phone can display the operation interface.

[0259] In some embodiments, if the user's hand is not included in a series of consecutive image frames captured when the mobile phone is displaying an interface, or if the mobile phone receives a click operation from the user on the interface, it indicates that the user has exited the finger mouse service. Therefore, the mobile phone may not perform gesture control operations. Afterwards, the mobile phone may wait for the user to trigger the finger mouse service again.

[0260] In this embodiment, a mobile phone is used as an example for illustration. However, in some embodiments, the electronic device can also be a VR device. If the electronic device is a VR device, after receiving a notification message, the VR device can directly display a finger mouse based on whether the captured image frame includes the user's hand. That is, if the image frame captured by the VR device does not include the user's hand, it means that the user is unlikely to view the notification message, therefore, the VR device does not need to display a finger mouse; if the image frame captured by the VR device includes the user's hand, it means that the user is unlikely to view the notification message, therefore, the VR device can display a finger mouse. Subsequently, the VR device can perform corresponding control operations based on the gestures corresponding to the user's hand in subsequent image frames.

[0261] It can be understood that the user's hand included in the above image frame is the back of the hand, that is, the VR device performs gesture recognition based on the feature points of the back of the hand.

[0262] In some embodiments, this application provides a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method for adjusting usage parameters as described above.

[0263] In some embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the usage parameter adjustment method described above.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0265] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0266] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0267] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0268] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0269] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An operation control method, characterized in that, include: The electronic device acquires a first image frame; wherein, the first image frame is the image frame currently acquired by the electronic device; If the electronic device detects that the first image frame includes a hand, and if an eye-tracking cursor is present on the display interface, the electronic device replaces the display of a finger mouse at the first position where the eye-tracking cursor is displayed; wherein, the first position is used to characterize the position of the user's eye gaze point on the display interface; the electronic device replacing the display of a finger mouse at the first position where the eye-tracking cursor is displayed includes: stopping the display of the eye-tracking cursor at the first position and displaying the finger mouse at the first position; The electronic device controls the finger mouse to perform a target operation corresponding to a first preset gesture; wherein, the first preset gesture is obtained by gesture recognition based on the hand included in the target image frame, the target image frame includes the first image frame and / or the second image frame, and the second image frame is the image frame captured after the electronic device displays the finger mouse; During the process of displaying the finger mouse, the electronic device acquires multiple third image frames in response to receiving a notification message; For each of the third image frames, if the third image frame includes a face, the electronic device performs eye-tracking recognition on the third image frame to obtain the coordinates of the user's gaze point in the third image frame; If the coordinates of the user's gaze point are the same in each of the third image frames including the face and the user's gaze duration is greater than a preset duration, then the electronic device displays the eye-tracking cursor on the display interface according to the gaze point coordinates and stops displaying the finger mouse. Specifically, when the electronic device displays an eye-tracking cursor, if the coordinates of the gaze point are located in the target control in the notification bar of the notification message, the electronic device performs visual deepening processing on the text of the target control; when the electronic device displays a finger mouse, if the coordinates corresponding to the finger mouse are located in the target control, the electronic device performs visual deepening processing on the text of the target control. The target control is any control in the notification bar; visually deepening the text of the target control includes at least one of: deepening the text of the target control and bolding the text of the target control.

2. The method according to claim 1, characterized in that, The electronic device controls the finger mouse to perform a target operation corresponding to a first preset gesture, including: When the second image frame includes a hand, and the first target gesture corresponding to the hand is the first preset gesture, the electronic device controls the finger mouse to perform the target operation corresponding to the first preset gesture.

3. The method according to claim 2, characterized in that, The method further includes: If the second image frame does not include a hand, or if the first target gesture is not the first preset gesture, the electronic device does not perform gesture control operations.

4. The method according to claim 1, characterized in that, The electronic device controls the finger mouse to perform a target operation corresponding to a first preset gesture, including: When the hand gesture included in the first image frame corresponds to the first preset gesture, the electronic device controls the finger mouse to perform the target operation corresponding to the first preset gesture.

5. The method according to any one of claims 1-4, characterized in that, Before the electronic device performs eye-tracking recognition on the third image frame to obtain eye-tracking recognition data, the method further includes: The electronic device performs face detection on the third image frame according to a face detection model to obtain a face detection result; wherein, the face detection result is used to indicate whether the third image frame includes a face; If the face detection result indicates that the third image frame includes a face, the electronic device performs eye-tracking recognition on the third image frame according to the eye-tracking recognition algorithm to obtain the eye-tracking recognition data.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: If the eye-tracking cursor is not present on the display interface, the electronic device performs gesture recognition on the hand included in the first image frame to obtain the second target gesture. When the second target gesture is a second preset gesture, the electronic device displays the finger mouse on the display interface according to a preset position; The electronic device controls the finger mouse to perform the target operation corresponding to the second preset gesture.

7. The method according to claim 6, characterized in that, The method further includes: If the second target gesture is not the second preset gesture, the electronic device continues to acquire the first image frame.

8. An electronic device, characterized in that, The electronic device includes a camera, a memory, and one or more processors; the camera, the memory, and the processor are coupled; the camera is used to acquire images, the memory is used to store computer program code, the computer program code including computer instructions; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

Images