Operation method of electronic equipment, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides an operation method of electronic equipment, the electronic equipment and a computer readable storage medium, in the operation method of the electronic equipment, the electronic equipment detects human eye fixation point tracking performed by a user to obtain a fixation point position of the user; and then the electronic equipment can execute the operation corresponding to the operation of the user according to the gaze point position of the user and the acquired operation of the user, so that the electronic equipment can perform the corresponding operation according to the content identification result of the gaze area of the user in combination with the operation of the user.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to an operating method for an electronic device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Stylus pen interaction is becoming an increasingly popular trend due to its ease of use. Currently, there are two main ways to interact with stylus pen: (1) function switching: switching functions by double-clicking the pen body; (2) bringing up the secondary menu: bringing up the secondary menu by pinching the pen body. However, switching functions by double-clicking the pen body requires pre-setting which two functions to switch between, which is complicated and can only be done between the pre-set functions, making the functionality relatively limited. Similarly, bringing up the menu by pinching the pen body can only bring up the secondary menu under a certain primary tool, which is also relatively limited.

[0003] In addition, with the increasing size of mobile phone screens, including the emergence of foldable screens, it has become inconvenient and slow for users to operate with one hand; and currently, when users perform remote air gesture operations, they can only perform page turning up and down. Summary of the Invention

[0004] This application provides an operation method for an electronic device, an electronic device, and a computer-readable storage medium, so that the electronic device can perform corresponding operations based on the content recognition results of the user's gaze area and in conjunction with the user's operation.

[0005] In a first aspect, embodiments of this application provide an operation method for an electronic device, comprising: the electronic device detecting a user's gaze point to obtain the user's gaze point position; and executing an operation corresponding to the user's operation based on the user's gaze point position and the obtained user operation.

[0006] In one possible implementation, before detecting the user's gaze point and obtaining the user's gaze point position, the method further includes: acquiring the user's operation; wherein the user's operation includes the user's triggering operation on a stylus, the stylus being connected to the electronic device; the user's operation is sent to the electronic device after being detected by the stylus.

[0007] In one possible implementation, the electronic device detects the user's eye gaze point and obtains the user's gaze point position by: in response to the user's triggering operation on the stylus, the electronic device detects the user's eye gaze point and obtains the user's gaze point position.

[0008] In one possible implementation, the user's triggering operation on the stylus includes: a double-click operation on the stylus body; the step of executing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: acquiring the icon that the user is looking at in the current display interface of the electronic device based on the user's gaze point position; wherein, the current display interface includes the display interface of the target application currently running on the electronic device, and the target application includes a text and image editing application; when the distance between the icon being looked at by the user and the gaze point position is less than or equal to a predetermined distance threshold, the function corresponding to the icon being looked at by the user is sent to the stylus, so that the stylus can switch from its current function to the function corresponding to the icon being looked at by the user.

[0009] In one possible implementation, the user's triggering operation on the stylus includes: a light pinching operation of the stylus body by the user; the step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: acquiring the icon that the user is looking at in the current display interface of the electronic device based on the user's gaze point position; when the distance between the icon being looked at by the user and the gaze point position is less than or equal to a predetermined distance threshold, displaying the interface corresponding to the icon being looked at by the user based on the user's light pinching operation on the stylus body.

[0010] In one possible implementation, the user's triggering operation on the stylus includes: a light pinching operation on the stylus body; the step of executing the operation corresponding to the user's gaze point position and the acquired user operation includes: acquiring the page that the user is looking at in the current display interface of the electronic device based on the user's gaze point position; wherein, the current display interface of the electronic device is a video browsing interface or an animated image browsing interface; and playing the content corresponding to the page on the page that the user is looking at based on the user's light pinching operation on the stylus body, so that the user can preview the content corresponding to the page.

[0011] In one possible implementation, the user's triggering operation on the stylus includes: a sliding operation performed by the user on the stylus; the step of performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: obtaining the interface currently being viewed by the user based on the user's gaze point position; and sliding the interface currently being viewed by the user based on the sliding operation performed by the user on the stylus.

[0012] In one possible implementation, the user's triggering operation on the stylus includes: a sliding operation performed by the user on the stylus; the step of performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: acquiring the icon that the user is looking at in the current display interface of the electronic device based on the user's gaze point position; when the distance between the icon being looked at and the gaze point position is less than or equal to a predetermined distance threshold, adjusting the progress of the icon being looked at based on the sliding operation performed by the user on the stylus.

[0013] In one possible implementation, before executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation, the method further includes: acquiring the user's operation; wherein, the user's operation includes the user's gesture operation.

[0014] In one possible implementation, the step of performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: acquiring a message pop-up window that the user is looking at in the current display interface of the electronic device based on the user's gaze point position; after determining that the message pop-up window being looked at by the user is selected, displaying the message pop-up window being looked at by the user based on the user's first gesture operation; or, after determining that the message pop-up window being looked at by the user is selected, hiding the message pop-up window being looked at by the user based on the user's second gesture operation.

[0015] In one possible implementation, determining that the message pop-up being viewed by the user is in a selected state includes: detecting the duration for which the user views the message pop-up; and determining that the message pop-up being viewed by the user is in a selected state when the duration for which the user views the message pop-up is greater than or equal to a predetermined duration threshold.

[0016] In one possible implementation, the first gesture operation is the user's finger sliding up and down on the screen of the electronic device; the second gesture operation is the user's finger sliding left and right on the screen of the electronic device.

[0017] In one possible implementation, the step of performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: acquiring the content that the user is gazing at in the current display interface of the electronic device based on the user's gaze point position; and performing a corresponding operation on the content that the user is gazing at based on the user's gesture operation; wherein the user's gesture operation includes gesture operations performed by the user within the perception range of the camera of the electronic device.

[0018] In one possible implementation, the user's gesture operation includes a two-finger pinch or a two-finger stretch operation; the corresponding operation on the content being looked at by the user based on the user's gesture operation includes: when the user's gesture operation is a two-finger pinch operation, shrinking the content being looked at by the user; when the user's gesture operation is a two-finger stretch operation, magnifying the content being looked at by the user.

[0019] In one possible implementation, the user's gesture operation includes swiping up with two fingers or swiping down with two fingers; the corresponding operation on the content the user is looking at based on the user's gesture operation includes: when the user's gesture operation is swiping up with two fingers, the content the user is looking at is displayed by swiping upwards; when the user's gesture operation is swiping down with two fingers, the content the user is looking at is displayed by swiping downwards.

[0020] In one possible implementation, the user's gesture operation includes a rotation gesture; the step of performing corresponding operations on the content the user is looking at based on the user's gesture operation includes: performing a corresponding rotation operation on the content the user is looking at based on the rotation direction of the user's gesture.

[0021] In one possible implementation, the electronic device includes one of at least two interconnected electronic devices; the step of performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: when the electronic device determines that the user's gaze point position is within its own screen range, performing an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation.

[0022] Secondly, embodiments of this application provide an electronic device, including: one or more processors; a memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method provided in the first aspect.

[0023] It should be understood that the second aspect of the embodiments of this application is consistent with the technical solution of the first aspect of the embodiments of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be described again.

[0024] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0025] Fourthly, embodiments of this application provide a computer program that, when executed by a computer, performs the method provided in the first aspect.

[0026] In one possible design, the program in the fourth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating how to configure the double-tap switching function of the stylus through the settings interface;

[0028] Figure 2 This is a schematic diagram of the interface in one-handed mode on a mobile phone.

[0029] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of a stylus provided in one embodiment of this application;

[0031] Figure 5 A flowchart illustrating an operation method of an electronic device provided in one embodiment of this application;

[0032] Figure 6 A schematic diagram illustrating an application scenario provided by one embodiment of this application;

[0033] Figure 7 A flowchart of an operation method for an electronic device is provided as an example of this implementation.

[0034] Figure 8 This is a schematic diagram of the current display interface of an electronic device 100 provided in one embodiment of this application;

[0035] Figure 9(a) is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0036] Figure 9(b) is a schematic diagram of the current display interface of the electronic device 100 provided in another embodiment of this application;

[0037] Figure 10 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0038] Figure 11 A flowchart of an operation method for an electronic device is provided as another example of this implementation.

[0039] Figure 12 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0041] Figure 14 A flowchart of the operation method of an electronic device is provided as another example of this implementation.

[0042] Figure 15 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0043] Figure 16 A schematic diagram illustrating an application scenario provided for another embodiment of this application;

[0044] Figure 17 A flowchart of an operation method for an electronic device is provided as an example of this implementation.

[0045] Figure 18 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application;

[0046] Figure 19 A flowchart of an operation method for an electronic device is provided as another example of this implementation.

[0047] Figure 20 A schematic diagram of a gesture operation provided in one embodiment of this application;

[0048] Figure 21 A schematic diagram illustrating an application scenario provided in yet another embodiment of this application;

[0049] Figure 22 A schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0050] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0051] In existing technologies, there are currently two main ways to interact with a stylus: switching functions and bringing up a secondary menu.

[0052] The function switching is triggered by double-clicking the pen body, but users can only configure different function switching settings through the settings interface or menu bar. Figure 1 As shown, Figure 1 This is a diagram illustrating how to configure the double-tap toggle function for the stylus pen through the settings interface. Figure 1As can be seen, in the settings interface, users can configure the double-tap switching function of the stylus. Specifically, users can set "switch between current tool and eraser", "switch between current tool and last used tool", "show color palette" or "global shortcut menu". After the user sets these settings, when the user double-tap the stylus, the stylus can only switch functions according to the user's settings. This function switching is relatively simple. If the user wants to change the function switched by double-tapping the stylus, the user can only re-set it through the settings interface, which is a relatively complicated operation.

[0053] In existing technologies, secondary menus are accessed by pinching the pen body. However, the same interface may have multiple different primary toolbars, such as colors or brushes. Accessing secondary menus by pinching the pen body can only bring up the secondary menu under a specific primary tool, making the functionality relatively limited. Furthermore, users need to pre-configure which primary tool's secondary menu to access by pinching the pen body, which also presents a problem of operational complexity.

[0054] In addition, regarding the issue of increasingly larger mobile phone screens making one-handed operation inconvenient, existing technologies use an interactive method to shrink the displayed content to the lower left or right corner of the phone screen, thereby facilitating one-handed operation. For example... Figure 2 As shown, Figure 2 This is a schematic diagram of the interface in one-handed mode on a mobile phone. However, this method reduces the screen display area and results in a poorer display quality.

[0055] Furthermore, in existing related technologies, air gesture operation can only perform page turning operations, and cannot select a single interface element from among multiple interface elements included in the display interface, thereby triggering the corresponding interaction.

[0056] Based on the above problems, this application provides an operation method for an electronic device, which enables the electronic device to perform corresponding operations based on the content recognition results of the user's gaze area and in conjunction with the user's operation.

[0057] The operation method of the electronic device provided in this application embodiment can be applied to electronic devices, wherein the electronic devices can be smartphones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc.; this application embodiment does not impose any restrictions on the specific type of electronic device.

[0058] For example, Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application, as shown below. Figure 3 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0059] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0061] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0062] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0064] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (DCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0065] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0066] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0067] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0068] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the electronic device 100 to capture images. The processor 110 and the display screen 194 communicate via the DSI interface to enable the electronic device 100 to display images.

[0069] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0070] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0071] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0072] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0073] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0074] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0075] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0076] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0077] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0078] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0079] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0080] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0081] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0082] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0083] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0084] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. In this embodiment, the camera 193 may include a red-green-blue (RGB) camera and / or an infrared (IR) camera.

[0085] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0086] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0087] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0088] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0089] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0090] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0091] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0092] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0093] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0094] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0095] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0096] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0097] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0098] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0099] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0100] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0101] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0102] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 may use the proximity sensor 180G to detect when a user holds the electronic device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0103] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0104] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0105] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs thermal protection by reducing the performance of a processor located near temperature sensor 180J to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 100 heats battery 142 to prevent abnormal shutdown of electronic device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0106] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0107] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0108] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0109] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0110] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0111] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0112] Furthermore, the implementation of the electronic device operation method provided in this application embodiment involves not only the electronic device side, but also the stylus.

[0113] For example, Figure 4 This is a schematic diagram of the structure of a stylus provided in one embodiment of this application, as shown below. Figure 4 As shown, the stylus 400 may include a processor 410, a sensor module 420, a pen body interaction module 430, a status indicator 440, a button 450, an electrode 460, a sensing circuit 470, and a Bluetooth module 480.

[0114] The processor 410 may include storage and processing circuitry for supporting the operation of the stylus 400. The storage and processing circuitry may include storage devices such as non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured as a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. The processing circuitry in the processor 410 can be used to control the operation of the stylus 400. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, application-specific integrated circuits, etc.

[0115] The sensor module 420 may include one or more sensors. For example, the sensor module 420 may include a pressure sensor. The pressure sensor may be located at the writing end of the stylus 400. Alternatively, the pressure sensor may be located inside the stylus barrel of the stylus 400, so that when one end of the stylus tip is subjected to force, the other end of the tip moves and applies the force to the pressure sensor. In one embodiment, the processor 410 can adjust the line thickness of the stylus 400 when writing based on the pressure detected by the pressure sensor.

[0116] Sensor module 420 may also include inertial sensors. Inertial sensors may include triaxial accelerometers and triaxial gyroscopes, and / or other components for measuring the movement of the stylus 400, such as a triaxial magnetometer which may be included in the sensor as a nine-axis inertial sensor. Additionally, sensor module 420 may include additional sensors, such as temperature sensors, ambient light sensors, light-based proximity sensors, contact sensors, magnetic sensors, and / or other sensors.

[0117] The pen body interaction module 430 is used to acquire the user's operation on the pen body, such as double-clicking or pinching the pen body. Then, the pen body interaction module 430 sends the double-click or pinching event to the electronic device 100 via the Bluetooth module 480 to trigger the electronic device 100 to detect the user's gaze point position.

[0118] The status indicator 440 can be a light-emitting diode (LED), and it is used to indicate the status of the stylus 400 to the user. The button 450 can include mechanical and non-mechanical buttons, and it can be used to collect button press information from the user.

[0119] In this embodiment of the application, the stylus 400 may include one or more electrodes 460, one electrode 460 may be located at the writing end of the stylus 400, and the other electrode 460 may be located inside the tip of the stylus.

[0120] Sensing circuit 470 can sense capacitive coupling between electrode 460 and the drive line of the capacitive touch sensor panel that interacts with stylus 400. Sensing circuit 470 may include an amplifier for receiving capacitance readings from the capacitive touch sensor panel, a clock for generating a demodulated signal, a phase shifter for generating a phase-shifted demodulated signal, a mixer for demodulating the capacitance reading using in-phase demodulation frequency components, and a mixer for demodulating the capacitance reading using quadrature demodulation frequency components. The result of the mixer demodulation can be used to determine an amplitude proportional to the capacitance, allowing stylus 400 to sense contact with the capacitive touch sensor panel.

[0121] Understandably, depending on actual needs, the stylus 400 may also include a microphone, speaker, audio generator, vibrator, camera, data port, and other devices. Users can use these devices to provide commands to control the operation of the stylus 400 and the electronic device 100 that interacts with the stylus 400, and to receive status information and other outputs.

[0122] The processor 410 can be used to run software on the stylus 400 that controls the operation of the stylus 400. During operation of the stylus 400, the software running on the processor 410 can process sensor input, button input, and input from other devices to monitor the movement of the stylus 400 and other user input. The software running on the processor 410 can detect user commands and can communicate with the electronic device 100.

[0123] To support wireless communication between the stylus 400 and the electronic device 100, the stylus 400 may include a wireless module. Figure 4 This example uses Bluetooth module 480 as an example of a wireless module. In specific implementations, the wireless module can also be a Wi-Fi hotspot module or a Wi-Fi peer-to-peer module, etc. This embodiment does not limit the specific form of the wireless module. Bluetooth module 480 may include a radio frequency transceiver, such as a transceiver. Bluetooth module 480 may also include one or more antennas. The transceiver can use the antennas to transmit and / or receive wireless signals. The wireless signals, depending on the type of wireless module, can be Bluetooth signals, wireless LAN signals, long-range signals such as cellular phone signals, near-field communication signals, or other wireless signals.

[0124] The stylus 400 may also include a charging module 490, which can support charging of the stylus 400 and provide power to the stylus 400.

[0125] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 3 The electronic device 100 with the structure shown has Figure 4 Taking the stylus 400 with the structure shown as an example, the operation method of the electronic device provided in the embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0126] Figure 5 A flowchart illustrating an operation method of an electronic device provided in one embodiment of this application is shown below. Figure 5 As shown, the operation method of the above-mentioned electronic device may include:

[0127] Step 501: The electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0128] Specifically, the electronic device 100 can detect the user's gaze point through the camera 193 and obtain the position of the user's gaze point.

[0129] Step 502: The electronic device 100 performs an operation corresponding to the user's gaze point position and the acquired user operation.

[0130] In the above-mentioned operation method of the electronic device, the electronic device 100 detects the user's gaze point by tracking the user's gaze point, obtains the user's gaze point position, and then the electronic device 100 can perform an operation corresponding to the user's operation based on the user's gaze point position and the obtained user operation. Thus, the electronic device 100 can perform corresponding operations based on the content recognition results of the user's gaze area and in combination with the user's operation.

[0131] Figure 5 In one implementation of the illustrated embodiment, before step 501, the electronic device 100 can acquire the user's operation; wherein, the user's operation can be a trigger operation on the stylus 400, and the stylus 400 is connected to the electronic device 100. Specifically, the stylus 400 can be connected to the antenna 2 and wireless communication module 160 of the electronic device 100 through the Bluetooth module 480 and the antenna of the stylus 400; after detecting the user's trigger operation on the stylus 400, the stylus 400 can send the user's trigger operation event to the electronic device 100, so that the electronic device 100 can acquire the user's trigger operation on the stylus 400.

[0132] In this implementation, step 501 can be: in response to the user's triggering operation on the stylus, the electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0133] The application scenarios for this implementation method can be as follows: Figure 6 As shown, Figure 6 This is a schematic diagram illustrating an application scenario provided by one embodiment of this application. Figure 6 The scenario shown includes a stylus 400 and an electronic device 100. The electronic device 100 and the stylus 400 can be connected via wireless communication methods such as Bluetooth and / or WiFi. In actual use, the electronic device 100 and the stylus 400 are used together.

[0134] In one example of this implementation, Figure 6 In the application scenario shown, the interaction process between the user, the stylus 400, and the electronic device 100 can be as follows: Figure 7 As shown, Figure 7 A flowchart of an operation method for an electronic device provided as an example of this implementation may include:

[0135] Step 701: After detecting the user's trigger operation on the stylus 400, the stylus 400 sends the user's trigger operation to the electronic device 100.

[0136] Among them, the user's triggering operation for the stylus includes: double-clicking the stylus 400 body.

[0137] Step 702: In response to the user's trigger operation on the stylus 400, the electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0138] In this example, after the user double-clicks the pen body, the stylus 400 sends the double-click event to the electronic device 100. In response to the user's double-click operation on the stylus 400, the electronic device 100 detects the user's gaze point through the camera 193 and obtains the user's gaze point position.

[0139] Step 703: The electronic device 100 obtains the icon that the user is looking at in the current display interface of the electronic device 100 based on the user's gaze point position.

[0140] The current display interface can be the display interface of the target application currently running on the electronic device 100. The target application can be a text and image editing application, such as a note-taking or drawing application.

[0141] Step 704: When the distance between the icon being gazed at by the user and the aforementioned gaze point position is less than or equal to a predetermined distance threshold, the electronic device 100 sends the function corresponding to the icon being gazed at by the user to the stylus 400.

[0142] Step 705: The stylus 400 switches from its current function to the function corresponding to the icon the user is looking at.

[0143] The predetermined distance threshold can be set by the system based on performance and / or implementation requirements during the specific implementation. This embodiment does not limit the size of the predetermined distance threshold.

[0144] Specifically, the electronic device 100 can use the user's gaze point as the center to obtain the icon on the current display screen that is closest to the gaze point. This icon closest to the gaze point is the icon the user is looking at. After obtaining the icon the user is looking at, the electronic device 100 can further detect the distance between the icon the user is looking at and the gaze point. When the distance between the icon the user is looking at and the gaze point is less than or equal to a predetermined distance threshold, the electronic device 100 sends the function corresponding to the icon the user is looking at to the stylus 400, and then the stylus 400 can switch from its current function to the function corresponding to the icon the user is looking at.

[0145] See Figure 8 The target application currently running on electronic device 100 is a note-taking application; therefore, the interface currently displayed on electronic device 100 is the interface of the note-taking application, and the user is using stylus 400... Figure 8 Write notes in the interface shown. Figure 8This is a schematic diagram of the current display interface of an electronic device 100 provided in one embodiment of this application.

[0146] If the current function of the stylus 400 is "ballpoint pen brush" ( Figure 8 In the image (where 81 represents a ballpoint pen brush), after the user double-clicks the stylus 400, the stylus 400 sends the double-click operation to the electronic device 100, triggering the electronic device 100 to detect the user's gaze point. The electronic device 100 can then determine the icon the user is currently looking at as a "pen brush" based on the user's gaze point location on the current display screen of the electronic device 100. Figure 8 In the image, 82 represents a "pen brush," and the distance between the "pen brush" icon 82 and the user's gaze point is less than or equal to a predetermined distance threshold. Therefore, the electronic device 100 can send the "pen brush" function to the stylus 400, allowing the stylus 400 to switch from the currently used "ballpoint pen brush" to the "pen brush." Figure 8 As can be seen, after the electronic device 100 determines that the distance between the "pen brush" icon 82 and the user's gaze point is less than or equal to a predetermined distance threshold, the "pen brush" icon 82 is selected and the background color is gray.

[0147] Furthermore, in this example, after the electronic device 100 detects the distance between the icon the user is looking at and the aforementioned gaze point location, if the distance between the icon the user is looking at and the aforementioned gaze point location is greater than a predetermined distance threshold, then the electronic device 100 can, according to the user's preset settings in the settings interface, notify the stylus 400 to switch tools, such as... Figure 1 As shown, the electronic device 100 can instruct the stylus 400 to switch the current tool with the eraser or the current tool with the last used tool; alternatively, the electronic device 100 can follow the user's preset settings in the settings interface, such as... Figure 1 As shown, this displays the color palette or global shortcut menu.

[0148] In other words, when the distance between the icon being looked at by the user and the aforementioned gaze point is less than or equal to a predetermined distance threshold, the current function of the stylus 400 is switched to the function corresponding to the icon being looked at by the user; when the distance between the icon being looked at by the user and the aforementioned gaze point is greater than the predetermined distance threshold, the operation is performed according to the settings preset by the user in the settings interface.

[0149] In this example, no prior settings are required in the settings interface. Users can simply double-click the stylus 400 to switch the currently used function. The operation is simple and convenient.

[0150] In another example of this implementation, Figure 6In the application scenario shown, the user's trigger operation for the stylus can be: the user lightly pinches the stylus 400 body.

[0151] and Figure 7 The process is the same as described above. After the user lightly pinches the body of the stylus 400, the stylus 400 can detect the pinching operation and then send it to the electronic device 100. In response to the user's double-click operation on the stylus 400, the electronic device 100 detects the user's gaze point through the camera 193 to obtain the user's gaze point position. Then, based on the user's gaze point position, the electronic device 100 obtains the icon that the user is looking at in the current display interface of the electronic device 100. When the distance between the icon being looked at and the gaze point position is less than or equal to a predetermined distance threshold, the interface corresponding to the icon being looked at is displayed based on the user's pinching operation on the stylus.

[0152] The predetermined distance threshold can be set by the system based on performance and / or implementation requirements during the specific implementation. This embodiment does not limit the size of the predetermined distance threshold.

[0153] Specifically, after obtaining the user's gaze point position, the electronic device 100 can use the user's gaze point position as the center to obtain the icon in the current display interface of the electronic device 100 that is closest to the gaze point position. This icon closest to the gaze point position is the icon that the user is looking at. After obtaining the icon that the user is looking at, the electronic device 100 can further detect the distance between the icon that the user is looking at and the gaze point position. When the distance between the icon that the user is looking at and the gaze point position is less than or equal to a predetermined distance threshold, the electronic device 100 can display the interface corresponding to the icon that the user is looking at based on the user's light pinch operation on the stylus 400.

[0154] Referring to Figure 9(a), the target application currently running on the electronic device 100 is a note-taking application. Therefore, the interface currently displayed on the electronic device 100 is the interface of the note-taking application, and the user is using the stylus 400 to write notes on the interface shown in Figure 9(a). Figure 9(a) is a schematic diagram of the currently displayed interface of the electronic device 100 provided in another embodiment of this application.

[0155] Assuming the current function of the stylus 400 is "ballpoint pen brush" (91 in Figure 9(a) is the ballpoint pen brush), after the user lightly pinches the body of the stylus 400, the stylus 400 sends the pinching operation to the electronic device 100, thereby triggering the electronic device 100 to detect the user's eye gaze point. Then, the electronic device 100 can determine the icon that the user is currently looking at on the current display interface of the electronic device 100, which is the icon 91 of the "ballpoint pen brush". If the distance between the marker 91 and the user's gaze point is less than or equal to a predetermined distance threshold, the electronic device 100 can display the interface 92 corresponding to the "ballpoint pen brush" icon 91 based on the user's light pinching operation on the stylus 400. The interface 92 can be a secondary menu of the "ballpoint pen brush" and displays the detailed setting parameters of the "ballpoint pen brush". The detailed setting parameters of the "ballpoint pen brush" may include line type, thickness and / or color, etc. This embodiment does not limit the detailed setting parameters of the "ballpoint pen brush".

[0156] Figure 9(b) is a schematic diagram of the current display interface of the electronic device 100 provided in another embodiment of this application. As shown in Figure 9(b), the target application currently running on the electronic device 100 is a drawing application, and the interface currently displayed on the electronic device 100 is the interface of the drawing application. The user is using a stylus 400 to perform drawing operations on the interface shown in Figure 9(b).

[0157] Assuming the current function of the stylus 400 is "ballpoint pen brush" (93 in Figure 9(b) is the ballpoint pen brush), after the user lightly pinches the body of the stylus 400, the stylus 400 sends the user's pinching operation to the electronic device 100, thereby triggering the electronic device 100 to detect the user's eye gaze point. Then, the electronic device 100 can obtain the icon 94 of "color" that the user is looking at in the current display interface of the electronic device 100 according to the user's gaze point position, and the distance between the "color" icon 94 and the user's gaze point position is less than or equal to a predetermined distance threshold. Thus, the electronic device 100 can display the interface 95 corresponding to the "color" icon 94 according to the user's pinching operation on the body of the stylus 400. The interface 95 can be a second-level menu of "color", and the interface 95 displays the detailed setting parameters of "color". The detailed setting parameters of "color" may include color wheel, color circle and / or color card, etc. This embodiment does not limit the detailed setting parameters of "color".

[0158] In addition, the interaction method between the stylus 400 and the electronic device 100 provided in this example can also be extended to define a new interactive menu. For example, when the user is looking at the "color" icon, the user's pinching of the stylus triggers the electronic device 100 to display a list of colors that the user has recently used, making it easier for the user to select quickly.

[0159] Figure 10 This is a schematic diagram of the current display interface of the electronic device 100 provided in another embodiment of this application, as shown below. Figure 10 As shown in (a), the electronic device 100 is currently displaying the main interface. Figure 10 In the illustrated embodiment, when a user is viewing the main interface currently displayed on the electronic device 100, they are holding a stylus 400. After the user lightly pinches the stylus 400, the stylus 400 sends the pinching action to the electronic device 100, triggering the electronic device 100 to detect the user's gaze point. The electronic device 100 can then determine the icon 1001 of the "Music" application that the user is currently viewing on the electronic device 100's display interface based on the user's gaze point location. When the distance between the "Music" application icon 1001 and the aforementioned gaze point location is less than or equal to a predetermined distance threshold, the electronic device 100 can display the interface corresponding to the "Music" application icon 1001 based on the user's pinching action on the stylus 400. Figure 10 As shown in (b).

[0160] In this example, no prior settings are required in the settings interface. When the user lightly pinches the stylus 400, the electronic device 100 displays the interface corresponding to the icon that the user is looking at. The operation is simple and convenient.

[0161] In another example of this implementation, Figure 6 In the application scenario shown, the interaction process between the user, the stylus 400, and the electronic device 100 can be as follows: Figure 11 As shown, Figure 11 A flowchart of an operation method for an electronic device, provided as another example of this implementation, may include:

[0162] Step 1101: After detecting the user's trigger operation on the stylus 400, the stylus 400 sends the user's trigger operation to the electronic device 100.

[0163] Among them, the user's triggering operation for the stylus includes: the user's light pinching operation on the stylus 400 body.

[0164] In step 1102, in response to the user's trigger operation on the stylus 400, the electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0165] In this example, after the user lightly pinches the pen body, the stylus 400 sends the pinching event to the electronic device 100. In response to the user's pinching of the stylus 400, the electronic device 100 detects the user's gaze point through the camera 193 and obtains the user's gaze point position.

[0166] Step 1103: The electronic device 100 obtains the page that the user is looking at in the current display interface of the electronic device 100 based on the user's gaze point position.

[0167] The electronic device 100 currently displays a video browsing interface or an animated image browsing interface.

[0168] Step 1104: Based on the user's light pinching operation on the stylus 400, the electronic device 100 plays the content corresponding to the page being viewed on the page, so that the user can preview the content corresponding to the page.

[0169] See Figure 12 , Figure 12 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application. Figure 12 In this context, the current display interface of the electronic device 100 is a video browsing interface, which includes 11 pages, and each page can currently display the cover of the corresponding video content.

[0170] When a user is viewing the video browsing interface currently displayed on electronic device 100, and is holding a stylus 400, after the user lightly pinches the stylus 400, the stylus 400 sends the pinching action to electronic device 100, thereby triggering electronic device 100 to detect the user's eye gaze point. Based on the user's gaze point position, electronic device 100 determines that the page the user is currently looking at in the displayed interface is page 5. Thus, based on the user's pinching action on the stylus 400, electronic device 100 can play the video content corresponding to page 5, allowing the user to preview the video content on page 5.

[0171] In practice, the electronic device 100 plays the content corresponding to the page on the user's viewing page based on the user's pinch operation on the stylus 400. This can be achieved as follows: if the page currently displayed by the user is the cover image of the corresponding video content, then the electronic device 100 plays the content corresponding to that page based on the user's pinch operation on the stylus 400; if the page currently displayed by the user is the video content corresponding to that page, then the electronic device 100 stops playing the video content based on the user's pinch operation on the stylus 400. Simply put, a single pinch of the stylus triggers the electronic device 100 to play the video content, and a second pinch triggers the electronic device 100 to stop playing the video content.

[0172] Alternatively, the electronic device 100 can play the content corresponding to the page on the user's viewing page based on the user's pinching action on the stylus 400 body. This can be achieved as follows: if the page currently displayed by the user shows the cover image of the corresponding video content, and the user continues to pinch the stylus 400 body, then the electronic device 100 plays the content corresponding to that page. If the page currently displayed by the user shows the corresponding video content, then the electronic device 100 stops playing the video content after the user stops pinching the stylus 400 body. In short, the electronic device 100 plays the video content while the user continues to pinch the stylus 400 body, and the electronic device 100 stops playing the video content when the user releases the stylus 400 body.

[0173] In addition, while the electronic device 100 is playing the content corresponding to the page that the user is looking at, the electronic device 100 can continue to detect the user's eye gaze point and obtain the position of the user's gaze point. When the user's gaze point position is still in the page, the electronic device 100 continues to play the corresponding content in the page. If the user's gaze point position leaves the page, the electronic device 100 can pause the playback of the content corresponding to the page or exit the playback of the content corresponding to the page.

[0174] In this example, without prior setup in the settings interface, the electronic device 100 can play the content of the page the user is looking at by lightly pinching the stylus 400, which reduces the risk of accidental touches and allows the user to quickly view the content they need.

[0175] In another example of this implementation, Figure 6 In the application scenario shown, the user's trigger operation on the stylus 400 can be: the user's sliding operation on the stylus 400.

[0176] In this example, after the user performs a swipe operation on the stylus 400, the stylus 400 can detect the swipe operation and then send it to the electronic device 100. In response to the swipe operation, the electronic device 100 detects the user's gaze point using the camera 193 to obtain the user's gaze point position. Then, based on the user's gaze point position, the electronic device 100 obtains the interface the user is currently looking at and, based on the swipe operation performed on the stylus 400, displays the interface the user is currently looking at.

[0177] See Figure 13 , Figure 13 This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application, wherein, Figure 13 (a) is the current display interface of electronic device 100. Figure 13 The interface shown in (a) displays a brief introduction to the Beijing Zoo. While browsing this introduction, the user is holding a stylus 400. After the user performs a sliding motion on the stylus 400 (assuming an upward swipe), the stylus 400 sends this upward swipe to the electronic device 100. This triggers the electronic device 100 to detect the user's gaze point, obtain the gaze point position, and then determine the interface the user is currently looking at based on that position. Figure 13 The interface shown in (a) allows the electronic device 100 to respond to an upward swipe operation performed by the user on the stylus 400. Figure 13 (a) The interface is displayed by sliding upwards, and the interface after sliding can be shown as follows: Figure 13 As shown in (b).

[0178] In this example, no prior settings are required in the settings interface. Users can slide their pen on the stylus 400 and the electronic device 100 will display the interface that the user is currently looking at. The operation is simple and convenient.

[0179] In another example of this implementation, Figure 6 In the application scenario shown, the interaction process between the user, the stylus 400, and the electronic device 100 can be as follows: Figure 14 As shown, Figure 14 A flowchart of an operation method for an electronic device, provided as another example of this implementation, may include:

[0180] Step 1401: After detecting the user's trigger operation on the stylus 400, the stylus 400 sends the user's trigger operation to the electronic device 100.

[0181] Among them, the user's trigger operation for the stylus 400 can be: the user's sliding operation on the stylus 400;

[0182] In step 1402, in response to the user's trigger operation on the stylus 400, the electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0183] In this example, after the user makes a swipe operation on the stylus 400, the stylus 400 can detect the swipe operation and then send it to the electronic device 100. In response to the swipe operation, the electronic device 100 detects the user's gaze point through the camera 193 and obtains the user's gaze point position.

[0184] Step 1403: The electronic device 100 obtains the icon that the user is looking at in the current display interface of the electronic device 100 based on the user's gaze point position.

[0185] The currently displayed interface can be the display interface of the target application currently running on the electronic device 100. In this example, the target application can be a text and image editing application, such as a note-taking or drawing application, but this example is not limited to this. The target application can also be other applications, such as video applications. This embodiment does not limit the target application.

[0186] Step 1404: When the distance between the icon being looked at by the user and the aforementioned gaze point is less than or equal to a predetermined distance threshold, the electronic device 100 adjusts the progress of the icon being looked at by the user based on the sliding operation performed by the user on the stylus 400.

[0187] The predetermined distance threshold can be set by the system based on performance and / or implementation requirements during the specific implementation. This embodiment does not limit the size of the predetermined distance threshold.

[0188] Specifically, the electronic device 100 can use the user's gaze point as the center of a circle to obtain the icon on the current display screen that is closest to the gaze point. This icon closest to the gaze point is the icon the user is looking at. After obtaining the icon the user is looking at, the electronic device 100 can further detect the distance between the icon the user is looking at and the gaze point. When the distance between the icon the user is looking at and the gaze point is less than or equal to a predetermined distance threshold, the electronic device 100 can adjust the progress of the icon the user is looking at based on the user's sliding operation on the stylus.

[0189] See also Figure 8 The target application currently running on electronic device 100 is a note-taking application; therefore, the interface currently displayed on electronic device 100 is the interface of the note-taking application, and the user is using stylus 400... Figure 8 Write notes in the interface shown.

[0190] If the stylus 400 is currently using the "ballpoint pen brush" function ( Figure 8 In the text, 81 represents a ballpoint pen brush. After the user slides the stylus 400, the stylus 400 sends this sliding action to the electronic device 100, triggering the electronic device 100 to detect the user's gaze point. The electronic device 100 can then determine the icon the user is currently viewing on the display screen as a "pen brush" based on the user's gaze point location. Figure 8 In the image, 82 represents a "pen brush," and the distance between the "pen brush" icon 82 and the user's gaze point is less than or equal to a predetermined distance threshold. Thus, the electronic device 100 can adjust the thickness of the "pen brush" based on the user's sliding operation on the stylus 400.

[0191] from Figure 8 As can be seen, after the electronic device 100 determines that the distance between the "pen brush" icon 82 and the user's gaze point is less than or equal to a predetermined distance threshold, the "pen brush" icon 82 is selected and the background color is gray.

[0192] Additionally, if the stylus 400 is currently using the "ballpoint pen brush" function ( Figure 8 In the text, 81 represents a ballpoint pen brush. After the user slides the stylus 400, the stylus 400 sends this sliding action to the electronic device 100, triggering the electronic device 100 to detect the user's gaze point. Based on the user's gaze point, the electronic device 100 can determine that the icon the user is currently looking at on the display screen is an eraser. Figure 8 In the image, 83 represents an eraser, and the distance between the eraser icon 83 and the user's gaze point is less than or equal to a predetermined distance threshold. Thus, the electronic device 100 can adjust the size of the eraser based on the user's sliding operation on the stylus 400.

[0193] Referring again to Figure 9(b), the target application currently running on the electronic device 100 is a drawing application. Therefore, the interface currently displayed on the electronic device 100 is the interface of the drawing application, and the user is using the stylus 400 to perform drawing operations on the interface shown in Figure 9(b).

[0194] If the current function of the stylus 400 is "ballpoint pen brush" (93 in Figure 9(b) is ballpoint pen brush), then after the user performs a sliding operation on the stylus 400, the stylus 400 sends the sliding operation performed by the user on the pen to the electronic device 100, thereby triggering the electronic device 100 to detect the user's eye gaze point. Then, the electronic device 100 can obtain the icon 93 of "ballpoint pen brush" that the user is looking at in the current display interface of the electronic device 100 according to the position of the user's gaze point. And the distance between the "ballpoint pen brush" icon 93 and the position of the user's gaze point is less than or equal to a predetermined distance threshold. Thus, the electronic device 100 can adjust the thickness of the "ballpoint pen brush" according to the sliding operation performed by the user on the stylus 400.

[0195] Additionally, if the stylus 400 is currently using the "ballpoint pen brush" function, after the user performs a sliding operation on the stylus 400, the stylus 400 sends the user's sliding operation to the electronic device 100, thereby triggering the electronic device 100 to detect the user's eye gaze point. Then, the electronic device 100 can obtain the icon that the user is looking at in the current display interface of the electronic device 100 as a progress icon 96 based on the user's gaze point position. The progress icon 96 is used to adjust the transparency, and the distance between the progress icon 96 and the user's gaze point position is less than or equal to a predetermined distance threshold. Thus, the electronic device 100 can adjust the transparency based on the user's sliding operation on the stylus 400.

[0196] Furthermore, see Figure 15 The target application currently running on electronic device 100 is a video application; therefore, the interface currently displayed on electronic device 100 is the interface of the video application. Figure 15This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application. In this example, while watching video content played by the electronic device 100, the user holds a stylus 400. After the user performs a sliding operation on the stylus 400, the stylus 400 sends the sliding operation to the electronic device 100, thereby triggering the electronic device 100 to detect the user's eye gaze point. Then, the electronic device 100 can determine that the icon the user is looking at in the current display interface of the electronic device 100 is the playback progress icon 1501, and the distance between the playback progress icon 1501 and the user's gaze point is less than or equal to a predetermined distance threshold. Therefore, the electronic device 100 can adjust the playback progress icon 1501 according to the user's sliding operation on the stylus 400. For example, when the user slides upward on the stylus 400, the playback progress icon 1501 is fast-forwarded; when the user slides downward on the stylus 400, the playback progress icon 1501 is rewound.

[0197] In this example, when the distance between the icon being gazed at by the user and the aforementioned gaze point location is greater than a predetermined distance threshold, the electronic device 100 may not perform any progress-type adjustments on the icon being gazed at by the user.

[0198] In this example, without prior settings in the settings interface, the user can perform a sliding operation on the stylus 400, and the electronic device 100 can adjust the progress of the icon that the user is looking at. The operation is simple and convenient.

[0199] In the above example of this implementation, electronic device 100 can be one of at least two interconnected electronic devices; thus, step 502 can be: when electronic device 100 determines that the user's gaze point is within its own screen range, it executes the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation.

[0200] See Figure 16 , Figure 16 This is a schematic diagram illustrating an application scenario provided by another embodiment of this application. Figure 16 The scenario shown illustrates a multi-stroke operation using the Stylus 400. Figure 16 The scenario shown may include: a stylus 400, a mobile phone 1602, and a tablet computer 1600, wherein the mobile phone 1602 and the tablet computer 1600 are two forms of electronic device 100, respectively.

[0201] Figure 16In this system, the mobile phone 1602 and the stylus 400 can connect via Bluetooth and / or WiFi and other wireless communication methods, and the tablet 1600 and the stylus 400 can also connect via Bluetooth and / or WiFi and other wireless communication methods; in addition, the mobile phone 1602 and the tablet 1600 can also connect via Bluetooth and / or WiFi and other wireless communication methods.

[0202] Figure 16 In the scenario shown, the mobile phone 1602 is used in conjunction with the stylus 400, and the tablet computer 1600 can also be used with the stylus 400. After the user triggers the stylus 400, both the mobile phone 1602 and the tablet computer 1600 detect the user's gaze point. When the tablet computer 1600 detects that the user's gaze point is within the screen area of ​​the tablet computer 1600, it executes the corresponding operation based on the user's gaze point position and the user's trigger operation on the stylus 400. Figure 16 For example, after the user lightly pinches the stylus 400, the stylus 400 sends the pinching action to the mobile phone 1602 and tablet computer 1600, triggering them to detect the user's gaze point. The tablet computer 1600 then determines that the user's gaze point is within its screen range. Based on this gaze point, it identifies the icon the user is currently looking at as the note icon 1601. When the distance between the note icon 1601 and the user's gaze point is less than or equal to a predetermined distance threshold, the tablet computer 1600 displays the interface corresponding to the note icon 1601, i.e., the tablet computer 1600 runs the note-taking application and displays its interface.

[0203] The above example only illustrates the operation of lightly pinching the stylus 400. Figure 16 The operation method of the electronic device in the illustrated scenario has been described. It can be understood that this application... Figures 7-15 The interaction between the stylus 400 and the electronic device 100 mentioned in the illustrated embodiment can all be applied to... Figure 16 The scenario shown will not be elaborated upon further here.

[0204] In the above implementation, after the user triggers the stylus 400, the stylus 400 sends the trigger operation to the electronic device 100. Only then does the electronic device 100 detect the user's eye gaze point, allowing eye-tracking detection to be activated only when needed, thus saving power. Furthermore, in this implementation, after receiving the user's trigger operation on the stylus 400, the electronic device 100 can perform the corresponding operation based on the user's gaze point position, without needing to pre-set it in the settings interface, making it simple and convenient to implement.

[0205] Figure 5 In another implementation of the illustrated embodiment, before step 502, the electronic device 100 may also acquire the user's operation; wherein, the user's operation may include the user's gesture operation.

[0206] In one example of this implementation, step 502 can be: the electronic device 100 obtains the message pop-up that the user is looking at in the current display interface of the electronic device 100 according to the user's gaze point position; after determining that the message pop-up that the user is looking at is selected, the message pop-up that the user is looking at is displayed according to the user's first gesture operation; or, after determining that the message pop-up that the user is looking at is selected, the message pop-up that the user is looking at is hidden according to the user's second gesture operation.

[0207] Figure 17 A flowchart of an operation method for an electronic device is provided as an example of this implementation, such as... Figure 17 As shown, it may include:

[0208] Step 1701: Electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0209] Step 1702, the electronic device 100 acquires the user's operation; wherein, the user's operation may include the user's gesture operation.

[0210] The gesture operation can be any operation performed by the user's finger on any position of the screen of the electronic device 100, and the gesture operation can include a first gesture operation and a second gesture operation.

[0211] Step 1703: The electronic device 100 obtains a message pop-up window indicating that the user is looking at the current display interface of the electronic device 100 based on the user's gaze point position.

[0212] Specifically, see Figure 18 , Figure 18This is a schematic diagram of the current display interface of an electronic device 100 provided in another embodiment of this application. When a user holds the electronic device 100 and a message pop-up appears on the electronic device 100, the user looks at the message pop-up. The electronic device 100 detects the user's eye gaze point, obtains the user's gaze point position, and obtains the message pop-up that the user is looking at in the current display interface of the electronic device 100 as message pop-up 1 based on the user's gaze point position.

[0213] Step 1704: Electronic device 100 detects the duration of the user's gaze at the aforementioned message pop-up. When the duration of the user's gaze at the aforementioned message pop-up is greater than or equal to a predetermined duration threshold, it determines that the message pop-up being gazed at by the user is in a selected state.

[0214] The predetermined duration threshold can be set by the user according to implementation requirements and / or system performance during specific implementation. In this embodiment, the size of the predetermined duration threshold is not limited. In this embodiment, the predetermined duration threshold is referred to as the first duration threshold.

[0215] like Figure 18 As shown, after the electronic device 100 obtains information about the user's gaze at message pop-up window 1 in the current display interface of the electronic device 100, the electronic device 100 detects the duration of the user's gaze at message pop-up window 1. When the duration of the user's gaze at message pop-up window 1 is greater than or equal to a predetermined duration threshold, the electronic device 100 determines that message pop-up window 1 is in a selected state. Figure 18 In the example, message pop-up 1 has a dot, indicating that message pop-up 1 is selected. However, this is only an example. In actual implementation, other methods can be used to indicate that message pop-up 1 is selected.

[0216] Steps 1702 and 1703-1704 can be executed in parallel or sequentially. In this embodiment, the execution order of steps 1702 and 1703-1704 is not limited.

[0217] Step 1705: The electronic device 100 displays a message pop-up window that the user is looking at based on the user's first gesture operation; or, based on the user's second gesture operation, hides the message pop-up window that the user is looking at.

[0218] The first gesture operation can be a user's finger sliding up and down on the screen of the electronic device 100; the second gesture operation can be a user's finger sliding left and right on the screen of the electronic device 100. Of course, the above gesture operations are merely examples, and this embodiment does not limit the gesture operations for displaying and hiding message pop-ups. Furthermore, it should be noted that the first and second gesture operations can be performed by the user's finger at any position on the screen of the electronic device 100, and are not limited to being performed within the display area of ​​the message pop-up that the user is looking at.

[0219] like Figure 18 As shown, after the user's finger slides up and down on the screen of the electronic device 100, the electronic device 100 displays the message pop-up window 1 that the user is looking at. At this time, the message pop-up window 1 can be as shown in 1801.

[0220] In addition, in this embodiment, as Figure 17 As shown, when the electronic device 100 detects that the duration of the user's eye gaze outside the message pop-up window 2 is greater than or equal to the second duration threshold, it can be determined that the message pop-up window 2 is in an unselected state.

[0221] In the above example, the electronic device 100 detects the user's gaze point, obtains the user's gaze point position, and then, based on the user's gaze point position and the user's gesture operation, can display or hide the user's gaze message pop-up. In this example, the user's gesture operation can be performed at any position on the screen of the electronic device 100, which facilitates one-handed operation by the user.

[0222] Figure 19 A flowchart of an operation method for an electronic device, as provided in another example of this implementation, is shown below. Figure 19 As shown, it may include:

[0223] Step 1901: Electronic device 100 detects the user's gaze point and obtains the user's gaze point position.

[0224] Step 1902: Electronic device 100 acquires user operations, including user gesture operations.

[0225] Steps 1901 and 1902 can be executed in parallel or sequentially. This example does not impose any restrictions on the execution order of steps 1901 and 1902.

[0226] Step 1903: The electronic device 100 obtains the content that the user is looking at in the current display interface of the electronic device 100 based on the user's gaze point position.

[0227] In step 1904, the electronic device 100 performs corresponding operations on the content that the user is looking at, based on the user's gesture operation.

[0228] The user's gesture operations can include gesture operations performed by the user within the perception range of the camera of the electronic device 100. Specifically, the user can perform gesture operations on the keyboard area of ​​the electronic device 100 itself, the keyboard area connected to the electronic device 100, the desktop area on which the electronic device 100 is placed, and / or in the air, as long as the gesture operations are performed within the perception range of the camera of the electronic device 100.

[0229] like Figure 19 As shown, when the user's gesture operations are gesture 1, gesture 2, ..., gesture N, the electronic device 100 can perform operation 1, operation 2, ..., operation N on the content that the user is looking at, respectively. In other words, the electronic device 100 can perform different operations on the content that the user is looking at, corresponding to different gestures. Figure 20 This is a schematic diagram of a gesture operation provided in one embodiment of this application. Figure 20 The document shows eight gesture operations. The following section will introduce some of the user's gesture operations as examples.

[0230] In one example, the user's gesture operation can be a two-finger pinch or a two-finger stretch; thus, based on the user's gesture operation, the corresponding operation on the content the user is looking at can be: when the user's gesture operation is a two-finger pinch, the content the user is looking at is zoomed out; when the user's gesture operation is a two-finger stretch, the content the user is looking at is magnified.

[0231] In another example, the user's gesture could be a two-finger swipe up or a two-finger swipe down. Therefore, based on the user's gesture, the corresponding action on the content the user is looking at could be: when the user's gesture is a two-finger swipe up, the content the user is looking at is displayed by swiping upwards; when the user's gesture is a two-finger swipe down, the content the user is looking at is displayed by swiping downwards.

[0232] In another example, the user's gesture operation can be a rotation gesture; thus, according to the user's gesture operation, the corresponding operation on the content the user is looking at can be: according to the rotation direction of the user's gesture, the corresponding rotation operation on the content the user is looking at can be performed.

[0233] In the above example, the electronic device 100 detects the user's gaze point and obtains the user's gaze point position. In addition, the electronic device 100 can obtain the user's gesture operation. Then, based on the user's gaze point position, the electronic device 100 can obtain the content that the user is looking at in the current display interface of the electronic device 100, and perform corresponding operations on the content that the user is looking at based on the user's gesture operation, thereby expanding the gesture interaction function. Moreover, as long as the user performs gesture operation within the perception range of the camera of the electronic device 100, the user can operate on the content that the user is looking at, which brings great convenience to the user's operation.

[0234] In this implementation, the electronic device 100 can be one of at least two interconnected electronic devices. Thus, based on the user's gaze position and the acquired user operation, the operation corresponding to the user's operation can be executed as follows: when the electronic device 100 determines that the user's gaze position is within its own screen range, it executes the operation corresponding to the user's operation based on the user's gaze position and the acquired user operation.

[0235] Figure 21 This is a schematic diagram illustrating an application scenario provided in another embodiment of this application, such as... Figure 21 As shown, 2101 and 2102 are two electronic devices that can connect via wireless communication methods such as Bluetooth and / or WiFi. Figure 21 In the cross-device scenario shown, electronic devices 2101 and 2102 can respectively detect the user's gaze point and obtain the user's gaze point position. When electronic device 2101 determines that the user's gaze point position is within its own screen range, electronic device 2101 executes an operation corresponding to the user's operation based on the user's gaze point position and the obtained user operation, for example: Figure 19 The illustrated embodiment includes operations such as sliding, zooming in, zooming out, and / or rotating. When the user's gaze point shifts to the electronic device 2102, after the electronic device 2102 determines that the user's gaze point is within its own screen range, the electronic device 2102 can perform operations corresponding to the user's operations based on the user's gaze point position and the acquired user operations.

[0236] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and other operations or variations thereof can be performed in the embodiments of this application. Furthermore, the steps may be performed in different orders as presented in the above embodiments, and it is not necessary to perform all the operations in the above embodiments.

[0237] It is understood that, in order to achieve the above-mentioned functions, electronic devices include hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0238] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0239] Figure 22 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. In the case where functional modules are divided according to their respective functions, Figure 22 A schematic diagram of a possible composition of the electronic device 2200 involved in the above embodiments is shown, such as... Figure 22 As shown, the electronic device 2200 may include: a receiving unit 2201, a processing unit 2202, and a transmitting unit 2203;

[0240] The receiving unit 2201 can be used to support the electronic device 2200 in executing steps 702, 1102 and 1402, and / or other processes used in the technical solutions described in the embodiments of this application.

[0241] The processing unit 2202 can be used to support the electronic device 2200 in executing steps 501-502, 702-703, 1102-1104, 1402-1404, 1701-1705 and 1901-1904, and / or other processes used in the technical solutions described in the embodiments of this application;

[0242] The transmitting unit 2203 can be used to support the electronic device 2200 in performing step 704, and / or in other processes of the technical solutions described in the embodiments of this application.

[0243] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0244] The electronic device 2200 provided in this embodiment is used to execute the operation method of the above-described electronic device, and thus can achieve the same effect as the above-described method.

[0245] It should be understood that electronic device 2200 can correspond to Figure 3 The electronic device 100 shown. The functions of the receiving unit 2201 and the transmitting unit 2203 can be determined by… Figure 3The processor 110, antenna 1, and mobile communication module 150 in the electronic device 100 shown are, and / or, implemented by the processor 110, antenna 2, and wireless communication module 160; the function of the processing unit 2202 can be achieved by... Figure 3 The processor 110, camera 193, and touch sensor 180K in the illustrated electronic device 100 are implemented.

[0246] When using integrated units, the electronic device 2200 may include a processing module, a storage module, and a communication module.

[0247] The processing module can be used to control and manage the actions of the electronic device 2200. For example, it can support the electronic device 2200 in executing the steps performed by the receiving unit 2201, processing unit 2202, and sending unit 2203. The storage module can support the electronic device 2200 in storing program code and data. The communication module can support communication between the electronic device 2200 and other devices.

[0248] The processing module can be a processor or controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as radio frequency circuitry, a Bluetooth chip, and / or a Wi-Fi chip.

[0249] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 2200 involved in this embodiment can be a device having... Figure 3 The device with the structure shown.

[0250] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute this application. Figures 5 to 21 The method provided in the illustrated embodiment.

[0251] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to execute this application. Figures 5 to 21 The method provided in the illustrated embodiment.

[0252] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0253] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0254] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0255] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0256] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for operating an electronic device, characterized in that, include: The electronic device detects the user's gaze point and obtains the position of the user's gaze point; Based on the user's gaze point position and the acquired user actions, execute the operation corresponding to the user's actions.

2. The method according to claim 1, characterized in that, Before detecting the user's gaze point and obtaining the user's gaze point position, the method further includes: The system acquires user actions; wherein the user actions include user triggering of a stylus, the stylus being connected to the electronic device; the user actions are detected by the stylus and then sent to the electronic device.

3. The method according to claim 2, characterized in that, The electronic device detects the user's gaze point and obtains the user's gaze point location by including: In response to the user's triggering operation with the stylus, the electronic device detects the user's gaze point and obtains the position of the user's gaze point.

4. The method according to claim 2, characterized in that, The user's triggering operation on the stylus includes: double-clicking the stylus body; The step of executing the operation corresponding to the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the icon that the user is looking at in the current display interface of the electronic device; wherein, the current display interface includes the display interface of the target application currently running on the electronic device, and the target application includes a text and image editing application; When the distance between the icon being gazed at by the user and the gaze point is less than or equal to a predetermined distance threshold, the function corresponding to the icon being gazed at by the user is sent to the stylus, so that the stylus can switch from its current function to the function corresponding to the icon being gazed at by the user.

5. The method according to claim 2, characterized in that, The user's triggering operation on the stylus includes: the user's light pinching operation on the stylus body; The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the icon that the user is looking at in the current display interface of the electronic device; When the distance between the icon being looked at by the user and the location of the gaze point is less than or equal to a predetermined distance threshold, the interface corresponding to the icon being looked at is displayed based on the user's light pinching operation on the stylus body.

6. The method according to claim 2, characterized in that, The user's triggering operation on the stylus includes: the user's light pinching operation on the stylus body; The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the page that the user is looking at in the current display interface of the electronic device; wherein, the current display interface of the electronic device is a video browsing interface or an animated image browsing interface; Based on the user's light pinching action on the stylus, the content corresponding to the page being viewed by the user is played on the page, allowing the user to preview the content.

7. The method according to claim 2, characterized in that, The user's triggering operation on the stylus includes: the user's sliding operation on the stylus body; The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the interface that the user is currently gazing at; Based on the user's swiping operation on the stylus, the interface currently being viewed by the user is displayed in a swiping manner.

8. The method according to claim 2, characterized in that, The user's triggering operation on the stylus includes: the user's sliding operation on the stylus body; The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the icon that the user is looking at in the current display interface of the electronic device; When the distance between the icon being gazed at by the user and the gaze point is less than or equal to a predetermined distance threshold, the progress of the icon being gazed at is adjusted according to the swiping operation performed by the user on the stylus.

9. The method according to claim 1, characterized in that, Before executing the operation corresponding to the user's action based on the user's gaze point position and the acquired user action, the method further includes: The user's actions are obtained; wherein, the user's actions include the user's gesture actions.

10. The method according to claim 9, characterized in that, The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the message pop-up window showing the user's gaze in the current display interface of the electronic device; After determining that the message pop-up being watched by the user is selected, the message pop-up being watched by the user is displayed according to the user's first gesture operation; or, after determining that the message pop-up being watched by the user is selected, the message pop-up being watched by the user is hidden according to the user's second gesture operation.

11. The method according to claim 10, characterized in that, The step of determining that the message pop-up window indicating the user's gaze is selected includes: Detect the duration for which the user gazes at the message pop-up; When the duration for which the user views the message pop-up is greater than or equal to a predetermined duration threshold, it is determined that the message pop-up being viewed by the user is in a selected state.

12. The method according to claim 10, characterized in that, The first gesture operation is the user's finger sliding up and down on the screen of the electronic device; the second gesture operation is the user's finger sliding left and right on the screen of the electronic device.

13. The method according to claim 9, characterized in that, The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: Based on the user's gaze point position, obtain the content that the user is gazing at in the current display interface of the electronic device; Based on the user's gestures, corresponding operations are performed on the content the user is looking at; wherein, the user's gestures include gestures performed by the user within the perception range of the camera of the electronic device.

14. The method according to claim 13, characterized in that, The user's gesture operations include pinching with two fingers or stretching with two fingers. The step of performing corresponding operations on the content being viewed by the user based on the user's gesture operation includes: When the user's gesture is a two-finger pinch operation, the content being viewed by the user is zoomed out. When the user's gesture is a two-finger stretch, the content being viewed by the user is magnified.

15. The method according to claim 13, characterized in that, The user's gesture operations include swiping up with two fingers or swiping down with two fingers; The step of performing corresponding operations on the content being viewed by the user based on the user's gesture operation includes: When the user's gesture is a two-finger swipe up, the content the user is looking at is displayed by swiping upwards; when the user's gesture is a two-finger swipe down, the content the user is looking at is displayed by swiping downwards.

16. The method according to claim 13, characterized in that, The user's gesture operations include rotation gestures; The step of performing corresponding operations on the content being viewed by the user based on the user's gesture operation includes: Based on the rotation direction of the user's gesture, the content being viewed by the user is rotated accordingly.

17. The method according to any one of claims 1-16, characterized in that, The electronic device includes one of at least two interconnected electronic devices; The step of executing the operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation includes: When the electronic device determines that the user's gaze point is within its own screen range, it executes an operation corresponding to the user's operation based on the user's gaze point position and the acquired user operation.

18. An electronic device, characterized in that, include: One or more processors; Memory; Multiple applications; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-17.

Images