Interaction method, electronic device, and system

Abstract

The present application provides an interaction method, an electronic device, and a system. The method comprises: displaying a first interface, wherein the first interface comprises a plurality of objects; when it is detected that a user's gaze is focused on a first position of the display screen, determining a first visual area on the basis of the first position; displaying the first visual area in the first interface, wherein the first visual area comprises at least one object among the plurality of objects, the at least one object comprises a first object, the number of the at least one object is N, the number of the plurality of objects is M, N is smaller than M, and N and M are positive integers; receiving first information sent by a second device; determining, on the basis of the first information, that an operation object is the first object; and executing a first operation on the first object on the basis of the first information. The present application can effectively reduce the fatigue of users' hands when performing interactive operations, and can achieve more precise interactive operations and increase the operation rate, thereby improving the interaction efficiency.

Description

Interaction method, electronic device and system

[0001] This application claims priority to the Chinese patent application with application number 202311508679.0 filed with the State Intellectual Property Office of China on November 13, 2023, and priority to the Chinese patent application with the invention name “An interactive method, electronic device and system”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an interaction method, electronic device, and system. Background Art

[0003] Currently, users can perform a series of operations (such as selecting objects, clicking objects, dragging objects, etc.) on objects displayed on electronic devices through external input devices (such as mice, touchpads, etc.), and can also directly use their fingers to perform a series of operations on objects displayed on electronic devices (such as devices equipped with touch interactive operating systems). When users perform interactive operations in the above manner, there are at least the following disadvantages: it is easy to make the user's hands feel tired, the operation speed is slow, and the interaction efficiency is low.

[0004] Summary of the Invention

[0005] The present application discloses an interaction method, electronic device and system, which can effectively reduce the fatigue of the user's hands when performing interactive operations, and can achieve more precise interactive operations and increase the operation speed, thereby improving the interaction efficiency.

[0006] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to a first device, wherein the first device is configured with a display screen, and the first device is connected to a second device, and the method includes: displaying a first interface, wherein the first interface includes multiple objects; detecting that the user's line of sight is looking at a first position of the display screen, and determining a first visual area based on the first position; displaying the first visual area in the first interface, wherein the first visual area includes at least one object of the multiple objects, the at least one object includes a first object, the number of the at least one object is N, and the number of the multiple objects is M, the N is less than the M, and the N and the M are both positive integers; receiving first information sent by the second device; determining that the operation object is the first object based on the first information; and performing a first operation on the first object based on the first information.

[0007] In the above method, the first device can display a first interface. When it is detected that the user's gaze is on the screen, the first visual area is determined according to the first position of the user's gaze, and the first visual area is highlighted in the first interface, optionally together with at least one object in the first visual area. At this time, the user's operable range is narrowed from the first interface to the first visual area. Then, the first device can receive the first information sent by the second device, and perform corresponding processing on the first object according to the first information. In this way, the user can first "roughly locate" the area to be operated (i.e., the above-mentioned first visual area) by gazing, and then "finely locate" the object to be operated (i.e., the above-mentioned first object) in the area through the electronic device 200. Therefore, the user does not need to move and operate the second device multiple times, which can effectively reduce the fatigue of the user's hands when performing interactive operations, allowing the user to achieve precise interactive operations with less effort, improve the rate during the interactive operation, thereby improving the interaction efficiency, and this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0008] In one possible implementation, the first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and determining the first visual area based on the first position includes: determining the first visual area based on the first position and preset rules corresponding to the first type of area.

[0009] In the above method, the first interface may include at least one type of area. Different types of areas correspond to different preset rules. Therefore, the determined visual areas are also different. This way of determining the visual area is more in line with the user's actual operation and can meet a variety of usage scenarios.

[0010] In one possible implementation, determining the first visual area based on the preset rules corresponding to the first position and the first type of area includes: determining the first visual area with the first position as the center and a preset distance as the radius; or determining the first visual area based on the number of objects within a preset range of the first position.

[0011] In one possible implementation, the first type of area includes a first functional area, and determining the first visual area based on the first position and a preset rule corresponding to the first type of area includes: when the first position is located in the first functional area, setting the first functional area as the first visual area.

[0012] In the above method, when the first type of area is an editable area, the first device can determine a first visual area of ​​a preset size. When the first type of area is a functional area, the first device can use the first functional area corresponding to the first position as the first visual area. This way of determining the visual area is more in line with the user's actual operation and can also meet a variety of usage scenarios.

[0013] In one possible implementation, the first operation is a selection operation; after performing the first operation on the first object according to the first information, the method further includes: when detecting that the position where the user's line of sight is looking at the display screen moves to a second position, determining a second visual area according to the second position; and displaying the second visual area in the first interface.

[0014] In one possible implementation, the first interface includes at least one type of area, the first position is located in the first type of area, and the first type of area is any one of the at least one type of area. The method also includes: determining whether the second type of area where the second position is located is of the same type as the first type of area; when the second type of area and the first type of area are of the same type, the second visual area includes the N first objects, and the N first objects are respectively located at the N positions in the second visual area; when the second type of area and the first type of area are of different types, the second visual area includes at least one functional option.

[0015] In the above method, when the first object is in a movable state, when it is detected that the position where the user's line of sight is looking at the display screen switches from one position to another position of the same type of area, the second visual area can be the visual area during the movement of the first object. The second visual area can indicate that the first object is currently in the process of moving, which is convenient for the user to view and operate. When the first object is in a selected state, when it is detected that the position where the user's line of sight is looking at the display screen moves within the same type of area, the size of the second visual area remains unchanged; when it is detected that the position where the user's line of sight is looking at the display screen switches from one type of area to another type of area, the size of the second visual area changes. This way of determining the visual area is more in line with the user's actual operation and can also meet a variety of usage scenarios.

[0016] In one possible implementation, when the second type area and the first type area are of the same type, after displaying the second visual area in the first interface, the method further includes: receiving second information sent by the second device; determining a third position in the second visual area based on the second information, the third position being any one of the N positions; and displaying the first object at the third position, the third position being different from the position of the first object in the first interface.

[0017] In the above method, when the second visual area indicates that the first object is currently being moved, the first device can move the first object to any position in the second visual area according to the user operation sent by the second device to implement the operation of moving the object, allowing the user to achieve precise interactive operations with less effort, improve the rate of the interactive operation process, and thus improve the interaction efficiency.

[0018] In one possible implementation, when the types of the second type area and the first type area are different, after displaying the second visual area in the first interface, the method also includes: receiving third information sent by the second device; determining that the operation object is a first function option based on the third information, and the first function option is any one of the at least one function options; performing a second operation on the first object based on the third information, and the second operation is the operation indicated by the first function option.

[0019] In the above method, the first device can select the object to be operated and the function options for operating the object through a combination of sight and operation to implement the editing function, which can effectively improve the user's editing efficiency. In addition, this method can meet the interaction needs of different users in different scenarios, expanding the scope of applicable scenarios.

[0020] In one possible implementation, the method of displaying the first visual area in the first interface includes at least one of the following: adding a border to the first visual area, changing the fill color of the first visual area, and enlarging and displaying the at least one object in the first visual area.

[0021] In the above method, by highlighting the first visual area, the user can conveniently view the current operable range, allowing the user to achieve precise interactive operations with less effort, thereby improving the rate of the interactive operation process and thus improving the interaction efficiency.

[0022] In a possible implementation, the size of the first visual area is related to at least one of the following: the size of the first interface, the size of the multiple objects, the number of the at least one object, and the position of the first visual area.

[0023] In a possible implementation, the first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

[0024] In a possible implementation, the first object is any one of the following: an icon, an editing object, or a function option.

[0025] In some examples, icons may include application icons, file icons, shortcut icons, and the like.

[0026] In one possible implementation, the first operation includes a selection operation and a moving display position; after performing the first operation on the first object according to the first information, the method further includes: displaying a second interface, and the position of the first object in the second interface is different from the position of the first object in the first interface.

[0027] In one possible implementation, the first operation includes a selection operation and displaying an interface corresponding to the first object; after performing the first operation on the first object according to the first information, the method further includes: displaying a third interface, which is an interface corresponding to the first object.

[0028] In one possible implementation, the first operation includes a selection operation and displaying a function menu related to the first object; after performing the first operation on the first object according to the first information, the method further includes: displaying a fourth interface, and the fourth interface includes a function menu related to the first object.

[0029] In the above method, precise interactive operations can be achieved by combining determining the user's visual area and operating the second device, or only operating the second device to achieve precise interactive operations, such as moving the display position of the first object, displaying the interface corresponding to the first object, displaying the function menu related to the first object, editing the first object, etc.

[0030] In a possible implementation, the second device is any one of the following: a mouse, a touchpad, and a sensor.

[0031] In a possible implementation, when the first interface is displayed, the area in which the second device acts includes the M objects; when the first visual area is displayed in the first interface, the area in which the second device acts includes the N objects.

[0032] In the above method, when the first interface does not include the first visual area, the user's operable range is the first interface (including M objects); when the first interface includes the first visual area, the user's operable range is the first visual area (including N objects). It can be understood that after the first device determines the first visual area, it can narrow the user's operable range from the first interface to the first visual area, and then carefully locate the object to be operated in the first visual area. Therefore, the user does not need to move and operate the second device multiple times, which can effectively reduce the user's hand fatigue when performing interactive operations, allowing users to achieve precise interactive operations with less energy, thereby improving the speed of the interactive operation process and thus improving the interaction efficiency.

[0033] In one possible implementation, the at least one object includes the first object and the second object, the distance between the position of the first object and the position of the second object in the first visual area is a first distance, and the first object in the first visual area is in a selected state; the method also includes: determining a second distance acting on the second device based on the first information and the first algorithm, the second distance being smaller than the first distance; and switching the selected object in the first visual area to the second object based on the second distance.

[0034] In the above method, a small sliding motion can be used to make the cursor of the first device jump significantly between icons, thereby avoiding the problem of not being able to move the finger to the target position at one time due to the small area of ​​the second device (such as a touchpad), reducing user operations, alleviating user hand fatigue, and improving interaction efficiency.

[0035] In a possible implementation, a connection mode between the first device and the second device includes any one of the following: a wired connection and a wireless connection.

[0036] In a second aspect, an embodiment of the present application provides an interaction method, which is applied to a second device, wherein the second device is connected to a first device, the first device is configured with a display screen, and the operating mode of the second device includes a first mode and a second mode. The method includes: receiving a first message sent by the first device; switching the operating mode of the second device from the first mode to the second mode according to the first message, wherein the first mode is a mode corresponding to the second device when the interface displayed by the first device does not include a visual area, and the second mode is a mode corresponding to the second device when the interface displayed by the first device includes the visual area.

[0037] In the above method, the second device may include a first mode and a second mode, wherein the first mode can be applied to scenarios where only the second device is operated and other scenarios that do not include a visual area, and the second mode can be applied to scenarios where line of sight and operation are combined. When the first device determines the user's visual area, it can send a first message to the second device to enable the second device to switch modes to better adapt to the current usage scenario, improve the rate during the interactive operation, and thus improve the interaction efficiency.

[0038] In a third aspect, the present application provides an electronic device comprising a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the interaction method in any possible implementation of the first aspect.

[0039] In a fourth aspect, the present application provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, wherein the computer program code comprises computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the interaction method in any possible implementation of the first aspect.

[0040] In a fifth aspect, the present application provides an electronic device comprising a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the interaction method in any possible implementation of the second aspect.

[0041] In a sixth aspect, the present application provides a computer storage medium storing a computer program, which, when executed by a processor, implements the interactive method in any possible implementation of any of the above aspects.

[0042] In a seventh aspect, the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the interaction method in any possible implementation of the first aspect above.

[0043] In an eighth aspect, the present application provides an electronic device, the electronic device including a method or apparatus for executing any one of the implementations of the first aspect of the present application. The electronic device is, for example, a chip.

[0044] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, the description of a technical feature, technical solution or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that an embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following is an introduction to the drawings used in this application.

[0046] FIG1 is a schematic diagram of the architecture of an interactive system 10 provided by the present application;

[0047] FIG2 is a schematic diagram of the hardware structure of an electronic device 100 provided in this application;

[0048] FIG3 is a schematic diagram of a software architecture of an electronic device 100 provided in this application;

[0049] Figures 4A-4I, 5A-5B, and 6A-6B are schematic diagrams of some user interfaces provided in this application;

[0050] FIG7 is a flow chart of an interactive method provided by the present application;

[0051] FIG8 is a flow chart of another interactive method provided by the present application;

[0052] FIG9 is a flow chart of another interaction method provided by the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents or, for example, A / B can represent A or B; "and / or" in the text is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0055] Currently, terminal interaction modes include the following two situations:

[0056] In the first scenario, users can use external input devices (such as a mouse, touchpad, etc.) to interact with objects displayed on the terminal. For example, they can move the cursor displayed on the display by moving the mouse, click the left mouse button to select an object, click the right mouse button to trigger the display of a function menu, etc. Another example is sliding a finger on the touchpad to move the displayed cursor, clicking the touchpad to select an object, and long pressing the touchpad to trigger the display of a function menu, etc. When interacting with a mouse, users need to repeatedly move the mouse and click the left and right mouse buttons. The number of movements is large, and the amplitude of each movement varies. This operation is cumbersome, and the user's hand is easily fatigued. The interaction rate is also slow, resulting in low interaction efficiency. Interaction via the touchpad also requires repeated sliding of the finger, and is limited by the size of the touchpad. When the actual distance the cursor needs to be moved (for example, the distance between icon A and icon B on the display) is greater than the longest distance on the touchpad, the finger needs to be moved multiple times. In addition, for precise interactive operations, such as moving the position of icon C or selecting a certain text, the fingers cannot be operated flexibly, which easily leads to fatigue and a slow interaction rate, resulting in low interaction efficiency.

[0057] Case 2: For devices equipped with a touch-screen interactive operating system, users can directly use their fingers to interact with objects displayed on the terminal, such as clicking an object on the screen to select it, long-pressing an object on the screen to trigger functions related to the object, etc. In this way, users need to hover their fingers, slide their fingers, click their fingers, etc. many times, and because the contact area between the finger and the screen is related to the thickness of the finger, users with thick fingers are prone to accidental touches, making it difficult to achieve accurate interactive operations. In addition, for devices with larger screen sizes (for example, greater than or equal to 13 inches), users need to raise their arms and operate with their fingers. Because the icons in their interface are larger in size and spacing, the distance and amplitude of arm and finger movement also increase, which can easily cause users to quickly feel fatigued, and the interaction rate is also slow, resulting in low interaction efficiency.

[0058] Furthermore, in all of the above interaction methods, the user first focuses their gaze on the object to be interacted with, and then performs an interaction operation on the object, which further increases the operation time and thus has low interaction efficiency.

[0059] An embodiment of the present application provides an interaction method, which is applied to an interactive system. The interactive system may include an electronic device 100 and an electronic device 200. The electronic device 100 and the electronic device 200 are connected. The user can control the electronic device 100 by operating the electronic device 200. The electronic device 200 can be understood as an external input device of the electronic device 100. The electronic device 100 can display a first interface. When it is detected that the user's line of sight is fixed on the screen, an area where the user's line of sight is fixed (which may be referred to as a visual area) is determined according to the position of the user's line of sight (which may be referred to as the line of sight position). The electronic device 100 can highlight the visual area in the first interface, and optionally at least one object in the visual area. At this time, the user's operable range is narrowed from the first interface to the visual area. Then, the electronic device 100 can obtain the user operation on the first object in the visual area through the electronic device 200, and perform corresponding processing on the first object according to the user operation, for example, moving the position, displaying a related function menu, etc.

[0060] In the above method, the user can first “roughly locate” the area to be operated (i.e., the above-mentioned visual area) by gazing at the area, and then “finely locate” the object to be operated in the area (i.e., the above-mentioned first object) through the electronic device 200. Therefore, the user does not need to move and operate the electronic device 200 multiple times, and can avoid the situation where the hand is easily fatigued due to many movements, cumbersome operations, and long operation time, thereby solving the problem of interacting through the mouse in the above-mentioned situation one. In addition, the user does not need to slide his fingers and accurately select objects on the touchpad or display screen multiple times, thereby solving the problem of interacting through the touchpad in the above-mentioned situation one, and also solving the problem of interacting through the touchpad in the above-mentioned situation two. Through the method of the present application, the fatigue of the user's hands when performing interactive operations can be effectively reduced, allowing the user to achieve precise interactive operations with less energy, improve the rate during the interactive operation, and thus improve the interaction efficiency. In addition, this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0061] The following introduces an interactive system 10 involved in an embodiment of the present application.

[0062] FIG1 exemplarily shows a schematic diagram of the architecture of an interactive system 10 .

[0063] As shown in FIG1 , the interactive system 10 may include an electronic device 100 and an electronic device 200. The electronic device 100 may communicate with the electronic device 200 via wired (e.g., universal serial bus (USB), twisted pair, coaxial cable, and optical fiber) and / or wireless (e.g., wireless local area network (WLAN), Bluetooth, and cellular communication network), and the user may control the electronic device 100 by operating the electronic device 200.

[0064] The electronic device 100 can be a mobile phone, a tablet computer, an all-in-one computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), as well as smart home devices such as smart TVs and projectors, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), or in-vehicle devices. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0065] In one embodiment, the electronic device 100 can be configured with a physical display screen, such as a laptop computer display screen, a tablet computer display screen, or a smart TV display screen. It can also be connected to an external display screen (such as a desktop monitor) and display through the external display screen. It can also be configured with a virtual display screen, such as an AR virtual display screen, a VR virtual display screen, or an MR virtual display screen.

[0066] The electronic device 200 can be understood as an external input device of the electronic device 100, such as a mechanical mouse, an optical mouse, a laser mouse, a floating mouse, a touch mouse, a touchpad (built-in touchpad, external touchpad), a keyboard, a handle, a gesture sensor, a microphone, a handwriting tablet, etc. The electronic device 200 can also be an external input device of other forms. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device 200.

[0067] In one embodiment, the electronic device 200 may receive a user operation and send information about the user operation to the electronic device 100 . The electronic device 100 may execute a corresponding processing flow in response to the user operation.

[0068] It can be understood that the number and form of the electronic devices shown in Figure 1 are only examples. In specific implementations, the number may be more or less. For example, the interactive system 10 may not include the electronic device 200, that is, the electronic device 200 may be integrated on the electronic device 100, such as a laptop computer with a touchpad.

[0069] Next, the structure of an exemplary electronic device provided by an embodiment of the present application is introduced.

[0070] FIG2 exemplarily shows a schematic diagram of the hardware structure of an electronic device 100 .

[0071] As shown in FIG2 , the electronic device 100 may include a processor 110, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a distance sensor 180F, a proximity light sensor 180G, etc.

[0072] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0074] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0075] Processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0076] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0077] The mobile communication module 150 can provide wireless communication solutions applied to the electronic device 100, including second generation mobile communication technology (2G) / third generation mobile communication technology (3G) / fourth generation mobile communication technology (4G) / fifth generation mobile communication technology (5G) / sixth generation mobile communication technology (6G).

[0078] The wireless communication module 160 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0079] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0080] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In one embodiment, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0081] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0082] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In one embodiment, the ISP can be located within camera 193.

[0083] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. In one embodiment, the electronic device 100 can capture an image of the user through the camera 193, and obtain the position of the user's gaze at the display screen 194 based on the image.

[0084] Distance sensor 180F is used to measure distance. Electronic device 100 can measure distance using infrared or laser technology. In one embodiment, when capturing a scene, electronic device 100 can use distance sensor 180F to measure distance for rapid focusing. In another embodiment, electronic device 100 can use distance sensor 180F to measure the distance between the user's eyes and display screen 194 to determine the position of the user's gaze on display screen 194.

[0085] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0086] The embodiment of the present application does not limit the module in the electronic device 100 for obtaining the position where the user's gaze is directed at the display screen 194.

[0087] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. For example, the software system with a layered architecture can be an Android system, a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0088] FIG3 exemplarily shows a schematic diagram of a software architecture of an electronic device 100 .

[0089] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0090] The application layer can include a series of application packages.

[0091] As shown in Figure 3, the application package may include applications such as camera, gallery, music, calendar, short message, call, navigation, Bluetooth, browser, and screen projection.

[0092] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0093] As shown in FIG3 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0094] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0095] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0096] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0097] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0098] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0099] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0100] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0101] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0102] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0104] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0105] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0106] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0107] A 2D graphics engine is a drawing engine for 2D drawings.

[0108] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0109] The following describes an application scenario involved in an embodiment of the present application and a user interface diagram for the scenario.

[0110] Scenario 1: Electronic device 100 displays the desktop. For a specific example, see Figure 4A. When the user's gaze is detected at the display screen, electronic device 100 can highlight the user's visual area, which can include at least one icon on the desktop. For a specific example, see Figure 4B. The user can operate electronic device 200 to select any icon within the visual area. For a specific example, see Figure 4C. When the user's gaze position on the display screen moves, the selected icon can be moved (for example, to a gallery application). For specific examples, see Figures 4E-4F.

[0111] As shown in FIG4A , the electronic device 100 may display a user interface 410, which is the desktop of the electronic device 100. The user interface 410 may include icons for multiple applications, such as an icon 411 for a gallery application, an icon 412 for a clock application, an icon 413 for a browser application, an icon 414 for a contact application, and icons for other applications. Without limitation, the desktop of the electronic device 100 may also include other icons, such as icons for files and shortcuts.

[0112] As shown in FIG4B , when user 300 is looking at the display screen (assuming user 300's line of sight is focused on point B), electronic device 100 can determine the point of focus of user 300's line of sight (which may be referred to as the line of sight position, i.e., point B) based on user 300's eye features (e.g., the relative positions of the pupil and Purkinje's spot) acquired by the front camera and the distance between user 300's eye and the display screen measured by the distance sensor. Next, electronic device 100 can determine user 300's visual area (e.g., a square area located in the upper left corner of the display screen, centered at point B and with a preset distance as a radius) based on user 300's line of sight position (point B) and the user interface currently displayed on the display screen. The electronic device 100 can display a user interface 420 based on a determined visual area. The user interface 420 can be the desktop of the electronic device 100. The user interface 420 is similar to the user interface 410 shown in FIG4A , except that the user interface 420 includes an area frame 421. The area frame 421 can represent the visual area of ​​the user 300. The icons 411, 412, 413, and 414 in the area frame 421 are all enlarged, which can represent that these icons are in a state that can be operated (e.g., selected, clicked). Without limitation, the method of highlighting the area frame 421 can include, but is not limited to: adding a border to the area frame 421, changing the fill color of the area frame 421 (different from the fill color of other areas in the user interface 420), etc.

[0113] In one embodiment, the position of the area frame 421 can be moved: when the electronic device 100 obtains a change in the user's line of sight position (for example, when the line of sight position moves from within the visual area to outside the visual area), the position of the visual area can be moved accordingly, that is, the position of the area frame 421 moves accordingly, and the object within the area frame 421 can also change. For example, when the line of sight position changes (assuming it moves from point B on the display screen to the middle of the camera application icon and the file management application icon outside the visual area), the user interface displayed by the electronic device 100 is similar to the user interface 420 shown in Figure 4B, except that the area frame 421 in the user interface does not include icons 411, 412, 413, and 414, but includes the camera application icon and the file management application icon. In another embodiment, the position of the area frame 421 may not move: when the electronic device 100 obtains a change in the user's line of sight position within the visual area, the position of the visual area may not move. For example, when the line of sight position moves from point B on the display screen to object A, the user interface displayed by the electronic device 100 is the same as the user interface 420 shown in Figure 4B.

[0114] In one embodiment, the user 300 can operate the electronic device 200 so that the cursor of the electronic device 100 can move within the visual area of ​​the user 300 according to the operation of the user 300 .

[0115] In one embodiment, when the electronic device 100 displays the user interface 420 shown in Figure 4B, when it is detected that the position of the user 300's gaze on the display screen moves within the visual area, for example, as shown in Figure 4C, when the user 300's gaze position moves from point B on the display screen to object A, the visual area displayed by the electronic device 100 remains unchanged. At this time, the user 300 can operate the electronic device 200 to select any object in the area box 421 (assuming it is icon 411). The electronic device 200 is, for example, a touch pad as shown in Figure 4C. The user 300 operation is, for example, a touch operation on object A (for example, corresponding to icon 411) in the visual area as shown in Figure 4C. The touch operation is, for example, a sliding operation to the upper left. After receiving the user 300 operation, the electronic device 200 can send information about the user 300 operation to the electronic device 100. The electronic device 100 can determine the object (icon 411) selected by the user 300 based on the received information about the user 300 operation, and display the user interface 430 shown in Figure 4C. User interface 430 is similar to user interface 420 shown in FIG. 4B , except that in user interface 430, icon 411 is selected. This selection is indicated by changing the background color of icon 411, thereby indicating that the gallery application has been selected. Without limitation, other methods for indicating an object's selection include, but are not limited to, adding a border to the object, enlarging the object, and the like.

[0116] FIG4C illustrates the electronic device 200 with a square touch panel as an example. For a specific example, please refer to FIG4D .

[0117] In some examples, FIG4D (A) exemplarily illustrates a click operation method for a touchpad. As shown in FIG4D (A), the electronic device 200 can be divided into four areas: area 1, area 2, area 3, and area 4. Different areas can correspond to different icons in the visual area. For example, area 1, area 2, area 3, and area 4 correspond to icons 411, 412, 413, and 414 in the area box 421 in the user interface 430 shown in FIG4C, respectively. The electronic device 200 can receive a user's touch operation (e.g., a single-click operation) on any area (assuming it is area 1) and send the touch operation to the electronic device 100. In one embodiment, the electronic device 100 can move the cursor of the electronic device 100 to the location of the icon 411 corresponding to area 1 according to the received touch operation. In another embodiment, the electronic device 100 can select the icon 411 corresponding to area 1 according to the received touch operation.

[0118] Without limitation, in other examples, FIG4D(B) exemplarily illustrates a sliding operation method for a touchpad. As shown in FIG4D(B), the electronic device 200 can receive sliding operations in different directions, such as, but not limited to, sliding up, sliding down, sliding to the left, sliding to the right, sliding to the upper left, sliding to the lower left, sliding to the upper right, sliding to the lower right, etc. In one embodiment, for sliding operations in different directions on the electronic device 200, the sliding operation moves a distance of 1, which can be used to move the cursor of the electronic device 100 in different directions, and the distance moved by the cursor is also a distance of 1. For example, if a user slides a finger to the upper left on the electronic device 200, the user can control the electronic device 100 to move the cursor to the upper left within the visual area. In another embodiment, for sliding operations in different directions of the electronic device 200, the distance moved by the sliding operation is distance 2, which can be used to select objects in different directions, wherein distance 1 is greater than distance 2. For example, when the user slides a finger to the upper left on the electronic device 200 (for example, from point C to point D shown in (B) of Figure 4D), the electronic device 100 can be controlled to select the upper left object in the visual area (for example, the upper left icon 411 in the area box 421 in the user interface 430 shown in Figure 4C). This operation method can make the cursor jump significantly between icons by sliding a small amount, avoiding the problem of not being able to use the finger to move to the target position at one time due to the small area of ​​the touchpad, reducing user operations, alleviating user hand fatigue, and improving interaction efficiency.

[0119] The above example is not limited to the case where four objects are included in the visual area (in this case, the touchpad can be divided according to the area shown in (A) of Figure 4D). In other examples, when more or fewer objects are included in the visual area, the area of ​​the touchpad will also change accordingly. For example, when the visual area includes nine objects (assuming that the visual area includes icon 411, icon 412, icon 431, icon 413, icon 414, icon 432, icon 433, icon 434, and icon 435 in the user interface 430 shown in Figure 4C), the touchpad can be divided according to the area shown in (C) of Figure 4D, that is, the electronic device 200 can be divided into nine areas: area 1-area 9. These nine areas can respectively correspond to the nine icons in the user interface 430 shown in Figure 4C, and will not be repeated.

[0120] The shape of the touchpad is not limited to the above-mentioned square touchpad. The shape of the touchpad can also be, but not limited to, circular, rectangular, triangular, or other polygonal shapes. The present application does not limit the shape of the touchpad.

[0121] In one embodiment, the user 300 can operate the electronic device 200 to change the size of the visual area. For example, when the electronic device 100 displays the user interface 420 shown in Figure 4B, the electronic device 200 is, for example, a touch panel as shown in Figure 4I, and the user 300 operation is, for example, a touch operation as shown in Figure 4I, and the touch operation is, for example, a pinch operation. After the electronic device 200 receives the operation of the user 300, it can send information about the operation of the user 300 to the electronic device 100. The electronic device 100 can adjust the size of the visual area according to the received information about the operation of the user 300, such as increasing the size of the visual area, and display the user interface 480 shown in Figure 4I. User interface 480 is similar to user interface 420 shown in Figure 4B, except that the size of area box 481 in user interface 480 is larger than the size of area box 421 in user interface 420. Area box 481 in user interface 480 may include icon 411, icon 412, icon 431, icon 413, icon 414, icon 432, icon 433, icon 434, and icon 435. These icons are all enlarged, which can indicate that these icons are in a state that can be operated (for example, selected or clicked).

[0122] In one embodiment, when user 300 selects object A (corresponding to icon 411) (for example, when electronic device 100 displays user interface 430 shown in FIG4C as having selected icon 411), user 300 can operate electronic device 200 to place selected object A in a state where it can be moved (hereinafter referred to as a movable state). User 300's operation, for example, is a touch operation on object A, such as a long press operation. After receiving the user 300 operation, electronic device 200 can send information about the user 300 operation to electronic device 100. Electronic device 100 can set the state of object A to a movable state based on the received information about the user 300 operation. At this time, electronic device 100 can display a user interface similar to user interface 430 shown in FIG4C, except that object A in this user interface is in a movable state, and the movable state can be displayed by shaking object A. Without limitation, the display method for an object in a movable state may include, but is not limited to, adding a background color to the object, adding a border to the object, and enlarging the display of the object. The display method for the selected state is different from the display method for the movable state.

[0123] In one embodiment, after object A is in a state where it can be moved, when it is detected that the position where the user 300's line of sight is looking at the display screen moves, for example, as shown in Figure 4E, the line of sight of the user 300 moves from object A on the display screen to point E. At this time, the visual area displayed by the electronic device 100 can move accordingly, and the position of the visual area is determined according to the position where the user 300's line of sight is looking. In the process of moving the visual area, the electronic device 100 can display object A at different positions within the visual area, and it can be characterized that these positions can all be used as the actual display position of object A (i.e., the display position after the move). When the user 300's line of sight moves to point E on the display screen, the electronic device 100 can display the user interface 440 shown in Figure 4E.

[0124] As shown in FIG4E , user interface 440 is similar to user interface 430 shown in FIG4C , except that area box 421 in user interface 430 is located in the upper left corner, while area box 441 in user interface 440 is located in the lower right corner. Area box 441 in user interface 440 includes four icons 442. Icons 442 are similar to icon 411, except that the word "Gallery" is not displayed below icon 442. Icon 442 is in a moving state, which can indicate that the gallery application is being moved. The positions of these four icons 442 are different, and icon 411 can be moved to the position of any of the four icons 442.

[0125] In one embodiment, when the electronic device 100 displays the user interface 440 shown in Figure 4E, the line of sight of the user 300 may move within the visual area. For example, as shown in Figure 4F, when the line of sight of the user 300 moves from point E on the display screen to the icon in the upper right corner of the visual area (corresponding to icon 411), the visual area displayed by the electronic device 100 remains unchanged. At this time, the user 300 can operate the electronic device 200 to select any position in the area box 441 (assuming it is the position of icon 442). The electronic device 200 is, for example, a touch pad as shown in Figure 4F. The operation of the user 300 is, for example, a touch operation on position 1 (for example, corresponding to the upper right position) within the visual area as shown in Figure 4F. The touch operation is, for example, a sliding operation to the upper right. After receiving the operation of the user 300, the electronic device 200 can send information about the operation of the user 300 to the electronic device 100. The electronic device 100 can determine the position selected by the user 300 (the upper right position) based on the received information about the operation of the user 300, and display the user interface 450 shown in Figure 4F. The user interface 450 is similar to the user interface 430 shown in FIG. 4C , except that the icon 411 in the user interface 430 is located at the upper left position, while the icon 411 in the user interface 450 is located at the lower right position.

[0126] In one embodiment, after the embodiment shown in Figure 4F, the user can operate the electronic device 200 to select other positions in the visual area except the position of the selected object, so that the electronic device 100 deselects the selected object. For example, the electronic device 100 can receive a user operation (such as a touch operation, such as a left sliding operation) for a position other than the position of the icon 411 in the area box 451 in the user interface 450 through the electronic device 200, and deselect the selected object (icon 411) according to the received user operation. At this time, the electronic device 100 can display a user interface similar to the user interface 450 shown in Figure 4F, except that the icon 411 in the user interface is in an unselected state. Without limitation, in another embodiment, the user can move the position of the display screen to which the user is directed. For example, it is detected that the user's line of sight moves from the lower right position (corresponding to the area box 441) in the user interface 450 shown in Figure 4F to the lower left position. At this time, the visual area displayed by the electronic device 100 can move accordingly. Next, the user can operate the electronic device 200 to select any position within the moved visual area so that the electronic device 100 deselects the selected object. For specific instructions, please refer to the above-mentioned instructions on the user operating the electronic device 200 to select other positions within the visual area except the position of the selected object, which will not be repeated here.

[0127] Not limited to the case where the visual area after the user's line of sight moves is a blank area in the above example, in another embodiment, after the embodiment shown in Figure 4C, it is detected that the position where the user 300's line of sight is looking at the display screen has moved. For example, as shown in Figure 4G, when the user 300's line of sight moves from object A on the display screen to point F, the electronic device 100 can display the user interface 460 shown in Figure 4G. The user interface 460 is similar to the user interface 430 shown in Figure 4C, with the difference that the area box 461 in the user interface 460 is in the upper right position. The area box 461 includes four icons 462. The icons 462 are similar to the icon 411, with the difference that the word "Gallery" is not displayed below the icon 462. The icon 462 is in a moving state, which can indicate that the gallery application is being moved. The positions of these four icons 462 are respectively the positions of the game center application, weather application, housekeeper application, and settings application in the user interface 430. The icon 411 can be moved to the position of any one of the four icons 462.

[0128] In one embodiment, when the electronic device 100 displays the user interface 460 shown in Figure 4G, the user 300 can operate the electronic device 200 to select any position in the area box 461 (assuming it is the position of the icon 462). For example, the electronic device 100 can receive a user operation on the position of the icon 462 in the area box 461 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, an operation of sliding to the lower right), determine the position selected by the user (the lower right position) based on the user operation, and display the user interface 470 shown in Figure 4H. User interface 470 is similar to user interface 430 shown in FIG4C , except that icon 411 in user interface 470 is located at the position of the settings app in user interface 430, the settings app in user interface 470 is located at the position of the video app in user interface 430, the video app in user interface 470 is located at the position of the camera app in user interface 430, the camera app in user interface 470 is located at the position of the file management app in user interface 430, the file management app in user interface 470 is located at the position of the calendar app in user interface 430, and the position of the calendar app in user interface 470 is different from the position of the calendar app in user interface 430. It can be understood that when the gallery app moves to the original position of the settings app, the positions of the settings app and other apps (video app, camera app, file management app, calendar app) located after the settings app are moved backward in sequence, wherein the order of the multiple apps can be from top to bottom and from left to right.

[0129] Without limitation, in other examples, when the Gallery app is moved to the original location of the Settings app, the Settings app can be moved to the location before the Gallery app was moved (the original location), that is, the positions of the Gallery app and the Settings app are swapped. Without limitation, in other examples, when the Gallery app is moved to the original location of the Settings app, the Gallery app and the Settings app can be in the same location (i.e., the original location of the Settings app), and the Gallery app and the Settings app can be in the same folder.

[0130] Not limited to the case of moving the selected object as in the above example, in other examples, after the implementation shown in Figure 4C, the user can operate the electronic device 200, such as a touch operation on the selected object (the touch operation is, for example, a double-click operation), so that the electronic device 100 performs a corresponding operation on the selected object according to the user operation, such as starting the application corresponding to the object, or displaying the function menu corresponding to the object.

[0131] Scenario 2: Electronic device 100 displays an editing interface, which may include multiple different areas. When detecting that the user is looking at the display screen, electronic device 100 may highlight the user's visual area, which may include at least one editable object in the editing area of ​​the editing interface. For a specific example, see (A) in FIG5A. The user may operate electronic device 200 to select an editable object in the visual area (e.g., select text content). For a specific example, see (B) in FIG5A.

[0132] As shown in FIG5A(A), electronic device 100 may display user interface 510, which may be an editing interface for a first application. User interface 510 may include a toolbar 511, a function area 512, a view area 513, and an editing area 514. Toolbar 511 may include multiple tool options, such as "File," "Start," and "Insert." The "Start" option in user interface 510 is selected. Function area 512 may display multiple function areas under the "Start" option, such as a clipboard function area, a slide function area, a font function area 5121, a paragraph function area, a drawing function area, and an editing function area. The functions contained in these multiple function areas may be used to perform corresponding editing operations on the content displayed on page 5141 in editing area 514. View area 513 may include slide 5131, which is selected. Editing area 514 may be used to edit the selected slide 5131. Page 5141 is an editable page of slide 5131. The user interface 510 also includes an area box 515, which can represent the user's visual area (assuming that the current user's line of sight is located at the character ":"). The area box 515 includes text content 5151, and the text content 5151 displays the characters "Reporter: Xiao Wang".

[0133] In one embodiment, when it is detected that the position where the user's line of sight gazes at the display screen moves in the editing area 514 / page 5141 (for example, when the position of the user's line of sight moves from within the visual area in the editing area 514 / page 5141 to outside the visual area), the position of the visual area can move correspondingly, that is, the position of the area frame 515 moves correspondingly in the editing area 514 / page 5141, and the objects within the area frame 515 can also change. For example, when the line of sight position changes (assuming it moves from the character ":" in the area frame 515 to the left eye of the image "kitten" on the page 5141), the user interface displayed by the electronic device 100 is similar to the user interface 510 shown in (A) of FIG. 5A. The difference is that at this time, the area frame 515 in the user interface does not include the text content 5151, but includes the image "kitten". In another embodiment, when it is detected that the position of the user's line of sight changes within the visual area, the position of the visual area may not move. For example, when the line of sight position moves from the character ":" in the area frame 515 to the character "king" in the area frame 515, the user interface displayed by the electronic device 100 is the same as the user interface 510 shown in (A) of FIG. 5A.

[0134] In one embodiment, the electronic device 100 can receive a user operation on the text content 5151 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, a double-click operation), determine the selected object (text content 5151) according to the received user operation, and display the user interface 520 shown in (B) of FIG. 5A. The user interface 520 is similar to the user interface 510 shown in (A) of FIG. 5A. The difference is that the text content 5151 in the user interface 520 is in a selected state, and the display mode of the selected state is to change the background color of the text content 5151.

[0135] In one embodiment, when the electronic device 100 displays the user interface 520 shown in (B) of FIG5A, the electronic device 100 can receive a user operation on the text content 5151 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, a long press operation), and set the text content 5151 to a movable state according to the received user operation. Then, when it is detected that the position of the user's gaze on the display screen moves in the editing area 514 / page 5141, for example, the user's gaze position moves from the character ":" in the area box 515 in the page 5141 to position 2 outside the area box 515 in the page 5141 (assuming that position 2 is a point in the upper right blank area of ​​the page 5141), at this time, the visual area displayed by the electronic device 100 can move accordingly, and the position of the visual area is determined according to the position of the user's gaze. During the movement of the visual area, the electronic device 100 can display the selected text content 5151 in the visual area. When the user's line of sight moves to position 2 within page 5141, the electronic device 100 can display a user interface similar to the user interface 520 shown in (B) of Figure 5A, except that the area box 515 (and text content 5151) in the user interface 520 is in the lower middle position, while the area box 515 (and text content 5151) in this user interface is in the upper right position.

[0136] Not limited to the above-mentioned example of moving the editable objects in the editing area 514 / page 5141 by combining determining the user's visual area and operating the electronic device 200, in another embodiment, only the electronic device 200 can be operated to achieve the movement of the editable objects in the editing area 514 / page 5141 and other refined operations (such as modifying / editing the editable objects).

[0137] Scenario 3: After (A) of Figure 5A shown in Scenario 2, the user can operate the electronic device 200 so that the cursor of the electronic device 100 can move within the visual area according to the user's operation. For specific examples, please refer to (A)-(B) of Figure 5B.

[0138] Exemplarily, when electronic device 100 displays user interface 510 shown in FIG. 5A (A), the user may receive a user operation (such as a touch operation, such as a single-click operation) through electronic device 200, determine the position of the cursor in the visual area based on the received user operation, and display user interface 560 shown in FIG. 5B (A). User interface 560 is similar to user interface 510 shown in FIG. 5A (A), except that a cursor 561 is displayed between the characters "Reporter:" and "Xiao Wang" displayed in text content 5151 in area frame 515 of user interface 560. Cursor 561 can be used to indicate an insertion point for text editing.

[0139] In one embodiment, after the embodiment shown in FIG5B(A), the user may receive a user operation (such as a touch operation, such as a rightward sliding operation) on the cursor 561 in the user interface 560 shown in FIG5B(A) via the electronic device 200, and control the cursor in the visual area to move rightward according to the received user operation, at which point the user interface 570 shown in FIG5B(B) may be displayed. The user interface 570 is similar to the user interface 560, except that the cursor 571 in the user interface 570 is located to the right of the displayed characters "Reporter: Xiao Wang".

[0140] Scenario 4: After (A) and (B) of FIG. 5A shown in Scenario 2, when it is detected that the position of the user's gaze on the display screen moves from the editing area 514 / page 5141 to the function area 512, the electronic device 100 can move the position of the visual area accordingly, and the objects in the visual area can also change. At this time, the moved visual area can include at least one editing function option in the editing interface. For specific examples, see (A) of FIG. 6A. The user can operate the electronic device 200 to select any function option in the moved visual area to perform an editing function operation on the selected editable object (for example, underlining text content). For specific examples, see (B) to (C) of FIG. 6A.

[0141] Exemplarily, when the electronic device 100 detects that the position where the user's gaze is looking at the display screen moves from the character ":" in page 5141 to the character "U" in the font function area 5121 in the user interface 510 shown in (A) of Figure 5A, the electronic device 100 can determine the user's visual area as the area where the font function area 5121 is located. At this time, the user interface 530 shown in (A) of Figure 6A can be displayed. The user interface 530 is similar to the user interface 520 shown in (B) of Figure 5A, except that the area box 531 in the user interface 530 is located at the position where the font function area 5121 is located. The area box 531 in the user interface 530 includes multiple editing function controls, such as control 532 and other controls. Control 532 can be used to underline the selected text content. These multiple editing function controls are all enlarged, which can indicate that these controls are in a state that can be operated (such as selected, clicked).

[0142] In one embodiment, the electronic device 100 can receive a user operation on the control 532 (the user operation is, for example, a touch operation, such as a left swipe operation) through the electronic device 200, determine the selected editing function option (control 532) based on the received user operation, and display the user interface 540 shown in FIG6A(B). The user interface 540 is similar to the user interface 530 shown in FIG6A(A), except that in the user interface 540, the control 532 is selected.

[0143] In one embodiment, when the electronic device 100 displays the user interface 540 shown in FIG6A(B), the electronic device 100 may receive a user operation (such as a touch operation, such as a single-click operation) through the electronic device 200, perform an editing function (such as a function corresponding to the control 532) on the selected editable object (such as the text content 5151) according to the received user operation, and display the user interface 550 shown in FIG6A(C). The user interface 550 is similar to the user interface 540 shown in FIG6A(B), except that the text content 5151 is underlined in the user interface 550.

[0144] Without being limited to the above embodiment, in another embodiment, when it is detected that the position where the user's gaze is directed at the display screen moves from the editing area 514 / page 5141 to the viewing area 513, the visual area after the movement may include at least one slide in the editing interface. In one embodiment, the electronic device 100 may receive a user operation on slide 1 through the electronic device 200 (the user operation is, for example, a touch operation, such as an upward sliding operation), and determine the selected slide (slide 1) based on the received user operation. At this time, the electronic device 100 may display a user interface in which slide 1 is selected. Then, the electronic device 100 may receive a user operation again through the electronic device 200 (the user operation is, for example, a touch operation, such as a single-click operation), and set the slide displayed in the editing area 514 to slide 1 based on the user operation.

[0145] Without being limited to the above embodiment, in another embodiment, when it is detected that the position where the user's gaze is looking at the display screen moves from the editing area 514 / page 5141 to the toolbar 511, the visual area after the move may include at least one tool option in the editing interface. In one embodiment, the electronic device 100 may receive a user operation (such as a touch operation, such as a right sliding operation) for tool option 1 (such as the "Insert" option) through the electronic device 200, and determine the selected tool option 1 ("Insert" option) based on the received user operation. At this time, the electronic device 100 may display a user interface in which tool option 1 is selected. Then, the electronic device 100 may receive a user operation (such as a touch operation, such as a single-click operation) through the electronic device 200 again, and set the content displayed in the function area 512 to the multiple functions included in tool option 1 based on the user operation.

[0146] In one embodiment, the user's visual area can be an area determined according to preset rules based on the user's line of sight position and the user interface displayed on the current display screen. In some examples, when it is detected that the user's line of sight position is located in an editable area in the user interface displayed on the current display screen, the electronic device 100 can determine the visual area with the line of sight position as the center and a preset distance as the radius, for example, the method of determining the user's visual area in Scene 1, Scene 2 and Scene 3. Not limited to this, in other examples, when it is detected that the user's line of sight position is located in a non-editable area (such as a functional area) in the user interface displayed on the current display screen, the electronic device 100 can use the functional area where the line of sight position is located as the user's visual area, for example, the method of determining the user's visual area in Scene 4. The shape of the visual area can be, but is not limited to, a circle, square, rectangle, triangle, or other polygons, etc., and this application does not limit this. The size of the visual area can be adjusted by the user operating the electronic device 200. For specific instructions, please refer to the description of Figure 4I above, which will not be repeated here.

[0147] The visual area is not limited to the case where one editable object is included in the above example. In another embodiment, the visual area may also include multiple editable objects, wherein these multiple editable objects may include incomplete and / or complete editable objects. As shown in Figure 6B, the electronic device 100 can display a user interface 580. The user interface 580 is similar to the user interface 510 shown in (A) of Figure 5A. The difference is that the area box 581 corresponding to the visual area in the user interface 580 may include three editable objects (text content 581, image 582, image 583), wherein the text content 581 and the image 582 are both complete objects, and the image 583 is an incomplete object (belonging to the part of the image "kitten smelling the flowers"). The electronic device 100 can receive a user operation (such as a touch operation, such as a double-click operation) on any object (complete / incomplete object) through the electronic device 200, and determine the selected object based on the received user operation. When the selected object is determined to be a complete object (such as text content 581 and image 582), the complete object can be selected; when the selected object is determined to be an incomplete object (such as image 583), the complete object to which the incomplete object belongs (such as the image "kitten smelling flowers") can be selected. Subsequently, the user can operate the electronic device 200 again to cause the electronic device 100 to perform a functional operation on the selected object (complete object).

[0148] The above example is not limited to the case where the electronic device 200 is a touch panel. In other examples, the electronic device 200 can also be a mouse, or other sensors. Not limited to this, in other examples, there can be no electronic device 200, and the above operations on the electronic device 200 can be replaced by operations using a finger to touch the display screen, or gesture operations, etc.

[0149] Next, the interaction method provided by the embodiment of the present application is introduced.

[0150] Please refer to Figure 7, which is a flow chart of an interactive method provided by an embodiment of the present application. The method can be applied to the interactive system 10 shown in Figure 1. The method may include but is not limited to the following steps:

[0151] S101: The electronic device 100 displays a first interface.

[0152] In one embodiment, the first interface may be the desktop of the electronic device 100. The first interface may include multiple objects, and the multiple objects may include a first object. The first object in the first interface is located at a first position. For example, the first interface is the user interface 410 shown in FIG4A , the first object is the icon 411 in the user interface 410, and the first position is the position of the icon 411 in the user interface 410. For another example, the first interface is the user interface 510 shown in FIG5A (A), the first object is the text content 5151 in the user interface 510, and the first position is the position of the text content 5151 in the user interface 510.

[0153] In one embodiment, the first interface may include an editable area, and multiple objects may be in the editable area, wherein the editable area means that operations can be performed on objects in the area, for example, objects in the area can be moved, edited, corresponding applications can be started, etc.

[0154] S102: The electronic device 100 detects a second position where the user's gaze is directed, determines a first visual area according to the second position, and highlights the first visual area in the first interface.

[0155] In one embodiment, the second position may be the position where the user's line of sight is focused on the display screen, which may be a point, such as the user's line of sight focus point. In some examples, the electronic device 100 may determine the line of sight focus point (i.e., the second position) based on the user's eye features (e.g., the relative positions of the pupil and Purkinje's spot) acquired by the front camera and the distance between the user's eye and the display screen measured by the distance sensor. In other examples, the electronic device 100 may detect the position where the user's line of sight is focused on the display screen using other sensors.

[0156] In one embodiment, the electronic device 100 can determine the first visual area based on the second position and the user interface currently displayed on the display screen. In some examples, after the electronic device 100 obtains the second position, it can determine which type of area of ​​the user interface currently displayed on the display screen the second position is located in, and determine the user's first visual area according to the preset rules corresponding to the type of area. For example, when the second position is point B on the display screen shown in FIG4B, the user interface 410 currently displayed on the display screen is an editable area, that is, point B is located in the editable area, the electronic device 100 can be centered on point B and with a preset distance R as a radius to determine the first visual area. The first visual area is, for example, the area box 421 in the user interface 420 shown in FIG4B. For another example, when the second position is the character ":" in the user interface 510 shown in FIG5A (A), and the character ":" is located in the editable area (editing area 514) of the user interface 510 currently displayed on the display screen, the electronic device 100 can also be centered on the character ":" and with a preset distance T as a radius to determine the first visual area. The first visual area is, for example, the area box 515 in the user interface 510 shown in FIG5A (A). It can be understood that the preset rule corresponding to the editable area can be to determine the first visual area with the second position as the center and a preset distance as the radius.

[0157] Without limitation to this, in other examples, the preset rule corresponding to the editable area may be to determine the first visual area based on the number of objects within a preset range of the second position. For example, when the second position is object A on the display screen shown in FIG4C , and object A is located in the editable area, the electronic device 100 may determine the range of object A and three objects within a preset range from object A (such as the clock application, browser application, and address book application in the user interface 430 shown in FIG4C ) as the first visual area (such as the area box 421 in the user interface 430). This application does not limit the specific manner of the preset rules corresponding to the editable area.

[0158] In one embodiment, the size of the first visual area may be related to the size of the first interface, the size of the objects in the first interface, the number of objects that the user pre-sets to include in the visual area (which may be simply referred to as the number of circled objects), etc. The shape of the first visual area may be, but is not limited to, a circle, square, rectangle, triangle, or other polygons, etc., and this application does not limit this.

[0159] In one embodiment, the size of the first visual area can be adjusted by the user operating the electronic device 200. For example, the electronic device 200 can send a touch operation for the first visual area to the electronic device 100 (for example, the touch operation is a pinch operation), and the electronic device 100 can adjust the size of the first visual area according to the touch operation, such as increasing the size of the first visual area. Optionally, the number of objects contained in the first visual area also increases. In some examples, it is assumed that the first visual area before adjustment is the area box 421 in the user interface 420 shown in Figure 4B, and the area box 421 contains 4 objects. The first visual area after adjustment is the area box 481 in the user interface 480 shown in Figure 4I, and the area box 481 contains 9 objects.

[0160] In one embodiment, the position of the first visual area may not move: when the electronic device 100 detects that the position of the user's gaze moves within the first visual area, the position of the first visual area may remain unchanged. For example, as shown in FIG4C , when the gaze position moves from point B on the display screen to object A, the position of the first visual area does not change. In another embodiment, the position of the first visual area may move: when the electronic device 100 detects that the position of the user's gaze moves from within the first visual area to outside the first visual area, the position of the first visual area may move accordingly. For example, as shown in FIG4E , when the gaze position moves from object A on the display screen to point E, the position of the first visual area may move accordingly.

[0161] In one embodiment, after the electronic device 100 determines the first visual area, it can narrow the user's operable range from the first interface to within the first visual area. For example, before S102, the cursor of the electronic device 100 is located outside the first visual area in the first interface, and after S102, the cursor of the electronic device 100 is located within the first visual area in the first interface.

[0162] In one embodiment, the electronic device 100 can highlight the first visual area in the first interface, wherein the method of highlighting the first visual area may include but is not limited to: adding a border to the first visual area, changing the fill color of the first visual area (different from the fill color of other areas in the first interface), enlarging the display of all objects in the first visual area, etc.

[0163] In one embodiment, the first visual area may include at least one object in the first interface (assuming that it includes the first object mentioned above), and any one of the objects may be used as an operated object.

[0164] S103: The electronic device 100 sends a first message to the electronic device 200.

[0165] In one embodiment, S103 is an optional step.

[0166] In one embodiment, the electronic device 200 may be an external input device of the electronic device 100, such as a mouse, a touchpad, or other sensors. After receiving the first message, the electronic device 200 may switch the current first operating mode of the electronic device 200 to the second operating mode according to the first message, wherein, when the first interface displayed by the electronic device 100 does not include a visual area, the electronic device 200 corresponds to the first operating mode, and the user-operable range corresponding to the first operating mode is the first interface; when the first interface displayed by the electronic device 100 includes a visual area, the electronic device 200 corresponds to the second operating mode, and the user-operable range corresponding to the second operating mode is the visual area.

[0167] S104: The electronic device 200 receives a first user operation on a first object in a first visual area.

[0168] In one embodiment, the electronic device 200 can be an external input device of the electronic device 100, such as a mouse, a touchpad, or other sensors. When the electronic device 200 is a touchpad, the operation method of the touchpad can be related to the number of objects in the first visual area. Specific examples can be found in (A) to (C) of Figure 4D, which will not be repeated here.

[0169] S105: The electronic device 200 sends information of the first user operation to the electronic device 100.

[0170] In one embodiment, the information of the first user operation may include, but is not limited to, first operation coordinates and a first operation mode. The first operation coordinates may be used to determine the operation object (first object), and the first operation mode may be used to select the first object and set the state of the first object to a movable state. Without limitation, the electronic device 200 may send the operation object (first object) and the first operation mode for the operation object to the electronic device 100.

[0171] S106: The electronic device 100 sets the first object to a movable state according to the information of the first user operation.

[0172] In one embodiment, the electronic device 100 can determine based on the information of the first user operation received: the operation object is the first object, the first operation method is to select the first object and set the state of the first object to a movable state, and the electronic device 100 can set the first object to a movable state. At this time, the user interface displayed by the electronic device 100 is similar to the first interface, the difference is that the first object in the user interface is in a movable state. In some examples, the user interface can be similar to the user interface 430 shown in Figure 4C, the difference is that the icon 411 in the user interface 430 is in a selected state, and the icon 411 (i.e., the first object) in the user interface is in a movable state, and the display method of the movable state is different from the display method of the selected state.

[0173] S107: When the electronic device 100 detects that the position where the user's gaze is moving from the second position to the third position, the electronic device 100 moves the first object from the first visual area to the second visual area.

[0174] In one embodiment, when the electronic device 100 detects that the user's line of sight position moves from the second position, it can determine whether the area type of the first interface displayed on the current display screen where the line of sight position is located during the movement has changed. Figure 7 takes the example where the area type corresponding to the line of sight position during the movement does not change (all are editable areas), that is, the second position and the third position are both located in the editable area, and other positions in the process of moving from the second position to the third position are also located in the editable area. Therefore, the method for determining the visual area during the movement can be the same, and the size of the visual area can remain unchanged.

[0175] In one embodiment, when the first object in the first visual area is in a movable state, when the electronic device 100 detects that the position where the user's gaze is directed moves, the first object and the visual area where the first object is located can be moved, wherein before moving the first object, the visual area where the first object is located is the first visual area, and during the process of moving the first object, the visual area where the first object is located is the second visual area. The first visual area may include a first object, optionally with other objects, and the second visual area may include multiple first objects located at different positions. Assuming that the first objects are located at N positions respectively, it can be represented that the first object can be moved to any one of the N positions in the second visual area. In some examples, the second position is, for example, object A on the display screen shown in Figure 4E, the third position is, for example, point E on the display screen shown in Figure 4E, and the second visual area is, for example, the area box 441 in the user interface 440 shown in Figure 4E, and the area box 441 includes the first objects at four positions (corresponding to the icon 442).

[0176] In one embodiment, when the first object in the first visual area is in a movable state, when the electronic device 100 detects that the position where the user's line of sight is being looked at has moved, the first visual area can be changed to a second visual area, and then the second visual area can be moved accordingly according to the trajectory of the movement of the user's line of sight. It can be understood that when the user's line of sight moves, the visual area displayed by the electronic device 100 can move accordingly. During the movement of the visual area, the electronic device 100 can display the first object at different positions within the visual area, which can represent that these positions can all be used as the display positions after the first object is moved, and the number of these positions can be the same as the number of circled objects in the visual area.

[0177] S108: The electronic device 200 receives a second user operation directed to a fourth position within the second visual area.

[0178] In one embodiment, the second visual area may include a plurality of first objects located at different positions, and the first object at any one position (assuming the fourth position) may be selected to move the first object to the fourth position.

[0179] In one embodiment, the description of S108 is similar to the description of S104 in FIG. 7 , and details can be found in the description of S104 in FIG. 7 , which will not be repeated here.

[0180] S109 : The electronic device 200 sends the information of the second user operation to the electronic device 100 .

[0181] In one embodiment, the information of the second user operation may include but is not limited to second operation coordinates and a second operation mode. The second operation coordinates may be used to determine the position after movement (fourth position), and the second operation mode may be used to move the first object to the fourth position.

[0182] S110: The electronic device 100 displays the first object at a fourth position in the first interface according to the information of the second user operation.

[0183] In one embodiment, the electronic device 100 can move the first object to the fourth position based on the received information of the second user operation, and display the first object at the fourth position in the first interface. It can be understood that the electronic device 100 moves the first object from the first position in the first interface to the fourth position. In some examples, the fourth position is, for example, the position of the icon 442 in the user interface 440 shown in Figure 4E, and the interface displaying the first object at the fourth position is, for example, the user interface 450 shown in Figure 4F, and the icon 411 (i.e., the first object) in the user interface 450 is in the fourth position.

[0184] Not limited to the above embodiment, in another embodiment, S107 of Figure 7 can be replaced as follows: the electronic device 200 can receive a third user operation on the first visual area and send information of the third user operation to the electronic device 100. The electronic device 100 can move the first visual area according to the third user operation to obtain the moved second visual area, and move the first object from the first visual area to the second visual area.

[0185] Not limited to the above embodiment, in another embodiment, S102 of Figure 7 can be replaced as follows: the electronic device 100 can obtain the cursor position in the first interface, and the cursor position can be determined by user operation, for example, the user operates the electronic device 200 to control the cursor position of the electronic device 100, etc., and the first area is determined according to the cursor position, and the first area (including the first object) is highlighted in the first interface. The instructions for determining the first area according to the cursor position can be found in the specific instructions for determining the first visual area according to the second position in S102, and will not be repeated here.

[0186] Not limited to the above embodiment, in another embodiment, after S105 of Figure 7, when the first object is in a selected state, the electronic device 200 can receive a fourth user operation on the first object and send information of the fourth user operation to the electronic device 100. The electronic device 100 can perform corresponding operations on the first object according to the fourth user operation, such as starting the application corresponding to the first object, or displaying a function menu corresponding to the first object.

[0187] The scenario examples in FIG7 can be seen in the aforementioned scenarios 1, 2, and 3. In scenario 1, the first object is an icon, such as icon 411 in user interface 410 shown in FIG4A ; the first visual area is area box 421 in user interface 420 shown in FIG4B ; and the second visual area is area box 441 in user interface 440 shown in FIG4E . In scenario 2, the first object is an editable object, such as text 5151 in user interface 510 shown in FIG5A (A); the first visual area is area box 515 in user interface 510 shown in FIG5A (A). In scenario 3, the first object is a cursor, such as cursor 561 in user interface 560 shown in FIG5B (A); and the first visual area is area box 515 in user interface 570 shown in FIG5B (A).

[0188] In the method shown in Figure 7, the electronic device 100 and the electronic device 200 are connected, and the user can control the electronic device 100 by operating the electronic device 200. The electronic device 200 can be understood as an external input device of the electronic device 100. The electronic device 100 can display a first interface, and when it is detected that the user's gaze is on the screen, the first visual area is determined according to the second position of the user's gaze. The electronic device 100 can highlight the first visual area in the first interface, and optionally at least one object in the first visual area. At this time, the user's operable range is narrowed from the first interface to the first visual area. Then, the electronic device 100 can obtain the user operation on the first object in the first visual area through the electronic device 200, and perform corresponding processing on the first object according to the user operation (such as moving the position, starting the corresponding application, displaying the relevant function menu, etc.). In this way, the user can first "roughly select and locate" the area to be operated (i.e., the above-mentioned first visual area) by gazing at the area, and then "finely locate" the object to be operated (i.e., the above-mentioned first object) in the area through the electronic device 200. Therefore, the user does not need to move and operate the electronic device 200 multiple times, and can avoid the situation where the hand is easily fatigued due to multiple movements, cumbersome operations, and long operation times. In addition, the user does not need to slide his fingers multiple times on the electronic device 200 (such as a touchpad) and accurately select objects. The method of the present application can effectively reduce the fatigue of the user's hands when performing interactive operations, allowing the user to perform accurate interactive operations with less effort, improve the speed of the interactive operation process, and thus improve the interaction efficiency. In addition, this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0189] Please refer to Figure 8, which is a flowchart of another interactive method provided by an embodiment of the present application. The method can be applied to the interactive system 10 shown in Figure 1. The method may include but is not limited to the following steps:

[0190] S201: The electronic device 100 displays a second interface.

[0191] In one embodiment, the second interface may be an editing interface of the first application, the second interface may include at least one object, and the at least one object may include a second object. For example, the second interface may be user interface 510 (excluding area frame 515) shown in FIG5A(A), and the second object may be text content 5151 in user interface 510.

[0192] In one embodiment, the second interface may include multiple different types of areas, such as an editable area (editing area), a functional area, a view area, etc. These multiple different types of areas may be divided by the first application according to its own functional options and the nature of the interactive target. For example, the user interface 510 (excluding the area frame 515) may include an editable area (editing area 514), a functional area (functional area 512 and toolbar 511), and a view area (view area 513), wherein the editing area 514 may be used to operate and display editable objects, and the functional area 512 may be used to perform functional operations corresponding to the functional options on the editable objects.

[0193] S202: The electronic device 100 detects a fifth position where the user's gaze is directed, determines a third visual area based on the fifth position, and highlights the third visual area in the second interface.

[0194] In one embodiment, the description of the fifth position is similar to that of the second position in S102 of Figure 7 , and reference may be made to the description of the second position in S102 of Figure 7 , which will not be repeated. The fifth position may be located in an editable area in the second interface.

[0195] In one embodiment, the electronic device 100 can determine the third visual area based on the fifth position and the second interface displayed on the current display screen. In some examples, after the electronic device 100 obtains the fifth position, it can determine which type of area the fifth position is located in the second interface, and determine the user's third visual area according to the preset rules corresponding to the type of area. For example, when the fifth position is the character ":" in the user interface 510 shown in (A) of Figure 5A, it is determined that the fifth position is located in an editable area (editing area) in the second interface. The electronic device 100 can determine the third visual area according to the preset rules corresponding to the editable area. The third visual area is, for example, the area box 515 in the user interface 510 shown in (A) of Figure 5A. Among them, examples of preset rules corresponding to editable areas can be found in the description in S102 of Figure 7 and will not be repeated here.

[0196] In one embodiment, the third visual area is an area within the editing area in the second interface. The third visual area may include at least one editable object / content (assuming it includes the second object), and any editable object / content may be used as an operated object.

[0197] In one embodiment, the description of S202 is similar to the description of S102 in Figure 7, with the difference that the size of the third visual area can be related to the size of the editing area in the second interface, the size of the object in the editing area, the size of the visual area pre-set by the user, etc., which will not be repeated.

[0198] S203: The electronic device 200 receives a fifth user operation on the second object within the third visual area.

[0199] In one embodiment, the description of S203 is similar to the description of S104 in FIG. 7 , and details may be found in the description of S104 in FIG. 7 , which will not be repeated here.

[0200] S204 : The electronic device 200 sends information of the fifth user operation to the electronic device 100 .

[0201] In one embodiment, the fifth user operation information may include but is not limited to third operation coordinates and a third operation mode. The third operation coordinates may be used to determine the operation object (second object), and the third operation mode may be used to select the second object.

[0202] S205: The electronic device 100 selects a second object according to the information of the fifth user operation.

[0203] In one embodiment, the electronic device 100 can determine based on the information of the fifth user operation received: the operation object is the second object, the third operation mode is to select the second object, and the electronic device 100 can set the second object to a selected state. At this time, the electronic device 100 can display an interface in which the second object is selected, for example, the user interface 520 shown in (B) of Figure 5A, and the text content 5151 (i.e., the second object) in the user interface 520 is in a selected state.

[0204] S206: When the electronic device 100 detects that the position where the user's gaze is moving from the fifth position to the sixth position, the electronic device 100 determines a fourth visual area according to the sixth position and highlights the fourth visual area in the second interface.

[0205] In one embodiment, the description of the sixth position is similar to the description of the fifth position in S202 of FIG. 8 , except that the sixth position may be located in a functional area in the second interface.

[0206] In one embodiment, when the second object is selected, when the electronic device 100 detects that the user's line of sight position moves from the fifth position, it can be determined whether the type of area of ​​the second interface displayed on the current display screen where the line of sight position is located during the movement changes. When it changes, the electronic device 100 can determine the visual area based on the changed type area (where the line of sight position is located). In some examples, the fifth position is located in an editable area in the second interface, and the sixth position after movement is, for example, the character "U" in the user interface 510 shown in (A) of Figure 5A. The sixth position can be located in a functional area in the second interface. Therefore, the electronic device 100 can determine the fourth visual area based on the sixth position according to the preset rules corresponding to the functional area. In some examples, the preset rules corresponding to the functional area can be to use the functional area where the sixth position is located as the fourth visual area, for example, to determine the font functional area 5121 where the character "U" is located as the fourth visual area, and the fourth visual area is, for example, the area box 531 in the user interface 530 shown in (A) of Figure 6A.

[0207] Not limited to this, in other examples, when the second object is selected, it is assumed that the sixth position after movement is a point in the upper right blank area of ​​page 5141 in the user interface 510 shown in (A) of Figure 5A. At this time, the sixth position is located in the editable area in the second interface. Therefore, the electronic device 100 can determine the visual area after movement according to the preset rules corresponding to the editable area. The size and shape of the visual area after movement are the same as those of the visual area before movement. The difference is that the object in the visual area after movement changes.

[0208] In one embodiment, the fourth visual area is an area within the functional area in the second interface. The fourth visual area may include at least one editing function option (assuming that it includes a third object). An editing function option may indicate an editing function. Any editing function option may be selected to perform a corresponding functional operation on the selected editable object / content.

[0209] In one embodiment, the description of S206 is similar to the description of S107 in Figure 7, with the difference being that, in the process of the user's line of sight moving from the fifth position to the sixth position, the type of area where the line of sight is located changes (for example, from an editing area to a functional area), and therefore, the method of determining the visual area may change, the size of the visual area may change, and the number of objects in the visual area may change.

[0210] It can be understood that when the electronic device 100 displays the editing interface, the user's eyes are fixed on different types of areas within the editing interface, and the number of objects circled by the visual areas corresponding to different types of areas may be different. For example, the area box 515 (in the editing area 514) in the user interface 510 shown in Figure 5A (A) only includes one editable object (i.e., text content 5151), and the area box 531 (in the function area 512) in the user interface 530 shown in Figure 6A (A) includes multiple editing function options (i.e., all controls contained in the area box 531).

[0211] Without limitation to this, in another embodiment, when the second object is in a movable state (for example, after S205, the electronic device 200 receives a touch operation (for example, a long press operation) for the second object and sends the touch operation to the electronic device 100, and the electronic device 100 sets the second object to a movable state after receiving the touch operation), if it is detected that the user's line of sight position moves from the fifth position (located in the editing area) to the seventh position (located in the editing area), that is, the area where the line of sight position before and after the movement is located is the editing area, the electronic device 100 can subsequently move the display position of the second object. For specific instructions, please refer to S107-S110 of Figure 7, and no further details will be given.

[0212] S207: The electronic device 200 receives a sixth user operation on the third object in the fourth visual area.

[0213] In one embodiment, the description of S207 is similar to the description of S104 in FIG. 7 , and details can be found in the description of S104 in FIG. 7 , which will not be repeated here.

[0214] S208 : The electronic device 200 sends information of the sixth user operation to the electronic device 100 .

[0215] In one embodiment, the information of the sixth user operation may include but is not limited to fourth operation coordinates and a fourth operation mode. The fourth operation coordinates can be used to determine the operation object (third object), and the fourth operation mode is used to select the third object and perform the functional operation indicated by the third object.

[0216] S209: The electronic device 100 edits the second object according to the editing function indicated by the third object.

[0217] In one embodiment, the electronic device 100 can determine based on the information of the sixth user operation: perform an editing function (selected) operation on the selected editable object, the electronic device 100 can edit the selected second object according to the editing function indicated by the third object, and obtain the processed object. For example, the third object is the control 532 in the user interface 530 shown in (A) of Figure 6A, and the editing function indicated by the third object is to underline the selected editable object. The electronic device 100 can underline the selected second object (for example, the text content 5151 in the user interface 520 shown in (B) of Figure 5A), and the processed object is, for example, the text content 5151 in the user interface 550 shown in (C) of Figure 6A.

[0218] Not limited to the above embodiment, in another embodiment, S202 of Figure 8 can be replaced as follows: the electronic device 100 can obtain the cursor position in the second interface, and the cursor position can be determined by user operation, for example, the user operates the electronic device 200 to control the cursor position of the electronic device 100, etc., and the second area is determined according to the cursor position, and the second area (including the second object) is highlighted in the second interface. The instructions for determining the second area according to the cursor position can be found in the specific instructions for determining the third visual area according to the fifth position in S202, and will not be repeated here.

[0219] The example scenario of Figure 8 can refer to the above-mentioned scenario 4, wherein the second interface is the editing interface of the first application, such as the user interface 510 shown in (A) of Figure 5A (excluding the area box 515), the second object is an editable object, such as the text content 5151 in the user interface 510 shown in (A) of Figure 5A, the third visual area is the area within the editing area, such as the area box 515 in the user interface 510 shown in (A) of Figure 5A, the fourth visual area is the area within the function area, such as the area box 531 in the user interface 530 shown in (A) of Figure 6A, and the third object is an editing function option, such as the control 532 in the user interface 530 shown in (A) of Figure 6A.

[0220] In the method shown in FIG8 , the electronic device 100 can select the object to be operated and the editing function options of the object by combining sight and operation to implement the editing function, which can effectively improve the user's editing efficiency. In addition, this method can meet the interaction needs of different users in different scenarios and expand the scope of applicable scenarios.

[0221] Please refer to Figure 9, which is a flowchart of another interactive method provided by an embodiment of the present application. This method can be applied to the interactive system 10 shown in Figure 1. The method may include but is not limited to the following steps:

[0222] S301: The first device displays a first interface.

[0223] In one embodiment, the first interface may be the desktop of the first device. The first interface may include multiple objects, and the multiple objects may include the first object. For the specific description of the first interface, please refer to the description of the first interface in S101 of Figure 7, and will not be repeated here.

[0224] In another embodiment, the first interface may be an editing interface of the first application. For the description of the first interface, please refer to the description of the second interface in S201 of FIG. 8 , which will not be repeated here.

[0225] In one embodiment, the first interface may include at least one type of area, such as an editable area, a function area, a view area, etc.

[0226] S302: The first device detects that the user's gaze is directed at a first position on the display screen, and determines a first visual area based on the first position.

[0227] In one embodiment, the detailed description of the first position can refer to the description of the second position in S102 of FIG. 7 , which will not be repeated here.

[0228] In one embodiment, the first position is located in a first type of area, and the first type of area is, for example, an editable area. The first device can determine the first visual area based on the first position and the preset rules corresponding to the first type of area. In some examples, the first device can determine the first visual area with the first position as the center and a preset distance as the radius. In other examples, the first device can determine the first visual area based on the number of objects within a preset range of the first position. For specific instructions, please refer to the instructions for determining the first visual area based on the second position in S102 of Figure 7, which will not be repeated here.

[0229] In one embodiment, the first visual area may include at least one object among multiple objects, and the at least one object includes the first object, wherein the number of the at least one object in the first visual area is N, and the number of multiple objects in the first interface is M, N and M are both positive integers, and N is less than M. For a specific description of the first visual area, please refer to the description of the first visual area in S102 of Figure 7, and no further details will be given.

[0230] S303: The first device displays a first visual area in the first interface.

[0231] In one embodiment, the description of S303 can refer to the specific description of highlighting the first visual area in the first interface in S102 of FIG. 7 , and will not be repeated here.

[0232] In one embodiment, when the first device displays the first interface, the area in which the second device acts includes M objects in the first interface; when the first device displays the first visual area in the first interface, the area in which the second device acts includes N objects.

[0233] S304: The first device receives the first information sent by the second device.

[0234] In one embodiment, the second device may be an external input device of the first device. For the description of the second device, please refer to the description of S104 in FIG. 7 , which will not be repeated here.

[0235] In one embodiment, the description of the first information can refer to the specific description of the information of the first user operation in S105 of FIG7 , which will not be repeated here.

[0236] S305: The first device determines, according to the first information, that the operation object is the first object.

[0237] S306: The first device performs a first operation on the first object according to the first information.

[0238] In one embodiment, the first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

[0239] In one embodiment, the first operation is a selection operation. After S306, when the first device detects that the position where the user's gaze is looking at the display screen moves to a second position, the first device can determine the second visual area based on the second position.

[0240] In one embodiment, the first device can determine whether the type of the second type area where the second position is located is the same as the type of the first type area (for example, an editable area) where the first position is located. When the type of the second type area is the same as the type of the first type area, the second visual area includes N first objects, and these N first objects are respectively located at N positions in the second visual area. For specific instructions, please refer to the description of the second visual area in S107 of Figure 7, which will not be repeated. Then, the first device can receive the second information sent by the second device, and determine the third position in the second visual area based on the second information. The third position is any one of the N positions. The first device can display the first object at the third position. The third position is different from the position of the first object in the first interface. For specific instructions, please refer to the description of S108-S110 of Figure 7, which will not be repeated.

[0241] In one embodiment, when the second type area and the first type area are of different types, the second visual area includes at least one functional option. For a detailed description, please refer to the description of the fourth visual area in S206 of Figure 8 and will not be repeated here. Next, the first device can receive third information sent by the second device and determine, based on the third information, that the operation object is a first functional option, where the first functional option is any one of the at least one functional option. The first device can perform a second operation on the first object based on the third information, where the second operation is the operation indicated by the first functional option. For a detailed description, please refer to S207-S209 of Figure 8.

[0242] In the method shown in Figure 9, the fatigue of the user's hands when performing interactive operations can be effectively reduced, allowing the user to achieve precise interactive operations with less effort, improving the speed of the interactive operation process, thereby improving the interaction efficiency. In addition, this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0243] The methods provided in the various embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DWD), or a semiconductor medium (e.g., a solid state drive (SSD)). As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive method, characterized in that: Applied to a first device, the first device is configured with a display screen, the first device is connected to a second device, and the method includes: Displaying a first interface, wherein the first interface includes a plurality of objects; Detecting that the user is looking at a first position of the display screen, and determining a first visual area according to the first position; Displaying the first visual area in the first interface, the first visual area includes at least one object among the multiple objects, the at least one object includes a first object, the number of the at least one object is N, the number of the multiple objects is M, N is less than M, and both N and M are positive integers; receiving first information sent by the second device; Determine, according to the first information, that the operation object is the first object; A first operation is performed on the first object according to the first information.

2. The method according to claim 1, characterized in that The first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and determining the first visual area according to the first position includes: The first visual area is determined according to the first position and a preset rule corresponding to the first type of area.

3. The method according to claim 2, characterized in that The determining the first visual area according to the first position and the preset rule corresponding to the first type of area includes: The first visual area is determined with the first position as the center and a preset distance as the radius; or, The first visual area is determined according to the number of objects within a preset range of the first position.

4. The method according to claim 2, characterized in that The first type of area includes a first functional area, and determining the first visual area according to the first position and a preset rule corresponding to the first type of area includes: When the first position is located in the first functional area, the first functional area is set as the first visual area.

5. The method according to any one of claims 1 to 4, characterized in that: The first operation is a selection operation; After performing the first operation on the first object according to the first information, the method further includes: When it is detected that the position where the user's line of sight is gazing at the display screen moves to a second position, determining a second visual area according to the second position; The second visual area is displayed in the first interface.

6. The method according to claim 5, characterized in that The first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and the method further includes: Determine whether the second type area where the second position is located is of the same type as the first type area; When the second type area and the first type area are of the same type, the second visual area includes the N first objects, and the N first objects are respectively located at the N positions in the second visual area; When the second type area and the first type area are of different types, the second visual area includes at least one functional option.

7. The method according to claim 5 or 6, characterized in that When the second type area is of the same type as the first type area, after displaying the second visual area in the first interface, the method further includes: receiving second information sent by the second device; Determine a third position in the second visual area according to the second information, where the third position is any one of the N positions; The first object is displayed at the third position, and the third position is different from the position of the first object in the first interface.

8. The method according to claim 5 or 6, characterized in that When the second type area is different from the first type area in type, after displaying the second visual area in the first interface, the method further includes: receiving third information sent by the second device; According to the third information, it is determined that the operation object is a first function option, and the first function option is any one of the at least one function option. Intend an option; A second operation is performed on the first object according to the third information, where the second operation is the operation indicated by the first function option.

9. The method according to any one of claims 1 to 8, characterized in that The manner of displaying the first visual area in the first interface includes at least one of the following: adding a border to the first visual area, changing a fill color of the first visual area, and enlarging and displaying the at least one object in the first visual area.

10. The method according to any one of claims 1 to 9, characterized in that: The size of the first visual area is related to at least one of the following: the size of the first interface, the size of the multiple objects, the number of the at least one object, and the position of the first visual area.

11. The method according to any one of claims 1 to 10, characterized in that: The first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

12. The method according to any one of claims 1 to 11, characterized in that: The first object is any one of the following: an icon, an editing object, or a function option.

13. The method according to any one of claims 1 to 12, characterized in that: The first operation includes a selection operation and a moving display position; After performing the first operation on the first object according to the first information, the method further includes: A second interface is displayed, wherein a position of the first object in the second interface is different from a position of the first object in the first interface.

14. The method according to any one of claims 1 to 13, characterized in that: The first operation includes a selection operation and displaying an interface corresponding to the first object; After performing the first operation on the first object according to the first information, the method further includes: A third interface is displayed, where the third interface is an interface corresponding to the first object.

15. The method according to any one of claims 1 to 14, characterized in that: The first operation includes a selection operation and displaying a function menu related to the first object; After performing the first operation on the first object according to the first information, the method further includes: A fourth interface is displayed, wherein the fourth interface includes a function menu related to the first object.

16. The method according to any one of claims 1 to 15, characterized in that: The second device is any one of the following: a mouse, a touch pad, or a sensor.

17. The method according to any one of claims 1 to 16, characterized in that: When the first interface is displayed, the area where the second device acts includes the M objects; when the first visual area is displayed in the first interface, the area where the second device acts includes the N objects.

18. The method according to any one of claims 1 to 17, characterized in that: The at least one object includes the first object and the second object, the distance between the position of the first object and the position of the second object in the first visual area is a first distance, and the first object in the first visual area is in a selected state; the method further includes: Determine a second distance acting on the second device according to the first information and the first algorithm, wherein the second distance is smaller than the first distance; The selected object in the first visual area is switched to the second object according to the second distance.

19. The method according to any one of claims 1 to 18, characterized in that: The connection mode between the first device and the second device includes any one of the following: wired connection and wireless connection.

20. An interactive method, characterized in that: Applied to a second device, the second device is connected to a first device, the first device is configured with a display screen, and the operation mode of the second device includes a first mode and a second mode. The method includes: receiving a first message sent by the first device; According to the first message, the operation mode of the second device is switched from the first mode to the second mode, wherein the first mode is a mode corresponding to the second device when the interface displayed by the first device does not include a visual area, and the second mode is a mode corresponding to the second device when the interface displayed by the first device includes the visual area.

21. An electronic device, characterized in that: The method comprises a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the method according to any one of claims 1 to 19 or any one of claim 20.

22. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 20 is implemented.

Images