Interaction method, electronic equipment and system

Abstract

The invention provides an interaction method, electronic equipment and a system, and the method comprises the steps: displaying a first interface which comprises a plurality of objects; detecting a first position where the user watches the display screen, and determining a first visual area according to the first position; a first visual area is displayed in the first interface, the first visual area comprises at least one object in multiple objects, the at least one object comprises the first object, the number of the at least one object is N, the number of the multiple objects is M, N is smaller than M, and N and M are positive integers; receiving first information sent by the second equipment; determining the operation object as a first object according to the first information; and executing a first operation on the first object according to the first information. By adopting the method and the device, the hand fatigue feeling when the user executes the interaction operation can be effectively relieved, more accurate interaction operation can be realized, the operation speed can be improved, and the interaction efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an interaction method, electronic device and system. Background Art

[0002] Currently, users can perform a series of operations (such as selecting objects, clicking objects, dragging objects, etc.) on objects displayed on electronic devices through external input devices (such as mice, touch pads, etc.), and can also directly use their fingers to perform a series of operations on objects displayed on electronic devices (such as devices equipped with touch interactive operating systems). When users perform interactive operations in the above manner, there are at least the following disadvantages: it is easy to make the user's hands feel tired, the operation rate is slow, and the interaction efficiency is low. Summary of the invention

[0003] The present application discloses an interaction method, electronic device and system, which can effectively reduce the fatigue of the user's hands when performing interactive operations, and can achieve more precise interactive operations and increase the operation rate, thereby improving the interaction efficiency.

[0004] In a first aspect, an embodiment of the present application provides an interaction method, which is applied to a first device, wherein the first device is configured with a display screen, and the first device is connected to a second device, the method comprising: displaying a first interface, wherein the first interface comprises multiple objects; detecting that a user's line of sight is fixed on a first position of the display screen, and determining a first visual area based on the first position; displaying the first visual area in the first interface, wherein the first visual area comprises at least one object of the multiple objects, the at least one object comprises a first object, the number of the at least one object is N, the number of the multiple objects is M, N is less than M, and both N and M are positive integers; receiving first information sent by the second device; determining that the operation object is the first object based on the first information; and performing a first operation on the first object based on the first information.

[0005] In the above method, the first device can display the first interface, and when it is detected that the user's line of sight is fixed on the screen, the first visual area is determined according to the first position of the user's line of sight, and the first visual area is highlighted in the first interface, and optionally at least one object in the first visual area. At this time, the user's operable range is narrowed from the first interface to the first visual area. Then, the first device can receive the first information sent by the second device, and perform corresponding processing on the first object according to the first information. In this way, the user can first "roughly locate" the area to be operated (i.e., the above-mentioned first visual area) by gazing at the line of sight, and then "selectively locate" the object to be operated in the area (i.e., the above-mentioned first object) through the electronic device 200. Therefore, the user does not need to move and operate the second device multiple times, which can effectively reduce the fatigue of the user's hands when performing interactive operations, allowing the user to achieve accurate interactive operations with less effort, improve the rate during the interactive operation, thereby improving the interaction efficiency, and this method can achieve a faster and more intuitive operation experience, further improving the user experience.

[0006] In one possible implementation, the first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and determining the first visual area based on the first position includes: determining the first visual area based on the first position and a preset rule corresponding to the first type of area.

[0007] In the above method, the first interface may include at least one type of area, and different types of areas correspond to different preset rules. Therefore, the determined visual areas are also different. This way of determining the visual area is more in line with the user's actual operation and can meet a variety of usage scenarios.

[0008] In a possible implementation, determining the first visual area according to the preset rules corresponding to the first position and the first type of area includes: determining the first visual area with the first position as the center and a preset distance as the radius; or determining the first visual area according to the number of objects within a preset range of the first position.

[0009] In a possible implementation, the first type of area includes a first functional area, and determining the first visual area according to the first position and a preset rule corresponding to the first type of area includes: when the first position is located in the first functional area, setting the first functional area as the first visual area.

[0010] In the above method, when the first type of area is an editable area, the first device can determine a first visual area of ​​a preset size. When the first type of area is a functional area, the first device can use the first functional area corresponding to the first position as the first visual area. This way of determining the visual area is more in line with the user's actual operation and can meet a variety of usage scenarios.

[0011] In a possible implementation, the first operation is a selection operation; after performing the first operation on the first object according to the first information, the method further includes: when detecting that the position where the user's line of sight is looking at the display screen moves to a second position, determining a second visual area according to the second position; and displaying the second visual area in the first interface.

[0012] In one possible implementation, the first interface includes at least one type of area, the first position is located in a first type of area, and the first type of area is any one of the at least one type of area, and the method further includes: determining whether the second type of area where the second position is located is of the same type as the first type of area; when the second type of area and the first type of area are of the same type, the second visual area includes the N first objects, and the N first objects are respectively located at the N positions in the second visual area; when the second type of area and the first type of area are of different types, the second visual area includes at least one functional option.

[0013] In the above method, when the first object is in a movable state, when it is detected that the position where the user's line of sight is looking at the display screen switches from one position to another position of the same type of area, the second visual area can be the visual area during the movement of the first object, and the second visual area can indicate that the first object is currently in the process of moving, which is convenient for the user to view and operate. When the first object is in a selected state, when it is detected that the position where the user's line of sight is looking at the display screen moves within the same type of area, the size of the second visual area remains unchanged; when it is detected that the position where the user's line of sight is looking at the display screen switches from one type of area to another type of area, the size of the second visual area changes. This way of determining the visual area is more in line with the user's actual operation and can also meet a variety of usage scenarios.

[0014] In a possible implementation, when the second type area and the first type area are of the same type, after displaying the second visual area in the first interface, the method further includes: receiving second information sent by the second device; determining a third position in the second visual area based on the second information, the third position being any one of the N positions; and displaying the first object at the third position, the third position being different from the position of the first object in the first interface.

[0015] In the above method, when the second visual area indicates that the first object is currently being moved, the first device can move the first object to any position in the second visual area according to the user operation sent by the second device to implement the operation of moving the object, allowing the user to achieve precise interactive operations with less effort, thereby improving the rate of the interactive operation process and thus improving the interaction efficiency.

[0016] In a possible implementation, when the second type area and the first type area are of different types, after displaying the second visual area in the first interface, the method also includes: receiving third information sent by the second device; determining that the operation object is a first function option based on the third information, and the first function option is any one of the at least one function options; performing a second operation on the first object based on the third information, and the second operation is the operation indicated by the first function option.

[0017] In the above method, the first device can select the object to be operated and the function options for operating the object by combining sight and operation to realize the editing function, which can effectively improve the editing efficiency of the user. In addition, this method can meet the interaction needs of different users in different scenarios and expand the scope of applicable scenarios.

[0018] In a possible implementation, the method of displaying the first visual area in the first interface includes at least one of the following: adding a border to the first visual area, changing the fill color of the first visual area, and enlarging and displaying the at least one object in the first visual area.

[0019] In the above method, by highlighting the first visual area, it is convenient for users to view the current operable range, allowing users to achieve precise interactive operations with less effort, thereby improving the rate of interactive operations and thus improving interaction efficiency.

[0020] In a possible implementation, the size of the first visual area is related to at least one of the following: the size of the first interface, the size of the multiple objects, the number of the at least one object, and the position of the first visual area.

[0021] In a possible implementation manner, the first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

[0022] In a possible implementation manner, the first object is any one of the following: an icon, an editing object, and a function option.

[0023] In some examples, icons may include application icons, file icons, shortcut icons, and the like.

[0024] In a possible implementation, the first operation includes a selection operation and a moving display position; after performing the first operation on the first object according to the first information, the method further includes: displaying a second interface, and the position of the first object in the second interface is different from the position of the first object in the first interface.

[0025] In a possible implementation, the first operation includes a selection operation and displaying an interface corresponding to the first object; after performing the first operation on the first object according to the first information, the method further includes: displaying a third interface, and the third interface is an interface corresponding to the first object.

[0026] In a possible implementation, the first operation includes a selection operation and displaying a function menu related to the first object; after performing the first operation on the first object according to the first information, the method further includes: displaying a fourth interface, and the fourth interface includes a function menu related to the first object.

[0027] In the above method, accurate interactive operations can be achieved by combining determining the user's visual area and operating the second device, or only operating the second device to achieve accurate interactive operations, such as moving the display position of the first object, displaying the interface corresponding to the first object, displaying the function menu related to the first object, editing the first object, etc.

[0028] In a possible implementation manner, the second device is any one of the following: a mouse, a touch pad, and a sensor.

[0029] In a possible implementation, when the first interface is displayed, the area in which the second device acts includes the M objects; when the first visual area is displayed in the first interface, the area in which the second device acts includes the N objects.

[0030] In the above method, when the first interface does not include the first visual area, the user's operable range is the first interface (including M objects); when the first interface includes the first visual area, the user's operable range is the first visual area (including N objects). It can be understood that after the first device determines the first visual area, it can narrow the user's operable range from the first interface to the first visual area, and then carefully locate the object to be operated in the first visual area. Therefore, the user does not need to move and operate the second device multiple times, which can effectively reduce the user's hand fatigue when performing interactive operations, allowing the user to achieve precise interactive operations with less energy, thereby improving the rate of interactive operations and thus improving interaction efficiency.

[0031] In one possible implementation, the at least one object includes the first object and the second object, the distance between the position of the first object and the position of the second object in the first visual area is a first distance, and the first object in the first visual area is in a selected state; the method also includes: determining a second distance acting on the second device based on the first information and a first algorithm, the second distance being smaller than the first distance; and switching the selected object in the first visual area to the second object based on the second distance.

[0032] In the above method, a relatively small slide can be used to make the cursor of the first device jump significantly between icons, thereby avoiding the problem of not being able to move the finger to the target position all at once due to the small area of ​​the second device (such as a touchpad), reducing user operations, alleviating user hand fatigue, and improving interaction efficiency.

[0033] In a possible implementation manner, a connection manner between the first device and the second device includes any one of the following: a wired connection and a wireless connection.

[0034] In a second aspect, an embodiment of the present application provides an interaction method, which is applied to a second device, wherein the second device is connected to a first device, the first device is configured with a display screen, and the operation mode of the second device includes a first mode and a second mode, and the method includes: receiving a first message sent by the first device; switching the operation mode of the second device from the first mode to the second mode according to the first message, wherein the first mode is a mode corresponding to the second device when the interface displayed by the first device does not include a visual area, and the second mode is a mode corresponding to the second device when the interface displayed by the first device includes the visual area.

[0035] In the above method, the second device may include a first mode and a second mode, wherein the first mode can be applied to scenarios where only the second device is operated and other scenarios that do not include a visual area, and the second mode can be applied to scenarios where line of sight and operation are combined. When the first device determines the user's visual area, it can send a first message to the second device to enable the second device to switch modes to better adapt to the current usage scenario, thereby increasing the rate during the interactive operation and thus improving the interaction efficiency.

[0036] In a third aspect, the present application provides an electronic device, including a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the interaction method in any possible implementation of the first aspect.

[0037] In a fourth aspect, the present application provides an electronic device, comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the interaction method in any possible implementation of the first aspect.

[0038] In a fifth aspect, the present application provides an electronic device, including a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the interaction method in any possible implementation of the second aspect.

[0039] In a sixth aspect, the present application provides a computer storage medium storing a computer program, which, when executed by a processor, implements the interactive method in any possible implementation of any of the above aspects.

[0040] In a seventh aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the interaction method in any possible implementation of the first aspect.

[0041] In an eighth aspect, the present application provides an electronic device, the electronic device comprising a method or device for executing any one of the implementations of the first aspect of the present application. The electronic device is, for example, a chip.

[0042] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The following is an introduction to the drawings used in this application.

[0044] Figure 1 is a schematic diagram of the architecture of an interactive system 10 provided by the present application;

[0045] Figure 2 is a hardware structure diagram of an electronic device 100 provided in this application;

[0046] Figure 3 is a schematic diagram of a software architecture of an electronic device 100 provided in this application;

[0047] Figure 4A-Figure 4I , Figure 5A-Figure 5B , Figure 6A-6B is a schematic diagram of some user interfaces provided by this application;

[0048] Figure 7 It is a flowchart of an interactive method provided by this application;

[0049] Figure 8 It is a flowchart of another interactive method provided by the present application;

[0050] Fig. 9 This is a flowchart of another interactive method provided by this application. DETAILED DESCRIPTION

[0051] The technical solution in the embodiment of the present application will be described below in conjunction with the accompanying drawings. In the description of the embodiment of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiment of the present application, "multiple" means two or more than two.

[0052] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0053] Currently, the terminal interaction modes include the following two situations:

[0054] In the first case, the user can use an external input device (such as a mouse, a touchpad, etc.) to perform interactive operations on the objects displayed on the terminal, such as moving the mouse to move the cursor displayed on the display screen, clicking the left button of the mouse to select an object, clicking the right button of the mouse to trigger the display function menu, etc. For another example, sliding a finger on the touchpad to move the displayed cursor, clicking the touchpad to select an object, long pressing the touchpad to trigger the display function menu, etc. When interacting with the mouse, the user needs to repeatedly move the mouse and click the left and right buttons of the mouse, with many movements and different amplitudes each time. The operation is cumbersome, the user's hands are prone to fatigue, and the interaction rate is also slow, resulting in low interaction efficiency. Interaction via the touchpad also requires sliding the finger repeatedly, and is limited by the size of the touchpad. When the actual distance the cursor needs to be moved (for example, the distance between icon A and icon B on the display) is greater than the longest distance on the touchpad, the finger needs to be moved multiple times. In addition, for precise interactive operations, such as moving the position of icon C or selecting a text, the fingers cannot be operated flexibly, which easily leads to fatigue and a slow interaction rate, resulting in low interaction efficiency.

[0055] Case 2: For devices equipped with a touch interactive operating system, users can directly use their fingers to interact with objects displayed on the terminal, such as clicking on an object on the screen to select it, long pressing an object on the screen to trigger functions related to the object, etc. In this way, users need to suspend their fingers, slide their fingers, click their fingers, etc. many times, and because the contact surface between the finger and the screen is related to the thickness of the finger, users with thick fingers are prone to accidental touches, making it difficult to achieve accurate interactive operations. In addition, for devices with larger screen sizes (for example, greater than or equal to 13 inches), users need to raise their arms and operate with their fingers, because the icons in their interface are larger in size and spacing, and the distance and amplitude of arm and finger movement also increase, which can easily cause users to quickly feel fatigued, and the interaction rate is also slow, resulting in low interaction efficiency.

[0056] Furthermore, in the above interaction methods, the user first focuses his / her eyes on the object to be interacted with, and then performs the interaction operation on the object, which further increases the operation time and thus has low interaction efficiency.

[0057] The embodiment of the present application provides an interactive method, which is applied to an interactive system. The interactive system may include an electronic device 100 and an electronic device 200. The electronic device 100 and the electronic device 200 are connected. The user can control the electronic device 100 by operating the electronic device 200. The electronic device 200 can be understood as an external input device of the electronic device 100. The electronic device 100 can display a first interface. When it is detected that the user's line of sight is fixed on the screen, an area where the user's line of sight is fixed (which can be referred to as a visual area) is determined according to the position of the user's line of sight (which can be referred to as the line of sight position). The electronic device 100 can highlight the visual area in the first interface, and optionally at least one object in the visual area. At this time, the user's operable range is narrowed from the first interface to the visual area. Then, the electronic device 100 can obtain the user operation on the first object in the visual area through the electronic device 200, and perform corresponding processing on the first object according to the user operation, for example, moving the position, displaying a related function menu, etc.

[0058] In the above method, the user can first "roughly locate" the area to be operated (i.e., the above visual area) by gazing at the line of sight, and then "finely locate" the object to be operated in the area (i.e., the above first object) through the electronic device 200. Therefore, the user does not need to move and operate the electronic device 200 multiple times, and can avoid the situation where the hand is easily fatigued due to many movements, cumbersome operations, and long operation time, which solves the problem of interaction through the mouse in the above situation one. In addition, the user does not need to slide his fingers and accurately select objects on the touchpad or display screen multiple times, which solves the problem of interaction through the touchpad in the above situation one, and also solves the problem of interaction through the above situation two. Through the method of the present application, the fatigue of the user's hands when performing interactive operations can be effectively reduced, allowing the user to achieve accurate interactive operations with less energy, improve the rate during the interactive operation, and thus improve the interaction efficiency. In addition, this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0059] An interactive system 10 involved in an embodiment of the present application is introduced below.

[0060] Figure 1 A schematic diagram of the architecture of an interactive system 10 is exemplarily shown.

[0061] like Figure 1 As shown, the interactive system 10 may include an electronic device 100 and an electronic device 200. The electronic device 100 may communicate with the electronic device 200 via wired (e.g., universal serial bus (USB), twisted pair, coaxial cable, optical fiber, etc.) and / or wireless (e.g., wireless local area networks (WLAN), Bluetooth, cellular communication networks, etc.), and the user may control the electronic device 100 by operating the electronic device 200. Among them:

[0062] The electronic device 100 may be a mobile phone, a tablet computer, an all-in-one computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), as well as smart home devices such as smart TVs and projectors, wearable devices such as smart bracelets, smart watches, and smart glasses, and extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), or vehicle-mounted devices. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0063] In one embodiment, the electronic device 100 can be configured with a physical display screen, such as a laptop computer display screen, a tablet computer display screen, or a smart TV display screen. It can also be connected to an external display screen (such as a desktop monitor) and display through the external display screen. It can also be configured with a virtual display screen, such as an AR virtual display screen, a VR virtual display screen, and an MR virtual display screen.

[0064] The electronic device 200 can be understood as an external input device of the electronic device 100, such as a mechanical mouse, an optical mouse, a laser mouse, a floating mouse, a touch mouse, a touchpad (built-in touchpad, external touchpad), a keyboard, a handle, a gesture sensor, a microphone, a handwriting tablet, etc. The electronic device 200 can also be an external input device in other forms. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device 200.

[0065] In one implementation, the electronic device 200 may receive a user operation and send information about the user operation to the electronic device 100 , and the electronic device 100 may execute a corresponding processing flow in response to the user operation.

[0066] Understandably, Figure 1 The number and form of the electronic devices shown are only examples. In a specific implementation, the number may be more or less. For example, the interactive system 10 may not include the electronic device 200, that is, the electronic device 200 may be integrated on the electronic device 100, such as a laptop computer with a touchpad.

[0067] Next, the structure of an exemplary electronic device provided by an embodiment of the present application is introduced.

[0068] Figure 2 A schematic diagram of the hardware structure of an electronic device 100 is exemplarily shown.

[0069] like Figure 2 As shown, the electronic device 100 may include a processor 110, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a distance sensor 180F, a proximity light sensor 180G, etc.

[0070] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0071] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0072] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0073] The processor 110 may also be provided with a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0074] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0075] The mobile communication module 150 can provide wireless communication solutions applied to the electronic device 100, including second generation mobile communication technology (second generation, 2G) / third generation mobile communication technology (third generation, 3G) / fourth generation mobile communication technology (fourth-generation, 4G) / fifth generation mobile communication technology (fifth generation, 5G) / sixth generation mobile communication technology (sixth generation, 6G).

[0076] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0077] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0078] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0079] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0080] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In one embodiment, ISP can be set in camera 193.

[0081] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. In one embodiment, the electronic device 100 may capture an image of the user through the camera 193, and obtain the position of the user's gaze on the display screen 194 based on the image.

[0082] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In one embodiment, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focus. In one embodiment, the electronic device 100 can use the distance sensor 180F to measure the distance between the user's eyes and the display screen 194 to determine the position of the user's gaze on the display screen 194.

[0083] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0084] The embodiment of the present application does not limit the module in the electronic device 100 for obtaining the position where the user's line of sight is looking at the display screen 194.

[0085] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. For example, the software system of the layered architecture may be an Android system, or a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0086] Figure 3 A schematic diagram of a software architecture of an electronic device 100 is exemplarily shown.

[0087] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0088] The application layer can include a series of application packages.

[0089] like Figure 3 As shown, the application package may include camera, gallery, music, calendar, short message, call, navigation, Bluetooth, browser, screen projection and other applications.

[0090] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0091] like Figure 3 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0092] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0093] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0094] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0095] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0096] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0097] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0098] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0099] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0100] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0101] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0102] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0103] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0104] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0105] A 2D graphics engine is a drawing engine for 2D drawings.

[0106] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0107] The following describes an application scenario involved in an embodiment of the present application and a user interface diagram in the scenario.

[0108] Scenario 1: The electronic device 100 displays the desktop. For a specific example, see Figure 4A When it is detected that the user is looking at the display screen, the electronic device 100 can highlight the user's visual area, which can include at least one icon on the desktop. For specific examples, see Figure 4B The user can operate the electronic device 200 to select any icon in the visual area. For specific examples, see Figure 4C When it is detected that the position where the user's gaze is looking at the display screen has moved, the selected icon can be moved (for example, the gallery application can be moved). For specific examples, see Figure 4E-4F .

[0109] like Figure 4AAs shown, the electronic device 100 can display a user interface 410, which is the desktop of the electronic device 100. The user interface 410 can include icons of multiple applications, such as an icon 411 of a gallery application, an icon 412 of a clock application, an icon 413 of a browser application, an icon 414 of a contact application, and icons of other applications. Not limited to this, the desktop of the electronic device 100 can also include other icons, such as icons of files, icons of shortcuts, etc.

[0110] like Figure 4B As shown, when the user 300's line of sight is fixed on the display screen (assuming that the user 300's line of sight is focused on point B), the electronic device 100 can determine the user 300's line of sight focus point (which may be referred to as the line of sight position, i.e., point B) based on the user 300's eye features (such as the relative positions of the pupil and Purkinje's plaque) acquired through the front camera and the distance between the user 300's eye and the display screen measured by the distance sensor. Next, the electronic device 100 can determine the user 300's visual area (such as a square area located at the upper left corner of the display screen with point B as the center and a preset distance as the radius) based on the user 300's line of sight position (point B) and the user interface currently displayed on the display screen. The electronic device 100 can display the user interface 420 based on the determined visual area. The user interface 420 may be the desktop of the electronic device 100. The user interface 420 and Figure 4A The user interface 410 shown is similar, except that the user interface 420 includes an area frame 421, which can represent the visual area of ​​the user 300, and the icons 411, 412, 413, and 414 in the area frame 421 are all enlarged, which can represent that these icons are in a state that can be operated (e.g., selected, clicked). Without limitation, the method of highlighting the area frame 421 can include but is not limited to: adding a border of the area frame 421, changing the fill color of the area frame 421 (different from the fill color of other areas in the user interface 420), etc.

[0111] In one embodiment, the position of the area frame 421 can be moved: when the electronic device 100 obtains that the user's line of sight position has changed (for example, when the line of sight position moves from within the visual area to outside the visual area), the position of the visual area can be moved accordingly, that is, the position of the area frame 421 moves accordingly, and the objects within the area frame 421 can also change. For example, when the line of sight position changes (assuming that it moves from point B on the display screen to the middle of the camera application icon and the file management application icon outside the visual area), the user interface and Figure 4BThe user interface 420 shown in FIG. 1 is similar to the user interface 420 shown in FIG. 1 , except that the area box 421 in the user interface does not include icons 411, 412, 413, and 414, but includes icons for camera applications and file management applications. In another embodiment, the position of area box 421 may not move: when the electronic device 100 detects that the user's line of sight position has changed within the visual area, the position of the visual area may not move. For example, when the line of sight position moves from point B on the display screen to object A, the user interface and the area box 421 displayed by the electronic device 100 may not move. Figure 4B The user interface 420 shown is the same.

[0112] In one implementation, the user 300 may operate the electronic device 200 so that the cursor of the electronic device 100 may move within the visual area of ​​the user 300 according to the operation of the user 300 .

[0113] In one embodiment, the electronic device 100 displays Figure 4B In the user interface 420 shown, when it is detected that the position where the user 300 is looking at the display screen moves within the visual area, for example Figure 4C As shown, when the sight line position of the user 300 moves from point B on the display screen to object A, the visual area displayed by the electronic device 100 remains unchanged. At this time, the user 300 can operate the electronic device 200 to select any object (assuming it is icon 411) in the area frame 421. For example, the electronic device 200 Figure 4C The touch panel shown in FIG. 1 is used by user 300 to operate, for example, Figure 4C As shown, a touch operation is performed on an object A (e.g., corresponding to icon 411) in the visual area, and the touch operation is, for example, a sliding operation to the upper left. After receiving the operation of the user 300, the electronic device 200 can send the information of the operation of the user 300 to the electronic device 100. The electronic device 100 can determine the object (icon 411) selected by the user 300 based on the received information of the operation of the user 300, and display Figure 4C The user interface 430 is shown. The user interface 430 and Figure 4B The user interface 420 is similar to the one shown, except that in the user interface 430, the icon 411 is in a selected state, and the selected state is displayed by changing the background color of the icon 411, which can indicate that the gallery application has been selected. Without limitation, the display method of the selected state of the object can include but is not limited to: adding a border to the object, enlarging the display object, etc.

[0114] Figure 4C Take the electronic device 200 as an example of a square touch panel. For specific examples, see Figure 4D .

[0115] In some examples, Figure 4D(A) shows an exemplary click operation method for a touch panel, such as Figure 4D As shown in (A), the electronic device 200 can be divided into four areas: area 1, area 2, area 3, and area 4. Different areas can correspond to different icons in the visual area. For example, area 1, area 2, area 3, and area 4 correspond to Figure 4C Icons 411, 412, 413, and 414 in the area frame 421 in the user interface 430 are shown. The electronic device 200 can receive a touch operation (e.g., a single click operation) of a user on any area (assuming it is area 1), and send the touch operation to the electronic device 100. In one embodiment, the electronic device 100 can move the cursor of the electronic device 100 to the position where the icon 411 corresponding to area 1 is located according to the received touch operation. In another embodiment, the electronic device 100 can select the icon 411 corresponding to area 1 according to the received touch operation.

[0116] Not limited to this, in other examples, Figure 4D (B) shows an exemplary sliding operation method for a touch panel, such as Figure 4D As shown in (B), the electronic device 200 can receive sliding operations in different directions, such as but not limited to: sliding up, sliding down, sliding to the left, sliding to the right, sliding to the upper left, sliding to the lower left, sliding to the upper right, sliding to the lower right, etc. In one embodiment, for sliding operations in different directions of the electronic device 200, the distance moved by the sliding operation is distance 1, which can be used to move the cursor of the electronic device 100 in different directions, and the distance moved by the cursor is also distance 1. For example, the user slides the finger to the upper left on the electronic device 200 to control the electronic device 100 to move the cursor to the upper left in the visual area. In another embodiment, for sliding operations in different directions of the electronic device 200, the distance moved by the sliding operation is distance 2, which can be used to select objects in different directions, wherein distance 1 is greater than distance 2. For example, the user slides the finger to the upper left on the electronic device 200 (for example, from Figure 4D When point C shown in (B) slides to point D), the electronic device 100 can be controlled to select the upper left object in the visual area (for example Figure 4C This operation mode can make the cursor jump greatly between icons by sliding slightly, avoiding the problem that the finger cannot be moved to the target position at one time due to the small area of ​​the touchpad, reducing user operations, alleviating user hand fatigue, and improving interaction efficiency.

[0117] The above example is not limited to the example that four objects are included in the visual area (the touch panel can be Figure 4D In some other examples, when the visual area includes more or fewer objects, the area of ​​the touchpad will also change accordingly. For example, when the visual area includes nine objects (assuming that the visual area includes Figure 4C When the icon 411, icon 412, icon 431, icon 413, icon 414, icon 432, icon 433, icon 434, icon 435 in the user interface 430 shown in FIG. Figure 4D The electronic device 200 can be divided into nine areas: area 1 to area 9. These nine areas can correspond to Figure 4C The nine icons in the user interface 430 are shown and are not described in detail.

[0118] The shape of the touch panel is not limited to the above-mentioned square touch panel as an example. The shape of the touch panel can also be, but not limited to, circular, rectangular, triangular, or other polygonal shapes. The present application does not limit the shape of the touch panel.

[0119] In one embodiment, the user 300 can operate the electronic device 200 to change the size of the visual area, for example, the electronic device 100 displays Figure 4B When the user interface 420 is shown, the electronic device 200 is, for example, Fig. 4I The touch panel shown in FIG. 1 is used by user 300 to operate, for example, Fig. 4I The touch operation shown is, for example, a pinch operation. After receiving the operation of the user 300, the electronic device 200 can send the information of the operation of the user 300 to the electronic device 100. The electronic device 100 can adjust the size of the visual area according to the received information of the operation of the user 300, for example, increase the size of the visual area and display Fig. 4I The user interface 480 is shown. The user interface 480 and Figure 4B The user interface 480 is similar to the user interface 420 shown in the figure, except that the size of the area box 481 in the user interface 480 is larger than the size of the area box 421 in the user interface 420, and the area box 481 in the user interface 480 may include icons 411, icon 412, icon 431, icon 413, icon 414, icon 432, icon 433, icon 434, and icon 435, all of which are enlarged, which can indicate that these icons are in a state that can be operated (for example, selected, clicked).

[0120] In one embodiment, when the user 300 selects object A (corresponding to icon 411) (for example, the electronic device 100 displays Figure 4C430 is selected), the user 300 can operate the electronic device 200 to put the selected object A in a state where it can be moved (which can be referred to as a movable state), the user 300 operation is, for example, a touch operation on the object A, and the touch operation is, for example, a long press operation. After receiving the operation of the user 300, the electronic device 200 can send the information of the operation of the user 300 to the electronic device 100, and the electronic device 100 can set the state of the object A to the movable state according to the received information of the operation of the user 300. At this time, the electronic device 100 can display and Figure 4C The user interface 430 shown is similar to the user interface shown in FIG. 4, except that the object A in the user interface is in a movable state, and the movable state can be displayed by shaking the object A. Without limitation, the display method of the object in the movable state may include but is not limited to: adding a background color of the object, adding a frame of the object, enlarging the display object, etc. The display method of the selected state is different from the display method of the movable state.

[0121] In one embodiment, after the object A is in a state where it can be moved, when it is detected that the position where the user 300 is looking at the display screen moves, for example Figure 4E As shown, the sight line position of the user 300 moves from the object A on the display screen to the point E. At this time, the visual area displayed by the electronic device 100 can move accordingly. The position of the visual area is determined according to the position of the user 300's sight line. During the movement of the visual area, the electronic device 100 can display the object A at different positions in the visual area. It can be represented that these positions can be used as the actual display position of the object A (i.e., the display position after the movement). When the sight line of the user 300 moves to the point E on the display screen, the electronic device 100 can display Figure 4E User interface 440 is shown.

[0122] like Figure 4E As shown, the user interface 440 and Figure 4C The user interface 430 shown is similar, except that the area box 421 in the user interface 430 is at the upper left position, while the area box 441 in the user interface 440 is at the lower right position. The area box 441 in the user interface 440 includes four icons 442. The icons 442 are similar to the icon 411, except that the character "Gallery" is not displayed below the icon 442. The icon 442 is in a moving state, which can indicate that the gallery application is being moved. The positions of the four icons 442 are different, and the icon 411 can be moved to the position of any one of the four icons 442.

[0123] In one embodiment, the electronic device 100 displays Figure 4EIn the user interface 440 shown, the sight position of the user 300 may move within the visual area, for example Figure 4F As shown, when the sight line position of the user 300 moves from the point E of the display screen to the icon in the upper right corner of the visual area (corresponding to the icon 411), the visual area displayed by the electronic device 100 remains unchanged. At this time, the user 300 can operate the electronic device 200 to select any position in the area frame 441 (assuming that it is the position of the icon 442). For example, the electronic device 200 Figure 4F The touch panel shown in FIG. 1 is used by user 300 to operate, for example, Figure 4F As shown, a touch operation is performed on position 1 (e.g., corresponding to the upper right position) in the visual area. The touch operation is, for example, a sliding operation to the upper right. After receiving the operation of the user 300, the electronic device 200 can send the information of the operation of the user 300 to the electronic device 100. The electronic device 100 can determine the position selected by the user 300 (the upper right position) based on the received information of the operation of the user 300, and display Figure 4F The user interface 450 is shown. The user interface 450 and Figure 4C The illustrated user interface 430 is similar, except that the icon 411 in the user interface 430 is located at the upper left position, while the icon 411 in the user interface 450 is located at the lower right position.

[0124] In one embodiment, Figure 4F After the embodiment shown, the user can operate the electronic device 200 to select other positions in the visual area except the position of the selected object, so that the electronic device 100 can deselect the selected object. For example, the electronic device 100 can receive a user operation (such as a touch operation, such as a left sliding operation) on a position other than the position of the icon 411 in the area frame 451 in the user interface 450 through the electronic device 200, and deselect the selected object (icon 411) according to the received user operation. At this time, the electronic device 100 can display and Figure 4F The user interface 450 shown in FIG. 1 is similar to the user interface 450 shown in FIG. 1 , except that the icon 411 in the user interface is not selected. Figure 4F The lower right position (corresponding to the area frame 441) in the user interface 450 shown is moved to the lower left position, and at this time, the visual area displayed by the electronic device 100 can be moved accordingly. Then, the user can operate the electronic device 200 to select any position in the moved visual area so that the electronic device 100 cancels the selection of the selected object. For specific instructions, please refer to the above description of the user operating the electronic device 200 to select other positions in the visual area except the position of the selected object, which will not be repeated here.

[0125] Not limited to the above example, the visual area after the user's line of sight moves is in a blank area. In another embodiment, Figure 4C After the embodiment shown, it is detected that the position where the user 300 is looking at the display screen moves, for example Figure 4G As shown, when the sight line position of the user 300 moves from the object A on the display screen to the point F, the electronic device 100 may display Figure 4G The user interface 460 is shown. The user interface 460 and Figure 4C The user interface 430 shown is similar, except that the area box 461 in the user interface 460 is in the upper right position, and the area box 461 includes four icons 462. The icons 462 are similar to the icon 411, except that the character "Gallery" is not displayed below the icon 462, and the icon 462 is in a moving state, which can indicate that the gallery application is being moved. The positions of these four icons 462 are the positions of the game center application, weather application, housekeeper application, and setting application in the user interface 430, respectively, and the icon 411 can be moved to the position of any one of the four icons 462.

[0126] In one embodiment, the electronic device 100 displays Figure 4G When the user interface 460 is shown, the user 300 can operate the electronic device 200 to select any position in the area frame 461 (assuming that it is the position of the icon 462). For example, the electronic device 100 can receive a user operation on the position of the icon 462 in the area frame 461 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, an operation of sliding to the lower right), determine the position selected by the user (the lower right position) according to the user operation, and display Figure 4H The user interface 470 is shown. The user interface 470 and Figure 4C The user interface 430 shown is similar, except that the icon 411 in the user interface 470 is located at the position of the setting application in the user interface 430, the setting application in the user interface 470 is located at the position of the video application in the user interface 430, the video application in the user interface 470 is located at the position of the camera application in the user interface 430, the camera application in the user interface 470 is located at the position of the file management application in the user interface 430, the file management application in the user interface 470 is located at the position of the calendar application in the user interface 430, and the position of the calendar application in the user interface 470 is different from the position of the calendar application in the user interface 430. It can be understood that when the gallery application moves to the original position of the setting application, the positions of the setting application and other applications (video application, camera application, file management application, calendar application) located after the position of the setting application are moved backward in sequence, wherein the front and back order of the multiple applications may be from top to bottom and from left to right.

[0127] Without limitation, in some other examples, when the gallery application is moved to the original position of the settings application, the settings application may be moved to the position before the gallery application was moved (original position), that is, the positions of the gallery application and the settings application are swapped. Without limitation, in some other examples, when the gallery application is moved to the original position of the settings application, the gallery application and the settings application may be in the same position (that is, the original position of the settings application), and the gallery application and the settings application are in the same folder.

[0128] Not limited to the above example of moving the selected object, in other examples, Figure 4C After the implementation shown, the user can operate the electronic device 200, such as performing a touch operation on a selected object (the touch operation is, for example, a double-click operation), so that the electronic device 100 performs a corresponding operation on the selected object according to the user operation, such as starting an application corresponding to the object, or displaying a function menu corresponding to the object.

[0129] Scenario 2: The electronic device 100 displays an editing interface, which may include multiple different areas. When the electronic device 100 detects that the user is looking at the display screen, it may highlight the user's visual area, which may include at least one editable object in the editing area of ​​the editing interface. For specific examples, see Figure 5A The user can operate the electronic device 200 to select an editable object (such as text content) in the visual area. For specific examples, see Figure 5A (B).

[0130] like Figure 5AAs shown in (A) of , the electronic device 100 may display a user interface 510, and the user interface 510 may be an editing interface of a first application. The user interface 510 may include a toolbar 511, a ribbon 512, a view area 513, and an editing area 514. Among them, the toolbar 511 may include multiple tool options, such as "File" option, "Home" option, "Insert" option, etc. The "Home" option in the user interface 510 is in a selected state. The ribbon 512 may display multiple ribbons under the "Home" option, such as a Clipboard ribbon, a Slide ribbon, a Font ribbon 5121, a Paragraph ribbon, a Drawing ribbon, and an Editing ribbon. The functions included in these multiple ribbons can be used to perform corresponding editing operations on the content displayed on the page 5141 in the editing area 514. The view area 513 may include a slide 5131, and the slide 5131 is in a selected state. The editing area 514 can be used to edit the selected slide 5131, and the page 5141 is an editable page of the slide 5131. The user interface 510 further includes a region box 515, and the region box 515 can represent the user's visual area (assuming that the current position of the user's line of sight is at the character ":"). The region box 515 includes text content 5151, and the text content 5151 displays the character "Reporter: Xiao Wang".

[0131] In one implementation, when it is detected that the position of the user's line of sight on the display screen moves in the editing area 514 / page 5141 (for example, when the position of the user's line of sight moves from within the visual area in the editing area 514 / page 5141 to outside the visual area), the position of the visual area can move correspondingly, that is, the position of the region box 515 moves correspondingly in the editing area 514 / page 5141, and the objects within the region box 515 can also change. For example, when the line of sight position changes (assuming it moves from the character ":" in the region box 515 to the left eye of the image "kitten" on the page 5141), the user interface displayed by the electronic device 100 and Figure 5A the user interface 510 shown in (A) of are similar. The difference is that at this time, the region box 515 in the user interface does not include the text content 5151, but includes the image "kitten". In another implementation, when it is detected that the position of the user's line of sight changes within the visual area, the position of the visual area may not move. For example, when the line of sight position moves from the character ":" in the region box 515 to the character "Wang" in the region box 515, the user interface displayed by the electronic device 100 and Figure 5A the user interface 510 shown in (A) of are the same.

[0132] In one embodiment, the electronic device 100 can receive a user operation on the text content 5151 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, a double-click operation), determine the selected object (text content 5151) according to the received user operation, and display Figure 5A The user interface 520 shown in (B) is shown in FIG. The user interface 520 and Figure 5A The user interface 510 shown in (A) is similar to the user interface 510 shown in (A), except that the text content 5151 in the user interface 520 is in a selected state, and the selected state is displayed by changing the background color of the text content 5151.

[0133] In one embodiment, the electronic device 100 displays Figure 5A When the user interface 520 shown in (B) is displayed, the electronic device 100 can receive a user operation on the text content 5151 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, a long press operation), and set the text content 5151 to a movable state according to the received user operation. Next, when it is detected that the position where the user's line of sight is looking at the display screen moves in the editing area 514 / page 5141, for example, the user's line of sight moves from the character ":" in the area box 515 in the page 5141 to position 2 outside the area box 515 in the page 5141 (assuming that position 2 is a point in the blank area in the upper right corner of the page 5141), at this time, the visual area displayed by the electronic device 100 can move accordingly, and the position of the visual area is determined according to the position of the user's line of sight. During the movement of the visual area, the electronic device 100 can display the selected text content 5151 in the visual area. When the user's line of sight moves to position 2 in the page 5141, the electronic device 100 can display and Figure 5A The user interface is similar to the user interface 520 shown in (B), except that the area box 515 (and text content 5151) in the user interface 520 is in the lower middle position, while the area box 515 (and text content 5151) in this user interface is in the upper right position.

[0134] Without being limited to the above-mentioned example of moving the editable objects in the editing area 514 / page 5141 by combining determining the user's visual area and operating the electronic device 200, in another embodiment, only the electronic device 200 may be operated to achieve movement of the editable objects in the editing area 514 / page 5141 and other refined operations (such as modifying / editing the editable objects).

[0135] Scenario 3: In Scenario 2 Figure 5AAfter (A), the user can operate the electronic device 200 so that the cursor of the electronic device 100 moves within the visual area according to the user's operation. For specific examples, see Figure 5B (A)- Figure 5B (B).

[0136] For example, the electronic device 100 displays Figure 5A When the user interface 510 shown in (A) is used, the user can receive a user operation (such as a touch operation, such as a single-click operation) through the electronic device 200, determine the position of the cursor in the visual area according to the received user operation, and display Figure 5B The user interface 560 shown in (A) is shown in FIG. The user interface 560 and Figure 5A The user interface 510 shown in (A) is similar to the user interface 510, except that a cursor 561 is displayed between the characters "Reporter:" and the characters "Xiao Wang" displayed in the text content 5151 in the area box 515 of the user interface 560, and the cursor 561 can be used to indicate the insertion point of text editing.

[0137] In one embodiment, Figure 5B After the implementation shown in (A) of Figure 5B The user operation (such as a touch operation, such as a right sliding operation) of the cursor 561 in the user interface 560 shown in (A) controls the cursor in the visual area to move rightward according to the received user operation, and at this time, it can be displayed Figure 5B (B) shows a user interface 570. The user interface 570 is similar to the user interface 560, except that a cursor 571 in the user interface 570 is located to the right of the displayed characters "Reporter: Xiao Wang".

[0138] Scene 4: In scene 2 Figure 5A (A) and Figure 5A After (B), when it is detected that the position of the user's gaze on the display screen moves from the editing area 514 / page 5141 to the function area 512, the electronic device 100 can move the position of the visual area accordingly, and the object in the visual area can also change. At this time, the moved visual area may include at least one editing function option in the editing interface. For specific examples, see Fig. 6A The user can operate the electronic device 200 to select any function option in the moved visual area to perform an editing function operation on the selected editable object (such as underlining the text content). For specific examples, see Fig. 6A (B)- Fig. 6A (C).

[0139] For example, when the electronic device 100 detects that the user's gaze on the display screen moves from the character ":" in page 5141 to Figure 5A When the character "U" in the font function area 5121 in the user interface 510 shown in (A) is selected, the electronic device 100 can determine the user's visual area as the area where the font function area 5121 is located, and can display Fig. 6A The user interface 530 shown in (A) is as follows: Figure 5A The user interface 530 is similar to the user interface 520 shown in (B), except that the area box 531 in the user interface 530 is located at the position of the font function area 5121, and the area box 531 in the user interface 530 includes multiple editing function controls, such as control 532 and other controls. Control 532 can be used to add an underline to the selected text content. These multiple editing function controls are all enlarged, which can indicate that these controls are in a state that can be operated (such as selected, clicked).

[0140] In one embodiment, the electronic device 100 can receive a user operation on the control 532 through the electronic device 200 (the user operation is, for example, a touch operation, and the touch operation is, for example, a left sliding operation), determine the selected editing function option (control 532) according to the received user operation, and display Fig. 6A The user interface 540 shown in (B) is shown in FIG. The user interface 540 and Fig. 6A The user interface 530 shown in (A) is similar to the user interface 530, except that in the user interface 540, the control 532 is in a selected state.

[0141] In one embodiment, the electronic device 100 displays Fig. 6A When the user interface 540 shown in (B) is displayed, the electronic device 100 can receive a user operation (such as a touch operation, such as a single-click operation) through the electronic device 200, perform an editing function (such as a function corresponding to the control 532) on the selected editable object (such as text content 5151) according to the received user operation, and display Fig. 6A The user interface 550 shown in (C) is shown in FIG. The user interface 550 and Fig. 6A The user interface 550 is similar to the user interface 540 shown in (B), except that, in the user interface 550, the text content 5151 has been underlined.

[0142] Not limited to the above embodiments, in another embodiment, when it is detected that the position where the user's line of sight is looking at the display screen moves from the editing area 514 / page 5141 to the viewing area 513, the visual area after the movement may include at least one slide in the editing interface. In one embodiment, the electronic device 100 may receive a user operation on slide 1 through the electronic device 200 (the user operation is, for example, a touch operation, such as an upward sliding operation), and determine the selected slide (slide 1) according to the received user operation. At this time, the electronic device 100 may display a user interface in which slide 1 is in a selected state. Then, the electronic device 100 may receive a user operation again through the electronic device 200 (the user operation is, for example, a touch operation, such as a single-click operation), and set the slide displayed in the editing area 514 to slide 1 according to the user operation.

[0143] Not limited to the above embodiments, in another embodiment, when it is detected that the position where the user's line of sight is looking at the display screen moves from the editing area 514 / page 5141 to the toolbar 511, the visual area after the movement may include at least one tool option in the editing interface. In one embodiment, the electronic device 100 may receive a user operation (such as a touch operation, such as a right sliding operation) for tool option 1 (such as the "Insert" option) through the electronic device 200, and determine the selected tool option 1 ("Insert" option) according to the received user operation, and the electronic device 100 may display a user interface in which the tool option 1 is in a selected state. Then, the electronic device 100 may receive a user operation (such as a touch operation, such as a single click operation) through the electronic device 200 again, and set the content displayed in the function area 512 to the multiple functions included in the tool option 1 according to the user operation.

[0144] In one embodiment, the user's visual area may be an area determined according to preset rules based on the user's line of sight position and the user interface currently displayed on the display screen. In some examples, when it is detected that the user's line of sight position is located in an editable area in the user interface currently displayed on the display screen, the electronic device 100 may determine the visual area with the line of sight position as the center and a preset distance as the radius, such as the method of determining the user's visual area in Scene 1, Scene 2, and Scene 3. Not limited to this, in other examples, when it is detected that the user's line of sight position is located in a non-editable area (such as a functional area) in the user interface currently displayed on the display screen, the electronic device 100 may use the functional area where the line of sight position is located as the user's visual area, such as the method of determining the user's visual area in Scene 4. The shape of the visual area may be, but is not limited to, a circle, square, rectangle, triangle, or other polygon, and the present application does not limit this. The size of the visual area can be adjusted by the user operating the electronic device 200. For specific instructions, please refer to the above Fig. 4I The description is not repeated here.

[0145] The visual area is not limited to the above example where one editable object is included in the visual area. In another embodiment, the visual area may also include multiple editable objects, wherein the multiple editable objects may include incomplete and / or complete editable objects. Figure 6B As shown, the electronic device 100 may display a user interface 580, wherein the user interface 580 and Figure 5A The user interface 510 shown in (A) is similar to the user interface 510 shown in (A), except that the area box 581 corresponding to the visual area in the user interface 580 may include three editable objects (text content 581, image 582, image 583), wherein the text content 581 and the image 582 are complete objects, and the image 583 is an incomplete object (a part of the image "a kitten smelling the fragrance of flowers"). The electronic device 100 can receive a user operation (such as a touch operation, such as a double-click operation) for any object (complete / incomplete object) through the electronic device 200, and determine the selected object according to the received user operation. When it is determined that the selected object is a complete object (such as text content 581 and image 582), the complete object can be selected; when it is determined that the selected object is an incomplete object (such as image 583), the complete object to which the incomplete object belongs (such as the image "a kitten smelling the fragrance of flowers") can be selected. Subsequently, the user can operate the electronic device 200 again to enable the electronic device 100 to perform a functional operation on the selected object (complete object).

[0146] The above example is not limited to the case where the electronic device 200 is a touch panel. In other examples, the electronic device 200 may also be a mouse, or other sensors, etc. Not limited to this, in other examples, there may be no electronic device 200, and the above operation on the electronic device 200 may be replaced by an operation of touching the display screen with a finger, or a gesture operation, etc.

[0147] Next, the interaction method provided by the embodiment of the present application is introduced.

[0148] See also Figure 7 , Figure 7 is a flowchart of an interactive method provided in an embodiment of the present application. The method can be applied to Figure 1 The interactive system 10 shown. The method may include but is not limited to the following steps:

[0149] S101: The electronic device 100 displays a first interface.

[0150] In one embodiment, the first interface may be a desktop of the electronic device 100. The first interface may include multiple objects, and the multiple objects may include a first object. The first object in the first interface is in a first position. For example, the first interface is Figure 4A In the user interface 410 shown in FIG. 4 , the first object is an icon 411 in the user interface 410, and the first position is the position of the icon 411 in the user interface 410. For another example, the first interface is Figure 5A In the user interface 510 shown in (A), the first object is the text content 5151 in the user interface 510 , and the first position is the position of the text content 5151 in the user interface 510 .

[0151] In one embodiment, the first interface may include an editable area, and multiple objects may be in the editable area, wherein the editable area means that operations can be performed on objects in the area, for example, objects in the area can be moved, edited, corresponding applications can be started, etc.

[0152] S102: The electronic device 100 detects a second position where the user's gaze is directed, determines a first visual area according to the second position, and highlights the first visual area in the first interface.

[0153] In one embodiment, the second position may be the position where the user's line of sight is focused on the display screen, and the position may be a point, such as the user's line of sight focus point. In some examples, the electronic device 100 may determine the line of sight focus point (i.e., the second position) based on the user's eye features (e.g., the relative positions of the pupil and the Purkinje spot) acquired by the front camera and the distance between the user's eye and the display screen measured by the distance sensor. In other examples, the electronic device 100 may detect the position where the user's line of sight is focused on the display screen through other sensors.

[0154] In one implementation, the electronic device 100 may determine the first visual area based on the second position and the user interface currently displayed on the display screen. In some examples, after the electronic device 100 obtains the second position, it may determine which type of area of ​​the user interface currently displayed on the display screen the second position is located in, and determine the user's first visual area according to the preset rules corresponding to the type of area. For example, when the second position is Figure 4B Point B on the display screen shown in the figure, the user interface 410 currently displayed on the display screen is an editable area, that is, point B is located in the editable area, and the electronic device 100 can determine the first visual area with point B as the center and a preset distance R as the radius. The first visual area is, for example, Figure 4B The area frame 421 in the user interface 420 shown in FIG. Figure 5A The character “:” in the user interface 510 shown in (A) is located in the editable area (editing area 514) of the user interface 510 currently displayed on the display screen. The electronic device 100 can also determine the first visual area with the character “:” as the center and the preset distance T as the radius. The first visual area is, for example, Figure 5A The area frame 515 in the user interface 510 shown in (A) of FIG. It can be understood that the preset rule corresponding to the editable area can be to determine the first visual area with the second position as the center and the preset distance as the radius.

[0155] Without limitation to this, in other examples, the preset rule corresponding to the editable area may be to determine the first visual area according to the number of objects within a preset range of the second position. For example, when the second position is Figure 4C The object A is shown on the display screen, and the object A is located in the editable area. The electronic device 100 can edit the object A and three objects (eg, Figure 4C The range where the clock application, browser application, and address book application in the user interface 430 are located is determined as the first visual area (eg, the area frame 421 in the user interface 430). The present application does not limit the specific manner of the preset rules corresponding to the editable area.

[0156] In one embodiment, the size of the first visual area may be related to the size of the first interface, the size of the objects in the first interface, the number of objects that the user pre-set visual area must include (which may be simply referred to as the number of circled objects), etc. The shape of the first visual area may be but is not limited to a circle, square, rectangle, triangle, or other polygons, etc., and the present application does not impose any limitations on this.

[0157] In one embodiment, the size of the first visual area can be adjusted by the user operating the electronic device 200. For example, the electronic device 200 can send a touch operation (for example, the touch operation is a pinch operation) to the electronic device 100 for the first visual area, and the electronic device 100 can adjust the size of the first visual area according to the touch operation, for example, increase the size of the first visual area. Optionally, the number of objects contained in the first visual area is also increased. In some examples, assuming that the first visual area before adjustment is Figure 4B The area frame 421 in the user interface 420 shown in the figure contains four objects. The adjusted first visual area is Fig. 4I The area box 481 in the user interface 480 shown contains 9 objects.

[0158] In one embodiment, the position of the first visual area may not move: when the electronic device 100 detects that the position of the user's gaze moves within the first visual area, the position of the first visual area may remain unchanged, for example Figure 4C As shown, when the sight line position moves from point B on the display screen to object A, the position of the first visual area does not change. In another embodiment, the position of the first visual area can be moved: when the electronic device 100 detects that the position of the user's sight line moves from within the first visual area to outside the first visual area, the position of the first visual area can be moved accordingly, for example Figure 4E As shown, when the sight line position moves from object A on the display screen to point E, the position of the first visual area can move accordingly.

[0159] In one embodiment, after the electronic device 100 determines the first visual area, the user's operable range can be narrowed from the first interface to within the first visual area. For example, before S102, the cursor of the electronic device 100 is outside the first visual area in the first interface, and after S102, the cursor of the electronic device 100 is within the first visual area in the first interface.

[0160] In one embodiment, the electronic device 100 can highlight the first visual area in the first interface, wherein the method of highlighting the first visual area can include but is not limited to: adding a border to the first visual area, changing the fill color of the first visual area (different from the fill color of other areas in the first interface), enlarging the display of all objects in the first visual area, etc.

[0161] In one implementation, the first visual area may include at least one object in the first interface (assuming that it includes the first object mentioned above), and any one of the objects may be used as an operated object.

[0162] S103: The electronic device 100 sends a first message to the electronic device 200.

[0163] In one implementation, S103 is an optional step.

[0164] In one embodiment, the electronic device 200 may be an external input device of the electronic device 100, such as a mouse, a touchpad, or other sensors. After receiving the first message, the electronic device 200 may switch the current first operation mode of the electronic device 200 to the second operation mode according to the first message, wherein when the first interface displayed by the electronic device 100 does not include a visual area, the electronic device 200 corresponds to the first operation mode, and the user operable range corresponding to the first operation mode is the first interface; when the first interface displayed by the electronic device 100 includes a visual area, the electronic device 200 corresponds to the second operation mode, and the user operable range corresponding to the second operation mode is the visual area.

[0165] S104: The electronic device 200 receives a first user operation on a first object in a first visual area.

[0166] In one embodiment, the electronic device 200 may be an external input device of the electronic device 100, such as a mouse, a touchpad, or other sensors. When the electronic device 200 is a touchpad, the operation mode of the touchpad may be related to the number of objects in the first visual area. For specific examples, see Figure 4D (A)- Figure 4D (C) will not be repeated here.

[0167] S105: The electronic device 200 sends the information of the first user operation to the electronic device 100.

[0168] In one embodiment, the information of the first user operation may include but is not limited to the first operation coordinates and the first operation mode, the first operation coordinates may be used to determine the operation object (first object), and the first operation mode may be used to select the first object and set the state of the first object to a movable state. Without limitation, the electronic device 200 may send the operation object (first object) and the first operation mode for the operation object to the electronic device 100.

[0169] S106: The electronic device 100 sets the first object to a movable state according to the information of the first user operation.

[0170] In one implementation, the electronic device 100 can determine, based on the received information of the first user operation, that the operation object is the first object, the first operation mode is to select the first object and set the state of the first object to be movable, and the electronic device 100 can set the first object to be movable. At this time, the user interface displayed by the electronic device 100 is similar to the first interface, except that the first object in the user interface is in a movable state. In some examples, the user interface can be Figure 4C The user interface 430 shown is similar, except that the icon 411 in the user interface 430 is in a selected state, and the icon 411 (ie, the first object) in the user interface is in a movable state, and the display method of the movable state is different from the display method of the selected state.

[0171] S107: When the electronic device 100 detects that the position where the user's gaze is moving moves from the second position to the third position, the electronic device 100 moves the first object from the first visual area to the second visual area.

[0172] In one embodiment, when the electronic device 100 detects that the user's line of sight moves from the second position, it can be determined whether the type of area where the line of sight is located in the first interface displayed on the current display screen during the movement changes. Figure 7 Take the example that the area type corresponding to the line of sight position during the movement does not change (all are editable areas), that is, the second position and the third position are both located in the editable area, and the other positions in the process of moving from the second position to the third position are also located in the editable area. Therefore, the method of determining the visual area during the movement can be the same, and the size of the visual area can remain unchanged.

[0173] In one embodiment, when the first object in the first visual area is in a movable state, when the electronic device 100 detects that the position where the user's line of sight is looking has moved, the first object and the visual area where the first object is located can be moved, wherein before the first object is moved, the visual area where the first object is located is the first visual area, and during the process of moving the first object, the visual area where the first object is located is the second visual area. The first visual area may include a first object, and optionally other objects, and the second visual area may include multiple first objects located at different positions. Assuming that the first objects are located at N positions respectively, it can be represented that the first object can be moved to any one of the N positions in the second visual area. In some examples, the second position is, for example, Figure 4E The third position of the object A on the display screen is, for example, Figure 4E Point E on the display screen shown, the second visual area is, for example, Figure 4E The area box 441 in the user interface 440 shown includes first objects (corresponding to icons 442 ) at four locations.

[0174] In one embodiment, when the first object in the first visual area is in a movable state, when the electronic device 100 detects that the position where the user's line of sight is being focused has moved, the first visual area can be changed to a second visual area, and then the second visual area can be moved accordingly according to the trajectory of the user's line of sight. It can be understood that when the user's line of sight moves, the visual area displayed by the electronic device 100 can move accordingly. During the movement of the visual area, the electronic device 100 can display the first object at different positions in the visual area, which can indicate that these positions can all be used as the display positions after the first object is moved, and the number of these positions can be the same as the number of circled objects in the visual area.

[0175] S108: The electronic device 200 receives a second user operation on a fourth position within the second visual area.

[0176] In one implementation, the second visual area may include a plurality of first objects located at different positions, and a first object at any one position (assuming the fourth position) may be selected to move the first object to the fourth position.

[0177] In one embodiment, the description of S108 and Figure 7 The instructions for S104 are similar, please refer to Figure 7 The description of S104 is omitted here.

[0178] S109: The electronic device 200 sends the information of the second user operation to the electronic device 100.

[0179] In one embodiment, the information of the second user operation may include but is not limited to second operation coordinates and a second operation mode. The second operation coordinates may be used to determine the position after movement (fourth position), and the second operation mode may be used to move the first object to the fourth position.

[0180] S110: The electronic device 100 displays the first object at a fourth position in the first interface according to the information of the second user operation.

[0181] In one embodiment, the electronic device 100 may move the first object to the fourth position according to the received information of the second user operation, and display the first object at the fourth position in the first interface. It can be understood that the electronic device 100 moves the first object from the first position in the first interface to the fourth position. In some examples, the fourth position is, for example, Figure 4E The position of the icon 442 in the user interface 440 shown in FIG. 4 is a first object displayed in the fourth position. Figure 4F In the user interface 450 shown, the icon 411 (ie, the first object) in the user interface 450 is at the fourth position.

[0182] Not limited to the above embodiment, in another embodiment, Figure 7 S107 can be replaced by: the electronic device 200 can receive a third user operation on the first visual area and send information about the third user operation to the electronic device 100. The electronic device 100 can move the first visual area according to the third user operation to obtain the moved second visual area, and move the first object from the first visual area to the second visual area.

[0183] Not limited to the above embodiment, in another embodiment, Figure 7 S102 can be replaced with: the electronic device 100 can obtain the cursor position in the first interface, and the cursor position can be determined by user operation, for example, the user operates the electronic device 200 to control the cursor position of the electronic device 100, etc., and the first area is determined according to the cursor position, and the first area (including the first object) is highlighted in the first interface. The instructions for determining the first area according to the cursor position can be found in the specific instructions for determining the first visual area according to the second position in S102, and will not be repeated here.

[0184] Not limited to the above embodiment, in another embodiment, Figure 7 After S105, when the first object is in a selected state, the electronic device 200 can receive a fourth user operation on the first object and send information about the fourth user operation to the electronic device 100. The electronic device 100 can perform a corresponding operation on the first object according to the fourth user operation, such as starting an application corresponding to the first object, or displaying a function menu corresponding to the first object.

[0185] Figure 7 For example, see the above-mentioned scenes 1, 2 and 3, where in scene 1, the first object is an icon, for example Figure 4A The icon 411 in the user interface 410 shown in the first visual area is Figure 4B The area frame 421 in the user interface 420 shown in the figure, the second visual area is Figure 4E The area box 441 in the user interface 440 is shown. In scenario 2, the first object is an editable object, for example Figure 5A The text content 5151 in the user interface 510 shown in (A) is the first visual area. Figure 5A In the area frame 515 in the user interface 510 shown in (A), in scenario 3, the first object is a cursor, for example Figure 5B The cursor 561 in the user interface 560 shown in (A) is a first visual area. Figure 5B The area box 515 in the user interface 570 shown in (A).

[0186] exist Figure 7In the method shown, the electronic device 100 is connected to the electronic device 200, and the user can control the electronic device 100 by operating the electronic device 200, and the electronic device 200 can be understood as an external input device of the electronic device 100. The electronic device 100 can display a first interface, and when it is detected that the user's line of sight is fixed on the screen, the first visual area is determined according to the second position of the user's line of sight. The electronic device 100 can highlight the first visual area in the first interface, and optionally at least one object in the first visual area, at which time the user's operable range is narrowed from the first interface to the first visual area. Then, the electronic device 100 can obtain the user operation for the first object in the first visual area through the electronic device 200, and perform corresponding processing on the first object according to the user operation (for example, moving the position, starting the corresponding application, displaying the relevant function menu, etc.). In this way, the user can first "roughly select and locate" the area to be operated (i.e., the above-mentioned first visual area) by gazing at the line of sight, and then "selectively locate" the object to be operated in the area (i.e., the above-mentioned first object) through the electronic device 200. Therefore, the user does not need to move and operate the electronic device 200 many times, and can avoid the situation where the hand is easily fatigued due to many movements, cumbersome operations, and long operation time. In addition, the user does not need to slide his fingers on the electronic device 200 (such as a touchpad) many times and accurately select objects. Through the method of the present application, the fatigue of the user's hands when performing interactive operations can be effectively reduced, allowing the user to achieve accurate interactive operations with less effort, improve the rate during the interactive operation process, thereby improving the interaction efficiency, and this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0187] See also Figure 8 , Figure 8 is a flowchart of another interactive method provided in an embodiment of the present application. This method can be applied to Figure 1 The interactive system 10 shown. The method may include but is not limited to the following steps:

[0188] S201: The electronic device 100 displays a second interface.

[0189] In one implementation, the second interface may be an editing interface of the first application, the second interface may include at least one object, and the at least one object may include a second object. Figure 5A In the user interface 510 shown in (A) (excluding the area frame 515 ), the second object may be text content 5151 in the user interface 510 .

[0190] In one embodiment, the second interface may include multiple different types of areas, such as an editable area (editing area), a functional area, a view area, etc. These multiple different types of areas may be divided by the first application according to its own functional options and the nature of the interactive target. For example, the user interface 510 (excluding the area frame 515) may include an editable area (editing area 514), a functional area (functional area 512 and toolbar 511), and a view area (view area 513), wherein the editing area 514 may be used to operate and display editable objects, and the functional area 512 may be used to perform functional operations corresponding to functional options on editable objects.

[0191] S202: The electronic device 100 detects a fifth position where the user's gaze is directed, determines a third visual area according to the fifth position, and highlights the third visual area in the second interface.

[0192] In one embodiment, the description of the fifth position and Figure 7 The description of the second position in S102 is similar, see Figure 7 The description of the second position in S102 is omitted. The fifth position may be located in an editable area in the second interface.

[0193] In one implementation, the electronic device 100 may determine the third visual area based on the fifth position and the second interface currently displayed on the display screen. In some examples, after obtaining the fifth position, the electronic device 100 may determine which type of area in the second interface the fifth position is located in, and determine the user's third visual area according to the preset rules corresponding to the type of area. For example, when the fifth position is Figure 5A The character “:” in the user interface 510 shown in (A) determines that the fifth position is located in the editable area (editing area) in the second interface. The electronic device 100 can determine the third visual area according to the preset rule corresponding to the editable area. The third visual area is, for example, Figure 5A The area frame 515 in the user interface 510 shown in (A) of FIG. 1 is a block diagram of a user interface 510. For examples of preset rules corresponding to editable areas, see Figure 7 The description in S102 is not repeated here.

[0194] In one embodiment, the third visual area is an area within the editing area in the second interface. The third visual area may include at least one editable object / content (assuming that it includes the second object), and any editable object / content may be used as an operated object.

[0195] In one embodiment, the description of S202 and Figure 7The description of S102 is similar, the difference is that the size of the third visual area can be related to the size of the editing area in the second interface, the size of the object in the editing area, the size of the visual area pre-set by the user, etc., which will not be repeated.

[0196] S203: The electronic device 200 receives a fifth user operation on the second object in the third visual area.

[0197] In one embodiment, the description of S203 and Figure 7 The instructions for S104 are similar, please refer to Figure 7 The description of S104 is omitted here.

[0198] S204 : The electronic device 200 sends the information of the fifth user operation to the electronic device 100 .

[0199] In one implementation, the fifth user operation information may include but is not limited to third operation coordinates and a third operation mode. The third operation coordinates may be used to determine the operation object (second object), and the third operation mode may be used to select the second object.

[0200] S205: The electronic device 100 selects a second object according to the information of the fifth user operation.

[0201] In one implementation, the electronic device 100 may determine, based on the received information of the fifth user operation, that the operation object is the second object, and the third operation mode is to select the second object. The electronic device 100 may set the second object to a selected state, and at this time the electronic device 100 may display an interface in which the second object is selected, for example, Figure 5A In the user interface 520 shown in (B), the text content 5151 (ie, the second object) in the user interface 520 is in a selected state.

[0202] S206: When the electronic device 100 detects that the position where the user's gaze moves from the fifth position to the sixth position, the fourth visual area is determined according to the sixth position, and the fourth visual area is highlighted in the second interface.

[0203] In one embodiment, the description of the sixth position and Figure 8 The description of the fifth position in S202 is similar, except that the sixth position can be located in the function area in the second interface.

[0204] In one embodiment, when the second object is selected, when the electronic device 100 detects that the user's line of sight moves from the fifth position, it can be determined whether the type of area where the line of sight is located in the second interface displayed on the current display screen during the movement has changed. If it has changed, the electronic device 100 can determine the visual area based on the changed type area (where the line of sight is located). In some examples, the fifth position is located in an editable area in the second interface, and the sixth position after the movement is, for example, Figure 5A The character "U" in the user interface 510 shown in (A) of FIG. 1 may be located in the function area in the second interface at the sixth position. Therefore, the electronic device 100 may determine the fourth visual area according to the sixth position according to the preset rule corresponding to the function area. In some examples, the preset rule corresponding to the function area may be to use the function area where the sixth position is located as the fourth visual area, for example, to determine the font function area 5121 where the character "U" is located as the fourth visual area. The fourth visual area is, for example, Fig. 6A The area box 531 in the user interface 530 shown in (A).

[0205] Without limitation to this, in other examples, when the second object is selected, assuming that the sixth position after the move is Figure 5A A point in the upper right blank area of ​​page 5141 in the user interface 510 shown in (A), at this time, the sixth position is located in the editable area in the second interface. Therefore, the electronic device 100 can determine the visual area after moving according to the preset rules corresponding to the editable area. The size and shape of the visual area after moving are the same as those of the visual area before moving. The difference is that the object in the visual area after moving changes.

[0206] In one embodiment, the fourth visual area is an area within the functional area in the second interface. The fourth visual area may include at least one editing function option (assuming that it includes a third object). An editing function option may indicate an editing function. Any editing function option may be selected to perform a corresponding functional operation on the selected editable object / content.

[0207] In one embodiment, the description of S206 and Figure 7 The description of S107 is similar to that of S107, except that, in the process of the user's line of sight moving from the fifth position to the sixth position, the type of area where the line of sight is located changes (for example, from the editing area to the functional area). Therefore, the method of determining the visual area may change, the size of the visual area may change, and the number of objects in the visual area may change.

[0208] It can be understood that when the electronic device 100 displays the editing interface, the user's eyes focus on different types of areas in the editing interface, and the number of objects enclosed by the visual areas corresponding to different types of areas may be different. For example, Figure 5A The editable object enclosed by the area frame 515 (in the editing area 514) in the user interface 510 shown in (A) includes only one (i.e., text content 5151). Fig. 6A The editing function options enclosed by the area box 531 (in the function area 512) in the user interface 530 shown in (A) include multiple (ie, all the controls included in the area box 531).

[0209] Without limitation to this, in another embodiment, when the second object is in a movable state (for example, after S205, the electronic device 200 receives a touch operation (for example, a long press operation) for the second object and sends the touch operation to the electronic device 100, and after receiving the touch operation, the electronic device 100 sets the second object to a movable state), if it is detected that the user's line of sight moves from the fifth position (located in the editing area) to the seventh position (located in the editing area), that is, the area where the line of sight before and after the movement is located is the editing area, the electronic device 100 may subsequently move the display position of the second object. For specific instructions, see Figure 7 S107-S110 are not described in detail.

[0210] S207: The electronic device 200 receives a sixth user operation on the third object in the fourth visual area.

[0211] In one embodiment, the description of S207 and Figure 7 The instructions for S104 are similar, please refer to Figure 7 The description of S104 is omitted here.

[0212] S208 : The electronic device 200 sends the information of the sixth user operation to the electronic device 100 .

[0213] In one embodiment, the information of the sixth user operation may include but is not limited to fourth operation coordinates and a fourth operation mode. The fourth operation coordinates can be used to determine the operation object (third object), and the fourth operation mode is used to select the third object and perform the functional operation indicated by the third object.

[0214] S209: The electronic device 100 edits the second object according to the editing function indicated by the third object.

[0215] In one implementation, the electronic device 100 can determine, based on the information of the sixth user operation, that an editing function (selected) operation is performed on the selected editable object, and the electronic device 100 can edit the selected second object according to the editing function indicated by the third object, and obtain the processed object. For example, the third object is Fig. 6A In the control 532 in the user interface 530 shown in (A), the editing function indicated by the third object is to underline the selected editable object, and the electronic device 100 can underline the selected second object (for example Figure 5A The text content 5151 in the user interface 520 shown in (B) is underlined, and the processed object is, for example, Fig. 6A The text content 5151 in the user interface 550 shown in (C).

[0216] Not limited to the above embodiment, in another embodiment, Figure 8 S202 can be replaced with: the electronic device 100 can obtain the cursor position in the second interface, and the cursor position can be determined by user operation, for example, the user operates the electronic device 200 to control the cursor position of the electronic device 100, etc., and the second area is determined according to the cursor position, and the second area (including the second object) is highlighted in the second interface. The instructions for determining the second area according to the cursor position can be found in the specific instructions for determining the third visual area according to the fifth position in S202, and will not be repeated here.

[0217] Figure 8 For example, see the above scenario 4, where the second interface is the editing interface of the first application, for example Figure 5A In the user interface 510 (excluding the area frame 515) shown in (A), the second object is an editable object, for example Figure 5A The text content 5151 in the user interface 510 shown in (A) of FIG. 5 , the third visual area is the area within the editing area, for example Figure 5A In the area frame 515 in the user interface 510 shown in (A), the fourth visual area is the area within the functional area, for example Fig. 6A In the area box 531 of the user interface 530 shown in (A), the third object is an editing function option, for example Fig. 6A Control 532 in user interface 530 shown in (A).

[0218] exist Figure 8 In the method shown, the electronic device 100 can select the object to be operated and operate the editing function options of the object by combining sight and operation to realize the editing function, which can effectively improve the editing efficiency of the user. In addition, this method can meet the interaction needs of different users in different scenarios and expand the scope of applicable scenarios.

[0219] See also Fig. 9 , Fig. 9 is a flowchart of another interactive method provided in an embodiment of the present application. This method can be applied to Figure 1 The interactive system 10 shown. The method may include but is not limited to the following steps:

[0220] S301: The first device displays a first interface.

[0221] In one embodiment, the first interface may be a desktop of the first device. The first interface may include multiple objects, and the multiple objects may include the first object. For a detailed description of the first interface, see Figure 7 The description of the first interface in S101 is omitted here.

[0222] In another embodiment, the first interface may be an editing interface of the first application. For a description of the first interface, see Figure 8 The description of the second interface in S201 is omitted here.

[0223] In one embodiment, the first interface may include at least one type of area, such as an editable area, a function area, a view area, etc.

[0224] S302: The first device detects that the user is looking at a first position on the display screen, and determines a first visual area according to the first position.

[0225] In one embodiment, the first position can be described in detail in Figure 7 The description of the second position in S102 is omitted here.

[0226] In one embodiment, the first position is located in a first type of area, and the first type of area is, for example, an editable area. The first device can determine the first visual area according to the preset rules corresponding to the first position and the first type of area. In some examples, the first device can determine the first visual area with the first position as the center and the preset distance as the radius. In other examples, the first device can determine the first visual area according to the number of objects within the preset range of the first position. For specific instructions, see Figure 7 The description of determining the first visual area according to the second position in S102 is not repeated here.

[0227] In one embodiment, the first visual area may include at least one object among the plurality of objects, the at least one object including the first object, wherein the number of the at least one object in the first visual area is N, the number of the plurality of objects in the first interface is M, N and M are both positive integers, N is less than M, and the specific description of the first visual area can be found in Figure 7 The description of the first visual area in S102 is omitted here.

[0228] S303: The first device displays a first visual area in a first interface.

[0229] In one implementation, the description of S303 can be found in Figure 7 The specific instructions for highlighting the first visual area in the first interface in S102 are not repeated here.

[0230] In one embodiment, when the first device displays the first interface, the area in which the second device acts includes M objects in the first interface; when the first device displays the first visual area in the first interface, the area in which the second device acts includes N objects.

[0231] S304: The first device receives the first information sent by the second device.

[0232] In one embodiment, the second device may be an external input device of the first device. For a description of the second device, see Figure 7 The description of S104 is omitted here.

[0233] In one embodiment, the description of the first information can be found in Figure 7 The specific description of the first user operation information in S105 is omitted here.

[0234] S305: The first device determines, according to the first information, that the operation object is the first object.

[0235] S306: The first device performs a first operation on the first object according to the first information.

[0236] In one implementation, the first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

[0237] In one embodiment, the first operation is a selection operation. After S306, when the first device detects that the position where the user's gaze is looking at the display screen moves to a second position, the first device can determine the second visual area based on the second position.

[0238] In one embodiment, the first device can determine whether the second type area where the second position is located is the same type as the first type area (e.g., editable area) where the first position is located. When the second type area is the same type as the first type area, the second visual area includes N first objects, and the N first objects are respectively located at N positions in the second visual area. For details, see Figure 7The description of the second visual area in S107 is omitted. Next, the first device may receive the second information sent by the second device, determine the third position in the second visual area according to the second information, the third position is any one of the N positions, and the first device may display the first object at the third position, which is different from the position of the first object in the first interface. For details, see Figure 7 The description of S108-S110 will not be repeated here.

[0239] In one embodiment, when the second type area and the first type area are of different types, the second visual area includes at least one functional option. For details, see Figure 8 The description of the fourth visual area in S206 is omitted. Next, the first device may receive the third information sent by the second device, determine the operation object as the first function option according to the third information, the first function option is any one of the at least one function option, and the first device may perform the second operation on the first object according to the third information, the second operation is the operation indicated by the first function option, and the specific description can be found in Figure 8 S207-S209.

[0240] exist Fig. 9 The method shown can effectively reduce the fatigue of the user's hands when performing interactive operations, allowing the user to achieve precise interactive operations with less effort, thereby increasing the rate of the interactive operation process and thus improving the interaction efficiency. In addition, this method can achieve a faster and more intuitive operating experience, further improving the user experience.

[0241] The methods provided in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DWD), or a semiconductor medium (e.g., a solid state drive (SSD)). As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive method, characterized in that: Applied to a first device, the first device is configured with a display screen, the first device is connected to a second device, and the method includes: Displaying a first interface, wherein the first interface includes a plurality of objects; Detecting that the user is looking at a first position of the display screen, and determining a first visual area according to the first position; Displaying the first visual area in the first interface, the first visual area includes at least one object among the multiple objects, the at least one object includes a first object, the number of the at least one object is N, the number of the multiple objects is M, N is less than M, and both N and M are positive integers; receiving first information sent by the second device; Determine, according to the first information, that the operation object is the first object; A first operation is performed on the first object according to the first information.

2. The method according to claim 1, characterized in that The first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and determining the first visual area according to the first position includes: The first visual area is determined according to the first position and a preset rule corresponding to the first type of area.

3. The method according to claim 2, characterized in that The determining the first visual area according to the first position and the preset rule corresponding to the first type of area includes: The first visual area is determined with the first position as the center and a preset distance as the radius; or, The first visual area is determined according to the number of objects within a preset range of the first position.

4. The method according to claim 2, characterized in that The first type of area includes a first functional area, and determining the first visual area according to the first position and a preset rule corresponding to the first type of area includes: When the first position is located in the first functional area, the first functional area is set as the first visual area.

5. The method according to any one of claims 1 to 4, characterized in that: The first operation is a selection operation; After performing the first operation on the first object according to the first information, the method further includes: When it is detected that the position where the user's line of sight is gazing at the display screen moves to a second position, determining a second visual area according to the second position; The second visual area is displayed in the first interface.

6. The method according to claim 5, characterized in that The first interface includes at least one type of area, the first position is located in the first type of area, the first type of area is any one of the at least one type of area, and the method further includes: Determine whether the second type area where the second position is located is of the same type as the first type area; When the second type area and the first type area are of the same type, the second visual area includes the N first objects, and the N first objects are respectively located at the N positions in the second visual area; When the second type area and the first type area are of different types, the second visual area includes at least one functional option.

7. The method according to claim 5 or 6, characterized in that When the second type area is of the same type as the first type area, after displaying the second visual area in the first interface, the method further includes: receiving second information sent by the second device; Determine a third position in the second visual area according to the second information, where the third position is any one of the N positions; The first object is displayed at the third position, and the third position is different from the position of the first object in the first interface.

8. The method according to claim 5 or 6, characterized in that When the second type area is different from the first type area in type, after displaying the second visual area in the first interface, the method further includes: receiving third information sent by the second device; Determine, according to the third information, that the operation object is a first function option, where the first function option is any one of the at least one function option; A second operation is performed on the first object according to the third information, where the second operation is the operation indicated by the first function option.

9. The method according to any one of claims 1 to 8, characterized in that The manner of displaying the first visual area in the first interface includes at least one of the following: adding a border to the first visual area, changing a fill color of the first visual area, and enlarging and displaying the at least one object in the first visual area.

10. The method according to any one of claims 1 to 9, characterized in that: The size of the first visual area is related to at least one of the following: the size of the first interface, the size of the multiple objects, the number of the at least one object, and the position of the first visual area.

11. The method according to any one of claims 1 to 10, characterized in that: The first operation includes at least one of the following: selecting, displaying an interface corresponding to the first object, displaying a function menu related to the first object, and moving a display position.

12. The method according to any one of claims 1 to 11, characterized in that: The first object is any one of the following: an icon, an editing object, or a function option.

13. The method according to any one of claims 1 to 12, characterized in that: The first operation includes a selection operation and a moving display position; After performing the first operation on the first object according to the first information, the method further includes: A second interface is displayed, wherein a position of the first object in the second interface is different from a position of the first object in the first interface.

14. The method according to any one of claims 1 to 13, characterized in that: The first operation includes a selection operation and displaying an interface corresponding to the first object; After performing the first operation on the first object according to the first information, the method further includes: A third interface is displayed, where the third interface is an interface corresponding to the first object.

15. The method according to any one of claims 1 to 14, characterized in that: The first operation includes a selection operation and displaying a function menu related to the first object; After performing the first operation on the first object according to the first information, the method further includes: A fourth interface is displayed, wherein the fourth interface includes a function menu related to the first object.

16. The method according to any one of claims 1 to 15, characterized in that: The second device is any one of the following: a mouse, a touch pad, or a sensor.

17. The method according to any one of claims 1 to 16, characterized in that: When the first interface is displayed, the area where the second device acts includes the M objects; when the first visual area is displayed in the first interface, the area where the second device acts includes the N objects.

18. The method according to any one of claims 1 to 17, characterized in that: The at least one object includes the first object and the second object, the distance between the position of the first object and the position of the second object in the first visual area is a first distance, and the first object in the first visual area is in a selected state; the method further includes: Determine a second distance acting on the second device according to the first information and the first algorithm, wherein the second distance is smaller than the first distance; The selected object in the first visual area is switched to the second object according to the second distance.

19. The method according to any one of claims 1 to 18, characterized in that: The connection mode between the first device and the second device includes any one of the following: wired connection and wireless connection.

20. An interactive method, characterized in that: Applied to a second device, the second device is connected to a first device, the first device is configured with a display screen, and the operation mode of the second device includes a first mode and a second mode. The method includes: receiving a first message sent by the first device; According to the first message, the operation mode of the second device is switched from the first mode to the second mode, wherein the first mode is a mode corresponding to the second device when the interface displayed by the first device does not include a visual area, and the second mode is a mode corresponding to the second device when the interface displayed by the first device includes the visual area.

21. An electronic device, characterized in that: The method comprises a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the method according to any one of claims 1 to 19 or any one of claim 20.

22. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 20 is implemented.

Images