Device control method, related device and communication system

Abstract

The invention discloses a device control method, a related device and a communication system, a user wears a control device (such as intelligent glasses), and the control device can obtain eyeball actions (such as eyeball gazing directions) of the user. The controlled device can be comprehensively determined based on the eyeball movement of the user and the image information and the position information of the device in the watching direction of the user, communication connection with the controlled device is established, the control device can further obtain control operation executed by the user in the modes of eye movement, gestures, voice, expressions and the like, and a control instruction is determined based on the control operation; further, the control device can send the control instruction to the controlled device based on the communication connection, and the controlled device can respond to the control instruction after receiving the control instruction. In this way, the user can directly control the controlled device as long as seeing the controlled device with the eyes, user operation is simplified, the efficiency and accuracy of device control are improved, user experience is improved, and the effect that the device can be controlled by being visible can be achieved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a device control method, related apparatus and communication system. Background Technology

[0002] With the rapid development of terminal technology, more and more smart devices (such as smart home devices) are entering users' lives. Currently, controlling these smart devices is cumbersome for users, resulting in a poor user experience. Summary of the Invention

[0003] This application provides a device control method, related apparatus, and communication system. The device control method can simplify user operation, improve the efficiency and accuracy of device control, enhance user experience, and achieve the effect of making the device visible and controllable.

[0004] In a first aspect, embodiments of this application provide a device control method applied to a first electronic device. The method includes: the first electronic device acquiring a user's eye movements, the eye movements including the eye gaze direction; the first electronic device using a second electronic device that meets a first condition in the eye gaze direction as a controlled device; and the first electronic device controlling the second electronic device; wherein the first condition includes: the distance between the second electronic device and the first electronic device is less than a first distance threshold, and / or, the device type of the second electronic device is the same as the first device type, the first device type being determined by the first electronic device based on an image of the second electronic device captured by a camera, and the device type of the second electronic device being obtained by the first electronic device from device information sent by the second electronic device.

[0005] The first electronic device can be electronic device 100, the second electronic device can be electronic device 200, and the first distance threshold can be a preset distance threshold.

[0006] By implementing the method provided in the first aspect, after a user wears a control device (i.e., a first electronic device, such as smart glasses), the control device can acquire the user's eye movements (e.g., eye gaze direction) and comprehensively determine the controlled device based on the user's eye movements, the image information and / or position information of the device in the user's gaze direction, and further control the controlled device. In this way, the user can directly control the controlled device simply by looking at it, simplifying user operation, improving the efficiency and accuracy of device control, enhancing the user experience, and achieving the effect of "the device is controllable as soon as it is visible."

[0007] In one possible implementation, the first electronic device is smart glasses.

[0008] In one possible implementation, the second electronic device is a screenless smart device, and the first electronic device controls the second electronic device, specifically including: the first electronic device outputting prompt information, the prompt information including the control operations that the second electronic device supports the user to perform; the first electronic device acquiring the user's control operations and determining control instructions based on the control operations; and the first electronic device sending control instructions to the second electronic device to control the second electronic device to perform the operations corresponding to the control instructions.

[0009] In this way, users can directly and accurately control the screenless smart device simply by looking at it, simplifying user operation, improving device control efficiency, and enhancing user experience, achieving the effect of "the device is controllable as long as it is visible".

[0010] In one possible implementation, the screenless smart device may include, but is not limited to, any of the following: light bulbs, air conditioners, and speakers.

[0011] In one possible implementation, the second electronic device is a smart device with a screen, and the first electronic device controls the second electronic device, specifically including: the first electronic device determining a first position of the gaze point on the screen of the second electronic device based on the user's eye gaze direction; the first electronic device sending a first control command to the second electronic device to control the second electronic device to focus on the content displayed at the first position.

[0012] In one possible implementation, a smart device with a screen includes any of the following: mobile phone, tablet, computer, smart screen, or in-vehicle system.

[0013] In one possible implementation, the method further includes: a first electronic device acquiring a user's control operation on the focused content, and determining a second control instruction based on the control operation; the first electronic device sending the second control instruction to a second electronic device to control the second electronic device to perform the operation corresponding to the second control instruction on the focused content.

[0014] In this way, users can directly and accurately control the displayed content simply by looking at it on the screen of the smart device. This achieves the same effect as directly operating the smart device with a keyboard, mouse, or touchscreen, freeing up the user's hands, simplifying operation, improving device control efficiency, and providing convenience and a better user experience.

[0015] In one possible implementation, both the first electronic device and the second electronic device are associated with a first account. The second electronic device is in a locked state, and the first electronic device controls the second electronic device. Specifically, the first electronic device obtains the user's unlock operation and determines an unlock command based on the unlock operation. The first electronic device sends an unlock command to the second electronic device to control the second electronic device to perform the unlock operation.

[0016] In this way, when the controlling device and the controlled device are logged into the same account (such as the first account mentioned above), the user can unlock the controlled device simply by looking at it while it is locked. This simplifies user operations, improves the efficiency and security of device unlocking, enhances the user experience, and solves the problems of poor user experience and security risks in identity authentication caused by collaborative unlocking through proximity sensing.

[0017] In one possible implementation, the second electronic device is a barcode scanning device, and the first electronic device controls the second electronic device, specifically including: the first electronic device acquiring the user's first control operation, determining a control instruction based on the first control operation, and displaying the first barcode; the first electronic device sending a control instruction to the second electronic device to control the second electronic device to perform a barcode scanning operation on the first barcode.

[0018] The first control operation mentioned above may be one of the following in the embodiments. Figure 6 Control operation 1 is shown.

[0019] In one possible implementation, the method further includes: a first electronic device acquiring a second control operation from a user; and the first electronic device performing a scanning operation on a second barcode on a second electronic device based on the second control operation.

[0020] The second control operation described above may be as follows in the embodiments. Figure 6 Control operation 2 is shown.

[0021] In this way, users can make the control device / scanning device perform scanning or be scanned simply by looking at the scanning device, thus completing the corresponding business (such as payment, identity authentication, etc.) conveniently and quickly, simplifying user operations, improving business execution efficiency, and enhancing user experience.

[0022] In one possible implementation, the control operation can be performed in one or more of the following ways: eye movement, gesture, voice, and facial expression.

[0023] In one possible implementation, before the first electronic device controls the second electronic device, the method further includes: establishing a communication connection between the first electronic device and the second electronic device, wherein the device type of the second electronic device is obtained by the first electronic device from device information sent by the second electronic device through the communication connection.

[0024] In a second aspect, embodiments of this application provide a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, causing the chip system to perform the method described in any possible implementation of the first aspect above.

[0025] Thirdly, embodiments of this application provide an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method described in any possible implementation of the first aspect above.

[0026] Fourthly, embodiments of this application provide a computer storage medium storing a computer program, the computer program including program instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any possible implementation of the first aspect above.

[0027] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method described in any possible implementation of the first aspect above. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a communication system provided in an embodiment of this application;

[0029] Figure 2 This is a schematic flowchart of a device control method provided in an embodiment of this application;

[0030] Figures 3A-3B This is a schematic diagram of a set of output prompt messages provided in an embodiment of this application;

[0031] Figure 4 This is a schematic flowchart of another device control method provided in an embodiment of this application;

[0032] Figures 5A-5D These are schematic diagrams of a set of user interfaces provided in the embodiments of this application;

[0033] Figure 6 This is a schematic flowchart of another device control method provided in an embodiment of this application;

[0034] Figures 7A-7B This is a schematic diagram of a set of scanning scenarios provided in the embodiments of this application;

[0035] Figure 8 This is a schematic flowchart of another device control method provided in an embodiment of this application;

[0036] Figures 9A-9B This is a schematic diagram of a set of user interfaces provided in an embodiment of this application;

[0037] Figure 10 This is a schematic flowchart of another device control method provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0040] It should be understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0041] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0042] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0043] In traditional device control methods, people need to move to the device to perform control operations. For example, with a light bulb, a person needs to move to the power switch and manually operate it to control the light's on and off. Similarly, with a computer, a person needs to move to the computer and manually operate the windows or objects displayed on the screen using a touchscreen or keyboard and mouse. These are traditional, object-centric interaction methods that are inefficient and provide a poor user experience. People cannot control devices in various directions and at different distances while standing or sitting still, especially since it is impossible to effectively control various types of devices at a relatively long distance (e.g., 1-10 meters) within a room using a single method or device.

[0044] Currently, users can control devices through smart device control applications on mobile phones, tablets, and other electronic devices. However, this control method has several drawbacks. First, it typically only supports controlling screenless smart devices by turning them on and off, and does not support direct interface control of smart devices with screens. Second, it requires users to perform relatively cumbersome steps (such as finding the application, opening the application, finding the desired device card, and opening the desired device card) to complete the entire control process. Third, when there are multiple identical or similar devices in the same space, such as multiple sets of lights in a living room (e.g., living room chandeliers, light strips, ceiling lights, sideboard lights, TV spotlights, main wall ambient lights, etc.), it is difficult for users to accurately remember the names of each set of lights, making it difficult to find the corresponding device card. This results in low device control efficiency and a poor user experience.

[0045] For controlling smart devices with screens, there is currently a screen mirroring technology that can mirror the interface content of the controlled device onto the screen of the controlling device. Users can then perform control operations on the screen of the controlling device. However, this method is usually used for remote control of smart devices with screens and cannot control smart devices without screens. In addition, since the screen sizes of the controlling device and the controlled device may differ, the display effect of the interface content will also be different, resulting in a poor user experience.

[0046] To address the aforementioned issues, this application provides a device control method. A user wears a control device (e.g., smart glasses). The control device can acquire the user's eye movements (e.g., eye gaze direction) and comprehensively determine the controlled device based on the user's eye movements, image information of the device in the user's gaze direction, and its location (or spatial position / coordinates). A communication connection is established with the controlled device. The control device can also acquire control operations performed by the user through eye movements, gestures, voice, facial expressions, etc., and determine control commands based on these operations. Furthermore, the control device can send the control command to the controlled device through the aforementioned communication connection. Upon receiving the control command, the controlled device can respond to it. In this way, the user can directly control the controlled device simply by looking at it, simplifying user operation, improving the efficiency and accuracy of device control, enhancing the user experience, and achieving the effect of "the device is controllable as soon as it is visible."

[0047] Below, we will first introduce a communication system provided by an embodiment of this application.

[0048] Figure 1 An exemplary embodiment of the communication system provided in this application is shown.

[0049] like Figure 1 As shown, the communication system may include multiple electronic devices. These multiple electronic devices may include: one electronic device 100 (i.e., a control device) and multiple electronic devices 200 (i.e., controlled devices).

[0050] The communication system includes multiple smart terminal devices, which can be of various types. This application embodiment does not limit the specific types of these multiple electronic devices.

[0051] In some examples, the electronic device 100 (also referred to as a spatial computing device) serves as a control device and can be a wearable device, a head-mounted display device, such as augmented reality (AR) / virtual reality (VR) / mixed reality (MR) smart glasses.

[0052] In this embodiment of the application, the device control method provided in this embodiment is described using smart glasses as an example of electronic device 100.

[0053] In some examples, electronic device 200, as the controlled device, can be a smart device with a screen or a smart device without a screen. For example, electronic device 200 can include mobile phones, tablets, desktop computers, laptops, handheld computers, laptops, smart screens, projectors, smart cockpits, in-vehicle systems, smart headphones, game consoles, whole-house smart central control screens, whole-house smart touch panels, QR code-based smart devices (such as QR code payment devices, community gates, airport / high-speed rail station gates, etc.), and can also include Internet of Things (IoT) devices, smart home devices such as smart speakers, smart water heaters, smart lights, smart air conditioners, robot vacuum cleaners, or smart sports equipment, smart medical devices such as blood pressure monitors, etc.

[0054] The communication system allows multiple electronic devices to establish communication connections, which may include, but are not limited to: wired connections, wireless connections such as Bluetooth (BT) connections, wireless local area networks (WLAN) such as wireless fidelity point-to-point (Wi-Fi P2P) connections, near field communication (NFC) connections, infrared (IR) connections, and remote connections (such as connections established through a server), etc.

[0055] Furthermore, multiple electronic devices in this communication system can also be connected and communicate in combination with any of the above methods, and this application embodiment does not impose any restrictions on this. That is to say, the electronic devices in this communication system can form a network (i.e., network) according to certain communication protocols and networking strategies, thereby enabling them to communicate with each other.

[0056] In this embodiment of the application, electronic device 100 can establish a communication connection with electronic device 200 and send control commands to electronic device 200 based on the communication connection to realize the control of electronic device 200.

[0057] Application Scenario 1: Controlling screenless smart devices

[0058] Figure 2 The specific flow of a device control method provided in an embodiment of this application is illustrated by way of example.

[0059] like Figure 2As shown, this method can be applied to a communication system that includes electronic device 100 (i.e., a control device, such as smart glasses) and electronic device 200 (i.e., a controlled device, such as a screenless smart device). The specific steps of this method are described in detail below:

[0060] S201, Electronic device 100 acquires the user's eye movements (e.g., eye gaze direction).

[0061] Taking smart glasses as an example, after a user wears electronic device 100, the electronic device 100 can acquire the user's eye movements. The user's eye movements may include, but are not limited to: eye gaze direction (i.e., line of sight), eye gaze trajectory, and blinking.

[0062] For example, electronic device 100 can acquire a user's eye movements using eye-tracking technology. Eye-tracking technology (also known as gaze tracking technology) utilizes sensors to capture and extract eye feature information, measure eye movement, and estimate the direction of gaze or the position of the eye's gaze point. When a person's eyes look in different directions, subtle changes occur in the eye, generating extractable features. Electronic device 100 can capture or scan these features to track eye changes in real time.

[0063] In some examples, when the electronic device 100 detects that the user's gaze is in a certain direction, it can immediately determine that the object in that direction is the object that the user is currently paying attention to (i.e., the device that the user currently wants to control as described in the embodiments of this application).

[0064] In other examples, since users' eyes may move frequently, to more accurately determine the object the user is focusing on (i.e., the object the user's eyes are focused on) and avoid misidentification, the user can be identified as currently focusing on an object in a certain direction if the user's gaze lingers in that direction for more than a preset time threshold (e.g., 1 second). For example, when the user's eyes stop moving in the direction of gaze for more than the preset time threshold, the device in that direction of gaze can be identified as the device the user is currently focusing on.

[0065] For example, eye-tracking technology can acquire a user's eye movement coordinate data. This eye movement coordinate data can be the coordinate data of multiple points in a coordinate system. Connecting these coordinate data points with a line yields the user's eye movement trajectory (i.e., eye gaze trajectory), which reflects the user's eye movements. By analyzing the data of the eye movement trajectory, the object being focused on by the user's gaze can be determined. For example, if the coordinate of a point on the eye movement trajectory is coordinate 1, then the object at coordinate 1 can be considered the object being focused on by the user's gaze.

[0066] In some examples, after the user wears the electronic device 100, but before the electronic device 100 acquires the user's eye movements, the electronic device 100 may also activate device control functions.

[0067] S202, Electronic device 100 determines the device that the user wants to control as electronic device 200 based on the user's eye movements.

[0068] In one possible implementation, the user's eye movement can be the direction of eye gaze. The electronic device 100 can acquire the user's eye gaze direction in real time. When the time for which the user's eye stops rotating in the gaze direction exceeds a preset time threshold, the device in that gaze direction can be considered as the device that the user currently wants to control (e.g., electronic device 200).

[0069] In practical applications, because a user's gaze may drift, the electronic device 100 may be unable to accurately determine the device the user wants to control based on the user's eye movements. To solve this problem, the electronic device 100 can combine eye-tracking technology and spatial positioning technology to comprehensively determine the device the user wants to control.

[0070] The aforementioned spatial positioning technologies may include, but are not limited to: lidar scanning technology, ultra-wideband (UWB) technology, multi-antenna array technology, and ultrasonic direction finding technology.

[0071] For example, electronic device 100 can combine the user's eye gaze direction and the device's position information in that gaze direction to comprehensively determine the device the user wants to control.

[0072] In some examples, the aforementioned location information may include the relative angle between the control device (i.e., electronic device 100) and the controlled device (e.g., electronic device 200). In one possible implementation, the control device may include an antenna array, which may include at least two antennas. The antenna array may be positioned at the end facing the controlled device when the control device controls the controlled device. The control device may determine the relative angle between the control device and the controlled device based on the signal incident angle (i.e., the incident angle of the communication signal between the control device and the controlled device) and a preset positioning algorithm. The signal incident angle may be determined by the antenna array, and the preset positioning algorithm may include, but is not limited to, an angle of arrival (AOA) positioning algorithm.

[0073] In other examples, the aforementioned location information may include the distance between the control device (i.e., electronic device 100) and the controlled device (e.g., electronic device 200). In one possible implementation, the control device may determine the distance between the control device and the controlled device based on signal strength, signal transmission time, or signal arrival time, and a preset positioning algorithm. The preset positioning algorithm may include, but is not limited to, time of flight (TOF) algorithms, received signal strength (RSS) algorithms, and time of arrival (TOA) algorithms.

[0074] Understandably, since some positioning technologies, such as UWB positioning technology, are bidirectional mutual positioning technologies, both the control device and the controlled device need to have UWB positioning technology implementation modules (for example, the control device includes at least two antennas, and the controlled device includes at least one antenna) in order to achieve the purpose of accurate positioning.

[0075] In one possible implementation, the controlled device whose relative angle to the control device is within a preset angle range can be considered as the device that the user currently wants to control.

[0076] In another possible implementation, a controlled device that is less than a preset distance threshold from the control device can be considered as the device that the user currently wants to control.

[0077] In another possible implementation, a controlled device whose relative angle to the control device is within a preset angle range and whose distance from the control device is less than a preset distance threshold can be considered as the device that the user currently wants to control.

[0078] In some examples, the electronic device 100 can also acquire image information of the device in the direction of the user's eye gaze in real time through a camera, and determine the device that the user currently wants to control based on the image information.

[0079] In one possible implementation, the electronic device 100 can display image information of multiple devices acquired in real time through a camera, and can support the user to select the device they want to control based on the image information. In this way, the electronic device 100 can determine the device the user wants to control based on the user's selection.

[0080] In another possible implementation, when the electronic device 100 acquires image information from multiple devices in real time through a camera, the electronic device 100 can combine the user's eye gaze direction and the position information of the devices in that gaze direction to comprehensively determine which of the multiple devices the user currently wants to control.

[0081] In another possible implementation, when the electronic device 100 acquires image information of a device in real time through a camera, the electronic device 100 can determine that the device is the one that the user currently wants to control.

[0082] S203, Electronic device 100 and electronic device 200 establish a communication connection.

[0083] Specifically, after determining that the device the user wants to control is electronic device 200, electronic device 100 can establish a communication connection with electronic device 200 in order to control electronic device 200.

[0084] In some examples, after electronic device 100 establishes a communication connection with electronic device 200, electronic device 200 can send device information of electronic device 200 to electronic device 100 based on the communication connection. The device information may include, but is not limited to, device type, device identifier, and device function.

[0085] In one possible implementation, electronic device 100 can acquire image information of electronic device 200 via a camera, and can identify the device type of electronic device 200 based on this image information using an image recognition algorithm. Furthermore, electronic device 100 can compare the identified device type of electronic device 200 with the device type included in the device information received from electronic device 200. If they match, electronic device 100 can further determine that the device the user wants to control is electronic device 200. This ensures the accuracy of the controlled device identification.

[0086] In another possible implementation, electronic device 100 can pre-store a mapping table between image information of the controlled device and device identifiers. After acquiring image information of electronic device 200 through a camera, electronic device 100 can look up the device identifier corresponding to the image information of electronic device 200 based on the aforementioned mapping table. Furthermore, electronic device 100 can compare the found device identifier corresponding to the image information of electronic device 200 with the device identifier included in the received device information from electronic device 200. If they match, electronic device 100 can further determine that the device the user wants to control is electronic device 200. This ensures the accuracy of controlled device identification.

[0087] It is easy to understand that if electronic device 100 and electronic device 200 have already established a communication connection before step S202 or before step S201, then step S203 can be skipped.

[0088] In this embodiment of the application, the electronic device 100 can combine the user's eye gaze direction and the device's position information and / or device information (e.g., device type) in that gaze direction to comprehensively determine the device that the user wants to control (i.e., the controlled device).

[0089] In one possible implementation, the electronic device 100 can first acquire device information of all devices within the image range captured by the camera, and then combine the user's eye gaze direction with the location information and / or device information (e.g., device type) of the devices in that gaze direction to comprehensively determine the controlled device.

[0090] In one possible implementation, the electronic device 100 can first determine the target device by combining the user's eye gaze direction and the device's position information in that gaze direction, and then obtain the device information (e.g., device type) of the target device, thereby further determining the controlled device.

[0091] The device information of the device in the aforementioned gaze direction may be received by the electronic device 100 through the communication connection established between the device and the electronic device 100, or it may be sent by the device in the form of broadcast.

[0092] In practical applications, there may be situations where the electronic device 100 identifies multiple controlled devices. In such cases, the electronic device 100 can select one device from these multiple devices as the controlled device based on a preset strategy. For example, the electronic device 100 can randomly select one device from these multiple devices; or, for another example, the electronic device 100 can select the device closest to the electronic device 100 from these multiple devices as the controlled device; or, for yet another example, the electronic device 100 can prompt the user to slightly change their eye gaze direction to aim at the device the user wants to control, so that the electronic device 100 can select the device closer to that gaze direction from among the multiple devices as the controlled device based on the slightly changed eye gaze direction.

[0093] S204. Electronic device 100 outputs a prompt message, which is used to prompt the user for control operations that can be performed on electronic device 200.

[0094] The aforementioned prompts may be output through a user interface, voice, or other means, and this application embodiment does not limit the output method.

[0095] For example, see Figure 3ATaking an electronic device 200 as an example of an electric light, the electronic device 100 can display a prompt interface, which can include the operable function options of the electronic device 200, such as ordinary switch, smart switch (e.g., a switch that can be used to turn on the light when people come and turn off the light when people leave), brightness adjustment, color temperature adjustment, color adjustment, time delay adjustment, mode (e.g., welcome mode, birthday mode, etc.) adjustment options.

[0096] For example, see Figure 3B Taking an electronic device 200 as an example of a light bulb, the electronic device 100 can also output voice prompts (for example, the light bulb can support the following functions for you to operate: normal switch, smart switch, brightness adjustment, color temperature adjustment, color adjustment, time delay adjustment, and mode adjustment. Which function do you need to adjust?).

[0097] In one possible implementation, the operable functions of the electronic device 200 included in the above-mentioned prompt information can be obtained by the electronic device 100 from the device information (including device functions) sent by the electronic device 200.

[0098] In another possible implementation, the operable functions of the electronic device 200 included in the above prompt information can be obtained from a list of operable functions that the electronic device 100 typically has, stored in advance, for the same type of device as the electronic device 200.

[0099] S205, Electronic device 100 acquires user control operations.

[0100] S206, Electronic device 100 determines control commands based on user control operations.

[0101] The user's control operations can be performed through eye movements, gestures, voice, facial expressions, etc., and this application embodiment does not limit this.

[0102] Taking eye movement as an example, in one possible implementation, different eye movements can represent different control operations. Eye movements can include blinking, looking up, looking down, looking left, looking right, and so on. Different control actions can correspond to different control commands. For example, if the controlled device is a light bulb, blinking can represent turning the light on / off. When a user blinks, it means they want to turn the light on / off, and the control command is the "turn on / off" command. Similarly, if the controlled device is an air conditioner, looking up can represent raising the temperature. When a user looks up, it means they want to increase the temperature, and the control command is the "raise temperature" command. Looking down can represent lowering the temperature. When a user looks down, it means they want to lower the temperature, and the control command is the "lower temperature" command. Furthermore, if the controlled device is a curtain, looking left can represent pulling it to the left. When a user looks left, it means they want to pull the curtain to the left, and the control command is the "pull left" command. Looking right can represent pulling it to the right. When a user looks right, it means they want to pull the curtain to the right, and the control command is the "pull right" command. And so on.

[0103] Taking eye tracking as an example, in another possible implementation, the electronic device 100 displays, for example... Figure 3A In the case of the indicated prompt interface, the user can operate the function options displayed on the interface through eye movement, thereby controlling the controlled device. Specifically, the electronic device 100 can obtain the user's current gaze position through eye-tracking technology, calculate the coordinates of that position in the screen coordinate system of the electronic device 100, and use the control corresponding to that coordinate (i.e., the aforementioned function option) as the target control that the user wants to manipulate. The user can operate the control through preset eye movements. For example, a blinking action can represent clicking the control. Suppose the user looks at... Figure 3A If the user selects a normal on / off switch option and triggers a blinking action, then the electronic device 100 can determine that the user wants to turn the electronic device 200 on / off, and can further determine that the control command is an on / off command.

[0104] Taking gestures as an example, different gestures can represent different control operations. Gestures can include swiping left, right, up, down, and tapping, and different control actions can correspond to different control commands. For example, if the controlled device is a light bulb, the user can tap, for example... Figure 3AThe interface shown illustrates how a single function option, such as a regular switch, indicates that the user wants to turn the light on / off, and the control command is the same. Similarly, for an air conditioner, swiping up indicates raising the temperature, and swiping down indicates lowering the temperature. For a curtain, swiping left indicates pulling it to the left, and swiping right indicates pulling it to the right.

[0105] In some examples, electronic device 100 can also support user-defined specific gestures to achieve corresponding control functions. For example, a specific gesture can be defined to control the controlled device to perform screen recording, open an application, or perform media playback control operations, etc.

[0106] Taking voice as an example, different voice commands can represent different control operations, and different control actions can correspond to different control commands. For example, if the controlled device is a light, the user can output the voice command "turn on the light" to indicate that the user wants to turn on the light, and the control command is the "turn on the light" command. Similarly, if the controlled device is an air conditioner, the user can output the voice command "adjust the temperature" to indicate that the user wants to adjust the air conditioner temperature, and the control command is the "adjust the temperature" command. Furthermore, if the controlled device is a curtain, the user can output the voice command "pull the curtain to the left" to indicate that the user wants to pull the curtain to the left, and the control command is the "pull to the left" command.

[0107] Taking facial expressions as an example, in one possible implementation, different facial expressions can represent different control operations. Facial expressions can include smiling, sticking out the tongue, raising eyebrows, wrinkling the nose, pouting, etc., and different control actions can correspond to different control commands. For example, if the controlled device is a light bulb, smiling can represent turning on the light, so a user smiling means the user wants to turn on the light, and the control command is the "turn on the light" command; pouting can represent turning off the light, so a user smiling means the user wants to turn off the light, and the control command is the "turn off the light" command; and so on.

[0108] It should be noted that the aforementioned gestures and expressions can be acquired and recognized by the electronic device 100 through its own sensors / camera, or they can be acquired and recognized by other devices (such as watches, cameras, etc.) that have established a communication connection with the electronic device 100. This application embodiment does not limit this.

[0109] It should be noted that the user's control operations described above can be performed through a combination of one or more of the following methods: eye movement, gestures, voice, and facial expressions. For example, eye movement combined with gestures, eye movement combined with facial expressions, or eye movement combined with voice. This allows users to control electronic device 200 more efficiently through electronic device 100, resulting in a better user experience. For instance, staring at the controlled device for 3 seconds indicates a control operation. However, this would require 3 seconds to achieve control, leading to a discontinuous experience and reduced efficiency. In this case, combining other methods, such as gestures, allows for immediate control of the device upon detecting the user's gesture indicating the operation, eliminating the need for a 3-second wait and improving device control efficiency and user experience.

[0110] S207, Electronic device 100 sends control commands to electronic device 200.

[0111] Specifically, electronic device 100 can send control commands to electronic device 200 based on the communication connection established above.

[0112] S208, Electronic device 200 responds to control commands.

[0113] Specifically, electronic device 200 can receive control commands sent by electronic device 100 based on the communication connection established above, and respond to the control commands to realize the function of user intention control.

[0114] Taking the electronic device 200 as an example, for instance, if the control command is to turn on the light, the electronic device 200 will respond to the control command by turning on the light; if the control command is to turn off the light, the electronic device 200 will respond to the control command by turning off the light; if the control command is to reduce the brightness, the electronic device 200 will respond to the control command by reducing its brightness value from brightness value 1 to brightness value 2; and so on.

[0115] By implementing the above Figure 2 The method provided in the illustrated embodiment allows users to directly and accurately control the screenless smart device simply by looking at it, simplifying user operations, improving device control efficiency, enhancing user experience, and achieving the effect of controlling the device as soon as it is visible.

[0116] Application Scenario 2: Controlling Smart Devices with Screens

[0117] Figure 4 The specific flow of another device control method provided in the embodiments of this application is illustrated by way of example.

[0118] like Figure 4As shown, this method can be applied to a communication system that includes electronic device 100 (i.e., a control device, such as smart glasses) and electronic device 200 (i.e., a controlled device, such as a smart device with a screen). The specific steps of this method are described in detail below:

[0119] S401, Electronic device 100 acquires the user's eye movements (e.g., eye gaze direction).

[0120] S402, Electronic device 100 determines the device that the user wants to control as electronic device 200 based on the user's eye movements.

[0121] S403, Electronic device 100 and electronic device 200 establish a communication connection.

[0122] Among them, the above steps S401-S403 are the same as those mentioned above. Figure 2 Steps S201-S203 are similar to those shown, and their specific execution process can be referred to the above. Figure 2 The relevant textual descriptions in steps S201-S203 shown will not be repeated here.

[0123] S404, Electronic device 100 obtains the user's control operation, which is the user's operation to control the content displayed on the screen of electronic device 200.

[0124] The user's control operations can be performed through eye movements, gestures, voice, facial expressions, etc., and this application embodiment does not limit this.

[0125] S405, Electronic device 100 determines control commands based on user control operations.

[0126] S406, Electronic device 100 sends control commands to electronic device 200.

[0127] S407, Electronic device 200 responds to control commands.

[0128] In this embodiment, the user can use eye movement to make the displayed content (e.g., controls, text, images, etc.) on the screen of the electronic device 200 appear to be in focus (optionally, the focus result may not be displayed, but the electronic device 200 has already learned which displayed content is in focus). That is to say, the user's eye gaze can replace the cursor, and the electronic device 100 can control the cursor displayed on the screen of the electronic device 200 to move according to the trajectory corresponding to the user's eye gaze trajectory by acquiring the user's eye gaze trajectory. For example, when the user's eye gaze trajectory is a trajectory moving from the center to the right, then the cursor displayed on the screen of the electronic device 200 will also move to the right, and when the user's eyes stop moving, the cursor will also stop moving.

[0129] Taking eye tracking as an example, the content displayed on the screen of an electronic device (200) can be controlled through methods such as spatial coordinate mapping. The following describes possible implementation methods for the aforementioned spatial coordinate mapping:

[0130] Possible implementation method 1

[0131] The electronic device 100 may have a spatial coordinate mapping table pre-stored, which may store the mapping relationship between the coordinates of the detection area and the coordinates of the display screen.

[0132] The aforementioned detection area can be a component (such as a display screen or lens) on the electronic device 100 positioned in front of the user's eyes, and the detection area can have a pre-set coordinate system.

[0133] The aforementioned display screen can be the screen of the electronic device 200. The mapping relationship between the coordinates of the detection area and the coordinates of the display screen can be preset by the developer based on the device information of the controlled device (such as device type), which can be used to determine the size of the aforementioned display screen.

[0134] Possible implementation method 2

[0135] Electronic device 100 can acquire the display image on the screen of electronic device 200 through a camera, and map multiple coordinates of the image to multiple coordinates of the detection area, thereby determining the mapping relationship between the coordinates of the detection area and the coordinates of the display image.

[0136] In practical applications, since the relative position between the user wearing the electronic device 100 and the controlled device may change, the electronic device 100 can acquire the display image on the screen of the electronic device 200 at regular intervals and remap multiple coordinates of the image to multiple coordinates of the detection area, thereby determining a new mapping relationship between the coordinates of the detection area and the coordinates of the display image to ensure the accuracy of device control.

[0137] In this embodiment, the electronic device 100 can determine the coordinates of the user's gaze point on the detection area based on the acquired user's eye gaze direction and the preset position of the detection area. Further, the electronic device 100 can obtain the coordinates of the user's gaze point on the display screen based on the mapping relationship between the coordinates of the detection area and the coordinates of the display screen. Further, the electronic device 100 can send this coordinate information to the electronic device 200 so that the electronic device 200 can know the display content on the screen corresponding to this coordinate information.

[0138] In other words, the embodiments of this application can first determine the specific location of the content displayed on the screen of the electronic device 200 that the user wants to control by observing the direction of eye gaze. Furthermore, eye movements (such as blinking), gestures, voice, facial expressions, etc., can be combined to control the displayed content (such as clicking, long-pressing, dragging, etc.).

[0139] It is easy to understand that the control instructions determined by the electronic device 100 may include: coordinate information of the content displayed on the screen of the electronic device 200 that the user wants to control, and information used to indicate the control operation.

[0140] It is easy to understand that during the process of the user's eyeballs rotating (i.e. the formation of the eyeball gaze trajectory), the electronic device 100 can obtain the coordinates of multiple gaze points of the user. The electronic device 100 can send the coordinate information of these multiple gaze points to the electronic device 200 in real time, so that the cursor displayed on the screen of the electronic device 200 moves with the movement of the gaze points.

[0141] For example, see Figure 5A Based on the user's eye gaze direction, electronic device 100 can determine that the content displayed on the screen of electronic device 200 that the user wants to control is the "Video 1" control. Electronic device 200 can then display a focusing effect on this control. As the user's eye gaze direction moves from the direction of the "Video 1" control to the direction of the "Video 2" control, the cursor displayed on the screen of electronic device 200 can move according to the trajectory of the user's eye gaze direction. For example, from... Figure 5A The positions shown are moved sequentially to... Figure 5B The location shown Figure 5C As shown in the diagram, during this process, when the user's gaze is not directed towards the "Video 1" control, the electronic device 200 can disable the focus effect for the "Video 1" control; when the user's gaze moves towards the "Video 2" control, the electronic device 200 can enable the focus effect for the "Video 2" control.

[0142] Continue reading Figure 5A Users can control the content displayed on the screen of electronic device 200 through eye movements, gestures, voice, and facial expressions. For example, a blink can represent a video playback operation. Suppose a user wants to play video 1, then when electronic device 100 detects the user's blink, it can determine that the control command is a video playback command. Furthermore, electronic device 100 can send the video playback command to electronic device 200, and electronic device 200 can respond to the video playback command and display... Figure 5D The user interface shown in the example plays the aforementioned video 1.

[0143] Continue reading Figure 5DAs is easily understood, the electronic device 100 can also support users to continue controlling the content displayed on the interface, such as dragging the playback progress bar, pausing playback, etc. For example, the progress bar can be slid to the right using a swipe gesture.

[0144] By implementing the above Figure 4 The method provided in the illustrated embodiment allows users to directly and accurately control the displayed content simply by looking at it on the screen of a smart device. This achieves the same effect as directly operating the keyboard, mouse, or touchscreen of the smart device, freeing up the user's hands, simplifying operation, improving device control efficiency, and providing convenience and a better user experience.

[0145] Application Scenario 3: Controlling QR code scanning smart devices

[0146] Figure 6 The specific flow of another device control method provided in the embodiments of this application is illustrated by way of example.

[0147] like Figure 6 As shown, this method can be applied to a communication system that includes electronic device 100 (i.e., a control device, such as smart glasses) and electronic device 200 (i.e., a controlled device, such as a barcode scanning smart device). The specific steps of this method are described in detail below:

[0148] S601, Electronic device 100 acquires the user's eye movements (e.g., eye gaze direction).

[0149] S602, Electronic device 100 determines the device that the user wants to control as electronic device 200 based on the user's eye movements.

[0150] S603, Electronic device 100 and electronic device 200 establish a communication connection.

[0151] Among them, the above steps S601-S603 are the same as those mentioned above. Figure 2 Steps S201-S203 are similar to those shown, and their specific execution process can be referred to the above. Figure 2 The relevant textual descriptions in steps S201-S203 shown will not be repeated here.

[0152] S604. Electronic device 100 outputs a prompt message, which is used to prompt the user to control electronic device 100 to perform a scanning operation or be scanned.

[0153] The aforementioned prompts may be output through a user interface, voice, or other means, and this application embodiment does not limit the output method.

[0154] In this embodiment of the application, the electronic device 100 can identify the scanning scenario and the scanning scenario.

[0155] After completing step S604 above, in a scenario where the code is being scanned, steps S605-S609 can be executed; in a scenario where the code is being scanned, steps S610-S611 can be executed. These are described below:

[0156] Scenarios where QR codes are scanned (including steps S605-S609):

[0157] S605, Electronic device 100 acquires user control operations 1.

[0158] The user's control operation 1 mentioned above can be performed through eye movement, gestures, voice, facial expressions, etc., and this application embodiment does not limit this.

[0159] In some examples, the user's control operation 1 described above may be a control operation to instruct the user to consent to the electronic device 100 performing the barcode scanning operation. In this case, the electronic device 100 may continue to execute subsequent steps S606-S608 to display the barcode for scanning.

[0160] In other examples, the user's control operation 1 described above can also be a control operation that triggers the electronic device 100 to perform a barcode scanning operation (e.g., bringing the electronic device 100 close to the electronic device 200 so that the electronic device 200 can scan the barcode displayed on the electronic device 100). In this case, the subsequent step S609 can continue to be executed.

[0161] S606, Electronic device 100 sends message 1 to electronic device 200, which is used to obtain information about barcodes (e.g., QR codes) that electronic device 200 can scan.

[0162] Specifically, after confirming that the user agrees to the electronic device 100 performing the scanning operation, the electronic device 100 can send message 1 to the electronic device 200 based on the communication connection established above.

[0163] The aforementioned barcodes can include one-dimensional barcodes, two-dimensional barcodes (or QR codes), etc. A barcode uses several geometric shapes corresponding to binary numbers to represent textual or numerical information. The scanning process can be explained as an electronic device capturing an image including the barcode using a camera, parsing the barcode in the image, and thus obtaining the information represented by the barcode.

[0164] In this embodiment of the application, a QR code is used as an example for illustration.

[0165] S607, electronic device 200 sends message 2 to electronic device 100, which includes information about a barcode (e.g., a QR code) that electronic device 100 requests to obtain.

[0166] Specifically, after electronic device 200 receives message 1 sent by electronic device 100, electronic device 200 can send message 2 to electronic device 100 based on the communication connection established above, so that electronic device 100 can display the barcode based on message 2.

[0167] S608, Electronic device 100 displays a barcode (e.g., a QR code) based on message 2.

[0168] Specifically, after electronic device 100 receives message 2 sent by electronic device 200, electronic device 100 can display, for example, based on message 2. Figure 7A The barcode shown is an example.

[0169] In some examples, electronic device 100 may pre-store information about barcodes (e.g., QR codes) that electronic device 200 can scan. In this case, steps S606-S607 above may not be performed.

[0170] S609, electronic device 200 performs a scanning operation and completes the corresponding business (such as payment, identity authentication, etc.).

[0171] Specifically, after displaying the barcode, the electronic device 100 can perform a scanning operation and complete the corresponding business.

[0172] In some examples, after confirming the user's consent for the electronic device 100 to perform the barcode scanning operation, the electronic device 100 may not display the barcode. Instead, the electronic device 100 and the electronic device 200 interact in the background based on the established communication connection to complete the corresponding business (such as payment, identity authentication, etc.). This can further simplify user operations, improve business execution efficiency, and provide users with a better experience (such as seamless payment experience, seamless authentication experience, etc.).

[0173] Before performing step S609, the process may further include: electronic device 100 determining a control command based on the control operation 1 and sending the control command to electronic device 200 to control electronic device 200 to perform a barcode scanning operation.

[0174] QR code scanning scenario (including steps S610-S611):

[0175] S610, Electronic device 100 acquires user control operations 2.

[0176] The user's control operation 2 mentioned above can be performed through eye movement, gestures, voice, facial expressions, etc., and this application embodiment does not limit this.

[0177] Among them, the user's control operation 2 can be a control operation used to instruct the user to agree to the electronic device 100 performing a scanning operation.

[0178] S611, Electronic device 100 performs a scanning operation on electronic device 200 based on user control operation 2, and completes the corresponding business (such as payment, identity authentication, etc.) of the scanning operation.

[0179] Specifically, after confirming that the user agrees to the electronic device 100 performing the scanning operation, the electronic device 100 can, for example... Figure 7B The example demonstrates performing a barcode scanning operation on electronic device 200 and completing the corresponding business process. Among these steps... Figure 7B The barcode shown in the example can be a barcode on electronic device 200 used to perform a scanning operation (e.g., a barcode displayed on the screen of electronic device 100).

[0180] In some examples, after confirming the user's consent for the electronic device 100 to perform the scanning operation, the electronic device 100 may also not explicitly perform the scanning operation (e.g., not display the code). Figure 7B Instead of the exemplary QR code scanning interface shown, electronic device 100 interacts with electronic device 200 in the background based on the established communication connection to complete the corresponding business (such as payment, identity authentication, etc.). This can further simplify user operations, improve business execution efficiency, and provide users with a better experience (such as seamless payment experience, seamless authentication experience, etc.).

[0181] In other examples, in the above-mentioned barcode scanning scenario, the controlled device may not be involved, but only a barcode. The user may be looking at only the barcode, not the controlled device. In this case, the electronic device 100 can output a prompt message when it detects that the user is looking at the barcode (for example, by obtaining the user's eye movements and determining the user is looking at the barcode through the image obtained by the camera). The prompt message is used to prompt the user to control the electronic device 100 to perform a barcode scanning operation. Further, steps S610-S611 can be executed.

[0182] By implementing the above Figure 6 The method provided in the illustrated embodiment allows users to perform scanning or being scanned operations simply by looking at the controlled device, thereby enabling convenient and quick completion of corresponding business transactions (such as payment, identity authentication, etc.), simplifying user operations, improving business execution efficiency, and enhancing user experience.

[0183] Application Scenario 4: Controlling smart devices logged in with the same account and locked.

[0184] Figure 8The specific flow of another device control method provided in the embodiments of this application is illustrated by way of example.

[0185] like Figure 8 As shown, this method can be applied to a communication system that includes electronic device 100 (i.e., a control device, such as smart glasses) and electronic device 200 (i.e., a controlled device, such as a smart device logged into the same account and locked). The specific steps of this method are described in detail below:

[0186] S801, Electronic device 100 acquires the user's eye movements (e.g., eye gaze direction).

[0187] S802, Electronic device 100 determines the device the user wants to control as electronic device 200 based on the user's eye movements.

[0188] S803, Electronic device 100 and electronic device 200 establish a communication connection.

[0189] Among them, the above steps S801-S803 are the same as those mentioned above. Figure 2 Steps S201-S203 are similar to those shown, and their specific execution process can be referred to the above. Figure 2 The relevant textual descriptions in steps S201-S203 shown will not be repeated here.

[0190] S804, Electronic device 100 outputs a prompt message to remind the user that the electronic device 200 can be unlocked.

[0191] The aforementioned prompts may be output through a user interface, voice, or other means, and this application embodiment does not limit the output method.

[0192] Before step S804 above, the method may further include: electronic device 100 obtaining information that electronic device 200 is in a locked state (e.g., a screen lock state). There can be various ways to obtain this information, and this embodiment does not limit this method. For example, electronic device 100 can obtain screen state information of electronic device 200 through a camera (e.g., ...). Figure 9A The interface element "Device locked" (or the screen of electronic device 200 is off) is displayed on the exemplary interface to indicate that electronic device 200 is locked; or electronic device 100 can actively request electronic device 200 to obtain its current status information.

[0193] In some examples, before step S804 above, the method may further include: electronic device 100 obtaining the account information logged in by electronic device 200, and determining whether the account logged in by electronic device 200 is consistent with the account logged in by electronic device 100 based on the account information; if so, then step S804 above is executed.

[0194] S805, Electronic device 100 obtains the user's unlock operation.

[0195] S806, Electronic device 100 generates an unlock request based on the user's unlocking operation.

[0196] The user's unlocking operation can be performed through eye movement, gestures, voice, facial expressions, etc., and this application embodiment does not limit this.

[0197] S807, Electronic device 100 sends an unlock request to electronic device 200.

[0198] Specifically, electronic device 100 can send an unlock request to electronic device 200 based on the communication connection established above.

[0199] S808, Electronic Device 200 responds to unlock request.

[0200] Specifically, electronic device 200 can receive unlock requests sent by electronic device 100 based on the established communication connection, and respond to the unlock requests by performing unlocking operations. For example, after responding to the unlock requests, electronic device 200 can... Figure 9A The lock state shown in the example switches to Figure 9B The unlocked state is shown.

[0201] In some examples, to further prevent security risks, the electronic device 200 may prompt the user whether to unlock before unlocking, and then unlock the electronic device 200 after the user confirms the unlock.

[0202] Understandably, after the electronic device 200 is successfully unlocked, the aforementioned steps can continue. Figure 4 The method provided in the illustrated embodiment is used to control the content displayed on the screen of the electronic device 200.

[0203] It should be noted that the above description only uses a laptop computer as an example of electronic device 200, and is not limited thereto. Electronic device 200 can also be other types of devices, and this application embodiment does not limit this.

[0204] It is understandable that, in the above Figure 8 In the illustrated embodiments, the electronic device 100 can unlock other devices logged into the same account using the methods provided in the embodiments of this application. In some other embodiments of this application, for example with user authorization, the electronic device 100 can also unlock other devices logged into different accounts using the methods provided in the embodiments of this application.

[0205] By implementing the above Figure 8The method provided in the illustrated embodiment allows users to unlock a controlled device simply by looking at it while it is locked. This simplifies user operations, improves the efficiency and security of device unlocking, enhances user experience, and solves the problems of poor user experience and security risks associated with identity authentication caused by collaborative unlocking through proximity sensing.

[0206] Application Scenario 5: Determining the controllable operations of a controlled device based on its device type and status.

[0207] Figure 10 The specific flow of another device control method provided in the embodiments of this application is illustrated by way of example.

[0208] like Figure 10 As shown, this method can be applied to a communication system that includes electronic device 100 (i.e., a control device, such as smart glasses) and electronic device 200 (i.e., a controlled device). The specific steps of this method are described in detail below:

[0209] S1001, Electronic device 100 acquires the user's eye movements (e.g., eye gaze direction).

[0210] S1002, Electronic device 100 determines the device that the user wants to control as electronic device 200 based on the user's eye movements.

[0211] S1003, Electronic device 100 and electronic device 200 establish a communication connection.

[0212] Among them, the above steps S1001-S1003 are the same as those mentioned above. Figure 2 Steps S201-S203 are similar to those shown, and their specific execution process can be referred to the above. Figure 2 The relevant textual descriptions in steps S201-S203 shown will not be repeated here.

[0213] S1004. Electronic device 100 determines whether electronic device 200 is a screenless smart device or a smart device with a screen based on the device type of electronic device 200.

[0214] The device type of the aforementioned electronic device 200 can be obtained by the electronic device 100 through the image information of the electronic device 200 acquired by the electronic device 100 via the camera and identified based on the image information by an image recognition algorithm, or it can be obtained by the electronic device 100 from the device information received from the electronic device 200.

[0215] For example, if the device type of electronic device 200 is a light bulb, air conditioner, curtain, speaker, etc., electronic device 100 can determine that electronic device 200 is a screenless smart device.

[0216] For example, when the device type of electronic device 200 is a tablet, laptop, smart screen, etc., electronic device 100 can determine that electronic device 200 is a smart device with a screen.

[0217] S1005, Electronic device 100 outputs prompt information 1, which is used to prompt the user for control operations that can be performed on electronic device 200.

[0218] S1006, Electronic device 100 acquires user control operations 1.

[0219] S1007, Electronic device 100 determines control command 1 based on user control operation 1.

[0220] S1008, Electronic device 100 sends control command 1 to electronic device 200.

[0221] S1009, Electronic device 200 responds to control command 1.

[0222] Among them, the above steps S1005-S1009 are the same as those mentioned above. Figure 2 Steps S204-S208 are similar to those shown, and their specific execution process can be referred to the above. Figure 2 The relevant textual descriptions in steps S204-S208 shown will not be repeated here.

[0223] S1010: Electronic device 100 determines whether electronic device 200 is in a locked state.

[0224] In one possible implementation, electronic device 100 can obtain the device state of electronic device 200 (e.g., screen lock / unlock state) through a camera.

[0225] When the electronic device 200 is in an unlocked state, the electronic device 100 can execute the subsequent step S1011.

[0226] When the electronic device 200 is in a locked state, the electronic device 100 can execute the subsequent step S1015.

[0227] S1011, Electronic device 100 obtains user control operation 2, which is an operation by the user to control the content displayed on the screen of electronic device 200.

[0228] S1012, Electronic device 100 determines control command 2 based on user control operation 2.

[0229] S1013, Electronic device 100 sends control command 2 to electronic device 200.

[0230] S1014, Electronic device 200 responds to control command 2.

[0231] Among them, the above steps S1011-S1014 are the same as those mentioned above. Figure 4 Steps S404-S407 are similar to those shown, and their specific execution process can be referred to the above. Figure 4 The relevant textual descriptions in steps S404-S407 shown will not be repeated here.

[0232] S1015, Electronic device 100 outputs prompt information 2, which prompts the user to perform an unlocking operation on electronic device 200.

[0233] S1016, Electronic device 100 obtains the user's unlock operation.

[0234] S1017, Electronic device 100 generates an unlock request based on the user's unlocking operation.

[0235] S1018, Electronic device 100 sends an unlock request to electronic device 200.

[0236] S1019, Electronic device 200 responds to unlock request.

[0237] Among them, steps S1015-S1019 are the same as those mentioned above. Figure 8 Steps S804-S808 are similar to those shown, and their specific execution process can be referred to the above. Figure 8 The relevant textual descriptions in steps S804-S808 shown will not be repeated here.

[0238] It is understandable that after the electronic device 200 is successfully unlocked, the above steps S1011-S1014 can be continued to control the content displayed on the screen of the electronic device 200.

[0239] By implementing the above Figure 10 The method provided in the illustrated embodiment can determine the controllable mode and controllable function (or controllable option) of the controlled device, regardless of the type of the controlled device (e.g., smart device with screen or smart device without screen) or the state of the device (e.g., locked state or unlocked state). Users can control the controlled device simply by looking at it, simplifying user operation, improving the efficiency and accuracy of device control, and enhancing the user experience.

[0240] The following is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application.

[0241] Figure 11 An exemplary embodiment of an electronic device 100 provided in this application is shown.

[0242] like Figure 11 As shown, the electronic device 100 may include: a processor 1101, a memory 1102, a communication module 1103, a sensor module 1104, a camera 1105, a display module 1106, and an audio module 1107. These components can be coupled together and communicate with each other.

[0243] Understandable, Figure 11 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. For example, the electronic device 100 may also include physical buttons such as power buttons, volume buttons, various interfaces, etc. Figure 11 The components shown can be implemented in hardware, software, or a combination of both.

[0244] Processor 1101 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution, enabling each component to perform corresponding functions, such as human-computer interaction, motion tracking / prediction, rendering and display, and audio processing.

[0245] The memory 1102 stores executable program code for performing the device control method provided in the embodiments of this application. The executable program code includes instructions. The memory 1102 may include a program storage area and a data storage area. The program storage area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as image data, etc.). Furthermore, the memory 1102 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 1101 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 1102 and / or instructions stored in memory disposed in the processor.

[0246] Communication module 1103 may include a wireless communication module. The wireless communication module can provide solutions for wireless communication applications such as WLAN, BT, GNSS, FM, IR, and UWB on electronic device 100. The wireless communication module may be one or more devices integrating at least one communication processing module. Communication module 1103 can support communication between electronic device 100 and electronic device 200.

[0247] In the embodiments of this application, the communication module can also be used to implement spatial positioning technology.

[0248] The sensor module 1104 may include an accelerometer, compass, gyroscope, magnetometer, or other sensors for detecting motion. The sensor module 1104 is used to collect corresponding data, such as the accelerometer collecting the acceleration of the electronic device 100, and the gyroscope collecting the speed of motion of the electronic device 100. The data collected by the sensor module 1104 can reflect the movement of the user's head while wearing the electronic device 100. In some embodiments, the sensor module 1104 may be an inertial measurement unit (IMU) disposed within the electronic device 100.

[0249] The sensor module 1104 may also include an optical sensor for use in conjunction with the camera 1105 to track the user's eye position and capture eye movement data. This eye movement data can be used, for example, to determine the user's interpupillary distance, the 3D position of each eye relative to the electronic device 100, the magnitude of torsion and rotation (i.e., turning, pitching, and swaying) of each eye, and the direction of gaze, etc. In one example, infrared light is emitted within the electronic device 100 and reflected from each eye. The reflected light is detected by the camera 1105 or the optical sensor, and the electronic device 100 can analyze the user's eye position, pupil diameter, movement state, etc., based on the detected data and changes in the infrared light reflected from each eye.

[0250] Camera 1105 can be used to capture still images or videos. These still images or videos can be externally focused images or videos of the user's surroundings, or internally focused images or videos. Camera 1105 can track the movement of one or both eyes of the user. Camera 1105 includes, but is not limited to, traditional color cameras (RGB cameras), depth cameras (RGB depth cameras), and dynamic vision sensor (DVS) cameras. Depth cameras can acquire depth information of the object being photographed. In some embodiments, camera 1105 can be used to capture images of the user's eyes. Electronic device 100 can determine the state of the user's eyes based on the images captured by camera 1105 and execute corresponding functions according to the state of the user's eyes. That is, the user can trigger electronic device 100 to execute corresponding functions by inputting eye movement operations on electronic device 100. The state of the user's eyes may include: whether they are turning, the direction of turning, whether they have not turned for a long time, the angle at which they are looking outwards, etc.

[0251] Electronic device 100 uses a GPU, display module 1106, and application processor to present or display images. The GPU is a microprocessor for image processing, connected to the display module 1106 and the application processor. Processor 1101 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0252] The display module 1106 may include one or more displays. In this embodiment, the display in the electronic device 100 can be used to receive data or content (e.g., rendered images) processed by the GPU of the electronic device 100 and display it. The display may include a display panel, which can be used to display images. The display panel may be a liquid crystal display device such as LCD, OLED, AMOLED, FLED, MiniLED, MicroLED, Micro-OLED, QLED, etc.

[0253] The audio module 1107 is used to acquire and output audio. The audio module 1107 may include, but is not limited to, a microphone, speaker, headphones, etc.

[0254] This application provides a chip system including: a processor coupled to a memory for storing programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods described in any of the above method embodiments.

[0255] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0256] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0257] For example, the chip system may be a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on chip (SoC), a central processor (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0258] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0259] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0260] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0261] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A device control method applied to a first electronic device, comprising: The method comprises: The first electronic device acquires the eyeball action of the user, and the eyeball action comprises an eyeball gaze direction; The first electronic device regards a second electronic device meeting a first condition in the eyeball gaze direction as a controlled device; The first electronic device controls the second electronic device; The first condition comprises that the distance between the second electronic device and the first electronic device is less than a first distance threshold, and / or the device type of the second electronic device is the same as a first device type, which is determined by the first electronic device based on an image of the second electronic device collected by a camera, and the device type of the second electronic device is acquired by the first electronic device from device information sent by the second electronic device.

2. The method of claim 1, wherein, The first electronic device is smart glasses.

3. The method according to claim 1 or 2, characterized in that, The second electronic device is a screenless smart device, and the first electronic device controls the second electronic device, specifically comprising: The first electronic device outputs prompt information, and the prompt information comprises a control operation supported by the second electronic device for the user; The first electronic device acquires the control operation of the user and determines a control instruction based on the control operation; The first electronic device sends the control instruction to the second electronic device to control the second electronic device to perform an operation corresponding to the control instruction.

4. The method of claim 3, wherein, The screenless smart device comprises any one of the following: a lamp, an air conditioner, and a sound box.

5. The method according to claim 1 or 2, characterized in that, The second electronic device is a smart device with a screen, and the first electronic device controls the second electronic device, specifically comprising: The first electronic device determines a first position of a gaze point of the user on the screen of the second electronic device based on the eyeball gaze direction of the user; The first electronic device sends a first control instruction to the second electronic device to control the second electronic device to display the content at the first position as focus content.

6. The method of claim 5, wherein, The smart device with a screen comprises any one of the following: a mobile phone, a tablet, a computer, a smart screen, and a car machine.

7. The method according to claim 5 or 6, characterized in that, The method further comprises: The first electronic device acquires a control operation of the user for the focus content and determines a second control instruction based on the control operation; The first electronic device sends the second control instruction to the second electronic device to control the second electronic device to perform an operation corresponding to the second control instruction for the focus content.

8. The method of claim 1 or 2, wherein, The first electronic device and the second electronic device are both associated with a first account, the second electronic device is in a locked state, and the first electronic device controls the second electronic device, specifically comprising: The first electronic device acquires a user unlocking operation and determines an unlocking instruction based on the unlocking operation; The first electronic device sends the unlocking instruction to the second electronic device to control the second electronic device to perform an unlocking operation.

9. The method of claim 1 or 2, wherein, The second electronic device is a code scanning device, and the first electronic device controls the second electronic device, specifically comprising: The first electronic device acquires a first control operation of the user and determines a control instruction based on the first control operation, and displays a first barcode; The first electronic device sends the control instruction to the second electronic device to control the second electronic device to perform a code scanning operation on the first code.

10. The method of claim 9, wherein, The method further includes: The first electronic device obtains a second control operation of a user; The first electronic device performs a code scanning operation on a second code on the second electronic device based on the second control operation.

11. The method according to any one of claims 3-10, characterized in that, The execution manner of the control operation includes any one or more of the following: eye movement, gesture, voice, expression.

12. The method according to any one of claims 1 to 11, characterized in that, Before the first electronic device controls the second electronic device, the method further includes: The first electronic device establishes a communication connection with the second electronic device, and the device type of the second electronic device is obtained by the first electronic device from device information sent by the second electronic device through the communication connection.

13. An electronic device, comprising: The electronic device includes one or more processors and one or more memories; wherein the one or more memories are coupled with the one or more processors, and the one or more memories are used to store computer program codes, the computer program codes include computer instructions, when the one or more processors execute the computer instructions, make the electronic device execute the method as claimed in any one of claims 1-12.

14. A computer storage medium, characterized in that The computer storage medium stores a computer program, the computer program includes program instructions, when the program instructions run on an electronic device, make the electronic device execute the method as claimed in any one of claims 1-12.

15. A computer program product, when the computer program product runs on a computer, makes the computer execute the method as claimed in any one of claims 1-12.

Images