Device linkage method and device based on directional interaction, device and medium

Abstract

The invention relates to an equipment linkage method and device based on directional interaction, equipment and a medium. The method is applied to the electronic equipment, and can comprise the steps that target interaction equipment watched by a user is determined, and the target interaction equipment and the electronic equipment belong to a target account; and sending a target operation instruction to the target interaction device through the target account. Through the technical scheme of the invention, under the condition that the electronic equipment cannot realize the function required by the user, the function required by the user can be realized through other equipment which is existing around the user and belongs to the target account with the electronic equipment. According to the mode, components do not need to be added to the electronic equipment, and the interaction efficiency and experience of the user can be greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of human-computer interaction technologies, and particularly to a device linkage method, device, equipment, and medium based on directional interaction. Background Art

[0002] When a user performs human-computer interaction, the corresponding interaction function often cannot be realized due to the lack of some functional components in the interacting device. For example, when a user interacts with audio glasses, since the audio glasses lack a screen as a functional component, the user cannot obtain image information (such information needs to be displayed to the user through a screen), which easily reduces the user's interaction experience. Summary of the Invention

[0003] The present disclosure provides a device linkage method, device, equipment, and medium based on directional interaction to solve the deficiencies in the related technologies.

[0004] According to the first aspect of the embodiments of the present disclosure, a device linkage method based on directional interaction is proposed. The method is applied to an electronic device, and the method includes:

[0005] Determine a target interaction device gazed at by the user, where the target interaction device and the electronic device belong to a target account;

[0006] Send a target operation instruction to the target interaction device through the target account.

[0007] Optionally, the determining the target interaction device gazed at by the user includes: obtaining the gaze angle information of the user's eyes and the image collected by the camera; mapping the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes; and determining the object included in the target area as the target interaction device gazed at by the user.

[0008] Optionally, when the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the camera and the user's eyes pre-calibrated; or, when the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the camera and the user's eyes temporarily calibrated.

[0009] Optionally, the calibration method for the relative position relationship includes at least one of the following: when both the camera and the user's eyes capture a two-dimensional code, obtaining first gaze angle information of the user's eyes with respect to the two-dimensional code and a calibration image collected by the camera and containing the two-dimensional code, determining the distance and pose of the camera relative to the two-dimensional code according to the position of the two-dimensional code in the calibration image, and determining the relative position relationship according to the distance and pose and the first gaze angle information; or, when either the camera or the user's eyes have captured a calibration object, obtaining description information of the calibration object and feeding back the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object that matches the description information, obtaining second gaze angle information of the user's eyes with respect to the calibration object and the distance and pose of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and pose and the second gaze angle information.

[0010] Optionally, the method further includes: obtaining identity information of the user, and determining whether the user has management authority over the target account according to the identity information; the sending the target operation instruction to the target interaction device through the target account includes: when it is determined that the user has management authority over the target account, sending the target operation instruction to the target interaction device through the target account.

[0011] Optionally, the sending the target operation instruction to the target interaction device through the target account includes: when the target interaction device is determined, sending a predefined default instruction to the target interaction device through the target account, and / or generating a corresponding target operation instruction in response to a target operation request initiated by the user, and sending the target operation instruction to the target interaction device.

[0012] Optionally, the sending the target operation instruction to the target interaction device through the target account includes: when a communication connection is established between the target interaction device and the electronic device, sending the target operation instruction to the target interaction device; or, when the target interaction device and the electronic device are respectively in communication connection with a control center, sending the target operation instruction to the control center so that the control center forwards the target operation instruction to the target interaction device.

[0013] According to a second aspect of the embodiments of the present disclosure, a device linkage device based on directional interaction is provided. The device is applied to an electronic device, and the device includes:

[0014] A device determination unit, configured to determine a target interaction device at which the user is gazing, where the target interaction device and the electronic device belong to a target account;

[0015] An instruction sending unit, configured to send a target operation instruction to the target interaction device through the target account

[0016] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0017] A processor;

[0018] A memory for storing instructions executable by the processor;

[0019] Wherein, the processor is configured to implement the method described in the embodiments of the first aspect above.

[0020] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the embodiments of the first aspect above are implemented.

[0021] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0022] As can be seen from the above embodiments, the present disclosure manages multiple devices through the same account. When it is determined that the target interaction device at which the user is gazing and the electronic device belong to the same target account, a target operation instruction is sent to the target interaction device through the target account, so that when the electronic device cannot implement the functions required by the user, the functions required by the user can be implemented through other devices existing around the user and belonging to the target account with the electronic device. This method neither requires adding components to the electronic device nor can greatly improve the user's interaction efficiency and experience.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0025] Figure 1 It is a schematic flowchart of a device linkage method based on directional interaction shown according to an embodiment of the present disclosure.

[0026] Figure 2It is a deployment schematic diagram of a camera and an eye movement module shown according to an embodiment of the present disclosure.

[0027] Figure 3 It is a schematic diagram of mapping gaze angle information to a target area in an image shown according to an embodiment of the present disclosure.

[0028] Figure 4 It is a schematic block diagram of a device linkage device based on directional interaction shown according to an embodiment of the present disclosure.

[0029] Figure 5 It is a schematic block diagram of an electronic device shown according to an embodiment of the present disclosure. Detailed implementation manners

[0030] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0031] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present disclosure. The singular forms "a" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0033] For the purpose of simplicity and easy understanding, when representing the size relationship in this article, the terms used are "greater than" or "less than", "higher than" or "lower than". However, for those skilled in the art, it can be understood that the term "greater than" also covers the meaning of "greater than or equal to", and "less than" also covers the meaning of "less than or equal to"; the term "higher than" covers the meaning of "higher than or equal to", and "lower than" also covers the meaning of "lower than or equal to".

[0034] Next, one or more embodiments of the present disclosure will be described in detail.

[0035] In one embodiment, the device linkage method based on directional interaction of the present disclosure can be applied to an electronic device, and the electronic device includes but is not limited to electronic devices with certain computing capabilities such as smartphones, desktop computers, tablet computers, laptop computers, e-book readers, smart watches, smart glasses, etc., and the electronic device can run software or websites in categories such as instant messaging, social networking, gaming, education, etc., and can be applied to scenarios of multi-person audio and video calls. It should be noted that: the application program of the client based on directional interaction device linkage can be pre-installed on the electronic device, so that the client can be started and run on the electronic device; of course, when using an online "client" such as HTML5 technology, there is no need to install the corresponding application program on the electronic device to obtain and run the client.

[0036] Figure 1 is a schematic flowchart of a device linkage method based on directional interaction shown according to an embodiment of the present disclosure. As Figure 1 shown, the method may include the following steps:

[0037] S101: Determine the target interaction device gazed at by the user, and the target interaction device and the electronic device belong to the target account.

[0038] The device where the user's line-of-sight focus is located is the target interaction device gazed at by the user. The target account can be any account, such as the user's account or the account of other users. That the target interaction device and the electronic device belong to the target account can be understood as that the target interaction device and the electronic device can be managed through the target account.

[0039] In one embodiment, the electronic device can jointly use a camera and an eye movement module to determine the target interaction device gazed at by the user. The eye movement module monitors the movement of the eyes by using technologies such as infrared rays, cameras, or electrodes. These technologies can accurately measure the position and movement of the user's eyeballs and convert them into digital signals. Therefore, the eye movement module can capture the gaze angle information of the user's eyes and send the gaze angle information to the electronic device. The camera is used to collect images and send the collected images to the electronic device. It should be noted that the eye movement module, the camera, and the electronic device are logically independent of each other and can be physically deployed on the same electronic device or separately deployed on multiple electronic devices. For example, taking a screenless audio glasses as the electronic device, the eye movement module and the camera can be deployed on the audio glasses, as Figure 2 shown. During the operation of the audio glasses, the audio glasses can implement the device linkage scheme based on directional interaction of the present disclosure to achieve the purpose of device linkage.

[0040] In addition to deploying the eye movement module and the camera on the user's body so that the user can carry the eye movement module and the camera with them, the eye movement module and the camera can also be deployed in the space outside the user's body. For example, when the user is facing a computer, the eye movement module can be deployed on the computer to capture the gaze angle information of the user's eyes in front of the computer. Another example is that when the camera is deployed on the ceiling of a meeting room, the camera can also collect images in the meeting room and send the images to an electronic device.

[0041] After receiving the gaze angle information of the user's eyes and the images collected by the camera, the electronic device can map the gaze angle information to the target area in the received images according to the relative position relationship between the camera and the user's eyes. The relative position relationship between the camera and the user's eyes refers to the spatial position or orientation relationship between the camera and the user's eyes, which can include a rotational relationship and / or a translational relationship, that is, through certain rotational operations and / or translational operations, the user's eyes can be moved to the position where the camera is located, or the camera can be moved to the position where the user's eyes are located. The electronic device can map the gaze angle information to the images collected by the camera according to the relative position relationship between the camera and the user's eyes to obtain a target area, and this target area is the user's gaze area. Then, the object contained in the target area can be determined as the target interaction device that the user is gazing at.

[0042] Figure 3 is a schematic diagram showing how to map the gaze angle information to the target area in the image shown in an exemplary embodiment. As Figure 3 shown, both the camera and the eye movement module are deployed on the audio glasses. When the user wears the audio glasses, the relative position relationship between the camera and the user's eyes is usually a fixed position relationship. For the sake of easy display, Figure 3 only the camera on the audio glasses is shown in, and the audio glasses and the eye movement module are not shown. Figure 3 The dotted line ① in shows the gaze angle (i.e., the gaze direction) of the user's eyes. According to the relative position relationship between the camera and the user's eyes, this gaze angle is mapped to the images collected by the camera, and a target area (the area within the dotted box) can be obtained. This target area corresponds to the user's gaze area in the real space. In Figure 3 , the object contained in this target area is a TV set. Therefore, it can be determined that the target interaction device that the user is gazing at is the TV set.

[0043] In this embodiment, by obtaining the image captured by the camera and the gaze angle information of the user's eyes, and mapping the gaze angle information to the image captured by the camera according to the relative position relationship between the camera and the user's eyes, the gaze area (i.e., the target area) of the user in the image captured by the camera is determined, and then the object included in the gaze area is determined as the target interaction device gazed at by the user. This method of determining the target interaction device does not need to display the image captured by the camera in front of the user, and can directly and accurately determine the real object (i.e., the target interaction device) gazed at by the user according to the gaze angle information of the user, greatly improving the determination efficiency and accuracy of the interaction device, so that the subsequent user can directly perform human-computer interaction with the real object he sees, effectively improving the interaction efficiency and the user's interaction experience.

[0044] In one embodiment, the relative position relationship between the camera and the user's eyes can be divided into a static relative position relationship and a dynamic relative position relationship. The static relative position relationship can be understood as that this relative position relationship is fixed and will not change. The dynamic relative position relationship can be understood as that this relative position relationship is in dynamic change.

[0045] When the camera moves with the user's head, the relative position relationship between the camera and the user's eyes is a static relative position relationship. For example, Figure 3 the camera deployed on the audio glasses will move with the user's head, so the relative position relationship between the camera and the user's eyes is fixed. In this case, before implementing the device linkage method based on directional interaction of the present disclosure, the static relative position relationship between the camera and the user's eyes can be pre-calibrated, so that the calibrated relative position relationship can be directly obtained when determining the target interaction device, which helps to improve the determination efficiency of the target interaction device.

[0046] When the camera does not move with the user's head, the relative position relationship between the camera and the user's eyes is a dynamic relative position relationship. For example, when the camera is deployed on the user's arm, the camera will not move with the user's head, so the relative position relationship between the camera and the user's eyes will change. Another example is when the camera is deployed in the space outside the user's body (such as the ceiling of a meeting room), the camera will also not move with the head of the user in the meeting room. In the above situations, it is necessary to temporarily calibrate the dynamic relative position relationship between the camera and the user's eyes at the current moment when implementing the device linkage method based on directional interaction of the present disclosure, so as to ensure the accuracy of the calibrated dynamic relative position relationship and accurately locate the target interaction device according to the accurate dynamic relative position relationship.

[0047] In one embodiment, the relative position relationship between the camera and the user's eyes can be calibrated by capturing the same object through the camera and the user's eyes. The specific calibration methods can include the following two methods:

[0048] ① The same object is a two-dimensional code of a specific size, and the two-dimensional code can obtain relative distance and attitude information, such as an ARUCO code. A two-dimensional code of a specific size can be displayed on the user's mobile phone. Then, when it is determined that both the camera and the user's eyes have captured the two-dimensional code, the first gaze angle information of the user's eyes with respect to the two-dimensional code and the calibration image containing the two-dimensional code collected by the camera are obtained. Among them, the first gaze angle information can be obtained through the eye movement module. Then, the calibration image is analyzed, and the distance and attitude of the camera relative to the two-dimensional code are determined according to the position of the two-dimensional code in the calibration image and the size of the two-dimensional code. The relative position relationship between the camera and the user's eyes is calculated according to the first gaze angle information and the distance and attitude of the camera relative to the two-dimensional code. The specific calculation method can refer to the content of determining the relative position relationship in the related technology, which will not be elaborated here.

[0049] ② The same object can be any calibration object other than the two-dimensional code. When either the camera or the user's eyes have captured the calibration object, the description information of the calibration object is obtained and sent to the other of the camera and the user's eyes, so that the other party can capture the calibration object that matches the description information. For example, for the image captured by the camera, the object included in the image is recognized, and then a calibration object is selected from the image, and the description information of the calibration object is fed back to the user (voice feedback, screen display feedback or other feedback methods), so that the user rotates the eyes to capture the calibration object according to the feedback description information. Another example is to obtain the description information of a certain calibration object within the user's field of view and send the description information to the camera, so that the camera captures the calibration object that matches the description information. When both the camera and the user's eyes have captured the calibration object, the second gaze angle information of the user's eyes with respect to the calibration object and the distance and attitude of the camera relative to the calibration object are obtained, and the relative position relationship between the camera and the user's eyes is determined according to the distance and attitude and the second gaze angle information.

[0050] In one embodiment, it is possible that the device the user is looking at and the electronic device do not belong to the target account. Therefore, after determining the device the user is looking at (hereinafter referred to as the "gaze device"), the gaze device can be identified to further determine whether the gaze device is the target interaction device. For example, it can be determined whether the gaze device and the electronic device belong to the target account by identifying the unique identifier of the gaze device. It can also be determined whether the gaze device and the electronic device belong to the target account through UWB pointing operation, which requires the deployment of UWB chips and array antennas in both the electronic device and the gaze device. The electronic device can send an inquiry request to the gaze device, and the inquiry request is used to ask whether the gaze device belongs to the target account. After the electronic device receives the information confirming the attribution returned by the gaze device, the gaze device can be determined as the target interaction device.

[0051] In addition, it can also be determined whether the gaze device is a device that can implement the functions required by the user by identifying the type of the gaze device. For example, if the function required by the user is to play music, then when it is determined that the gaze device is a refrigerator, the gaze device cannot play music, so the gaze device cannot be used as the target interaction device. If it is identified that the gaze device does not belong to the target account with the electronic device, or the gaze device cannot implement the functions required by the user, the electronic device can return a prompt message such as "The device is incorrect. Do you want to replace it with another device?" to prompt the user whether to replace it with another interaction device.

[0052] In this embodiment, the gaze device is determined as the target interaction device for subsequent interaction operations only when the gaze device and the electronic device belong to the target account and the type of the gaze device is capable of implementing the functions required by the user, which helps to avoid invalid interaction operations and improve the interaction efficiency.

[0053] S102: Send a target operation instruction to the target interaction device through the target account.

[0054] After identifying the target interaction device, the electronic device can send a target operation instruction to the target interaction device through the target account, and the target interaction device can execute corresponding operations in response to the received target operation instruction, thereby realizing the interaction between the user and the target interaction device. For example, a query instruction can be sent to the target interaction device to query relevant information of the target interaction device (including information such as the status, performance, and appearance of the target interaction device); a control instruction can also be sent to the target interaction device to control the switch of the target interaction device or adjust device parameters, etc. Combining Figure 2For the audio glasses shown, when a user needs to obtain certain image information, since the audio glasses lack a screen as a functional component, they cannot display images to the user. In such a case, the user can focus their line of sight on a device around them that has a screen as a functional component, such as a mobile phone, a TV on the wall, or a screen in a vehicle, etc. The audio glasses determine the target interaction device that the user is looking at and send an image playback instruction to the target interaction device through the target account, thereby realizing playing the required images for the user on the screen of the target interaction device. The played images (or media data) can be sent by the electronic device (i.e., the audio glasses) to the target interaction device, or can be obtained by the target interaction device itself from the cloud.

[0055] In the above embodiments, in the case where the electronic device cannot implement the functions required by the user, the functions required by the user are implemented through other devices existing around the user and belonging to the target account of the electronic device. This method neither requires adding components to the electronic device nor can greatly improve the user's interaction efficiency and experience.

[0056] In one embodiment, before sending a target operation instruction to the target interaction device, it is also necessary to obtain the user's identity information and determine whether the user has the management authority for the target account based on this identity information. When the target account is the user's own account, the user has the management authority for the target account. When the target account is another user's account, if the other user has opened the management authority of the target account to this user, then this user has the management authority for the target account; if the other user has not opened the management authority of the target account to this user, then this user does not have the management authority for the target account.

[0057] When the user has the management authority for the target account, it means that the user can control the target interaction device through the target account. Therefore, if it is determined that the user has the management authority for the target account, the electronic device can send a target operation instruction to the target interaction device through the target account; if it is determined that the user does not have the management authority for the target account, the electronic device can return a prompt message similar to "no permission" to the user to prompt the user whether to need to replace other interaction devices. This method helps to avoid invalid instruction sending operations and improve the interaction efficiency.

[0058] In one embodiment, the target operation instruction may be a predefined default instruction, and the default instruction includes but is not limited to device on / off instructions, parameter increase / decrease instructions, etc. For example, assuming the target interaction device is an air conditioner, when the electronic device has determined the target interaction device, the electronic device may send the default instruction of turning on the device to the air conditioner to turn on the air conditioner. From the user's perspective, the user only needs to look at the air conditioner, and the air conditioner will turn on. This way of predefining default instructions helps to directly send the predefined default instructions to the target interaction device after determining the target interaction device, thereby improving the interaction efficiency.

[0059] In addition, the target operation instruction may also be an operation instruction temporarily generated by the electronic device in response to a target operation request initiated by the user, and the operation instruction matches the target operation request. For example, the user gazes at a speaker and wants the speaker to play a specific track. In this case, the user can initiate a play request for the specific track, and the electronic device generates a play instruction for the specific track in response to the request and sends the play instruction to the speaker so that the speaker plays the specific track. Among them, the media data of the specific track may be sent by the electronic device to the speaker or obtained by the speaker from the cloud by itself. This way of generating corresponding target operation instructions according to the target operation requests initiated by the user helps to send corresponding operation instructions to the target interaction device according to the actual needs of the user, thereby realizing personalized interaction between the user and the target interaction device.

[0060] In one embodiment, the target interaction device may directly establish a communication connection with the electronic device. In this case, the electronic device may directly send the target operation instruction to the target interaction device. In addition, the target interaction device and the electronic device may also be respectively connected to a control center. In this case, the process of sending the target operation instruction is as follows: the electronic device first sends the target operation instruction to the control center, and then the control center forwards the target operation instruction to the target interaction device.

[0061] Corresponding to the foregoing embodiments of the device linkage method based on directivity interaction, the present disclosure also provides embodiments of a device linkage device based on directivity interaction.

[0062] Please refer to Figure 4 , Figure 4 which is a schematic block diagram of a device linkage device based on directivity interaction provided by an exemplary embodiment. The device may be applied to an electronic device, and the device may include: a device determination unit 402 and an instruction sending unit 404. Among them:

[0063] The device determination unit 402 is configured to determine a target interaction device gazed at by a user, and the target interaction device and the electronic device belong to a target account.

[0064] The instruction sending unit 404 is configured to send a target operation instruction to the target interaction device through the target account.

[0065] Optionally, the device determination unit 402 is specifically configured to: obtain the gaze angle information of the user's eyes and the image captured by the camera; map the gaze angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes; and determine the object included in the target area as the target interaction device gazed by the user.

[0066] Optionally, when the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the camera and the user's eyes pre-calibrated; or, when the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the camera and the user's eyes temporarily calibrated.

[0067] Optionally, the calibration method of the relative position relationship includes at least one of the following: when both the camera and the user's eyes capture a two-dimensional code, obtain the first gaze angle information of the user's eyes for the two-dimensional code and the calibration image containing the two-dimensional code captured by the camera, determine the distance and posture of the camera relative to the two-dimensional code according to the position of the two-dimensional code in the calibration image, and determine the relative position relationship according to the distance and posture and the first gaze angle information; or, when either the camera or the user's eyes have captured a calibration object, obtain the description information of the calibration object and feedback the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object matching the description information, obtain the second gaze angle information of the user's eyes for the calibration object and the distance and posture of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and posture and the second gaze angle information.

[0068] Optionally, the apparatus further includes:

[0069] A judgment unit 406, configured to obtain the identity information of the user and judge whether the user has the management authority for the target account according to the identity information;

[0070] The instruction sending unit 404 is specifically configured to: when it is determined that the user has the management authority for the target account, send the target operation instruction to the target interaction device through the target account.

[0071] Optionally, the instruction sending unit 404 is specifically configured to: when the target interaction device is determined, send a predefined default instruction to the target interaction device through the target account, and / or generate a corresponding target operation instruction in response to a target operation request initiated by the user, and send the target operation instruction to the target interaction device.

[0072] Optionally, the instruction sending unit 404 is specifically configured to: when a communication connection is established between the target interaction device and the electronic device, send the target operation instruction to the target interaction device; or, when the target interaction device and the electronic device are respectively connected to a control center in a communication manner, send the target operation instruction to the control center so that the control center forwards the target operation instruction to the target interaction device.

[0073] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the related methods, and will not be elaborated herein.

[0074] For the device embodiments, since they are basically corresponding to the method embodiments, the relevant parts can refer to the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present disclosure. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0075] Figure 5 It is a schematic block diagram of an electronic device shown according to an embodiment of the present disclosure. For example, the electronic device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0076] Refer to Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502, a memory 504, a power component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0077] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0078] The memory 504 is configured to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, and the like. The memory 504 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0079] The power component 506 provides power to various components of the electronic device 500. The power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 500.

[0080] The multimedia component 508 includes a screen that provides an output interface between the electronic device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0081] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker for outputting audio signals.

[0082] The I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, and the peripheral interface modules may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0083] The sensor component 514 includes one or more sensors for providing status assessments of various aspects of the electronic device 500. For example, the sensor component 514 can detect the on / off state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor component 514 can also detect a change in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor component 514 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 514 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 514 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0084] The communication component 516 is configured to facilitate communication between the electronic device 500 and other devices in a wired or wireless manner. The electronic device 500 can access a wireless network based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0085] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the method described in any of the above embodiments.

[0086] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as a memory 504 including instructions, is further provided. The above instructions can be executed by a processor 520 of the electronic device 500 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0087] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0088] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A device linkage method based on directional interaction, characterized in that, The method is applied to an electronic device, and the method includes: Determine a target interaction device at which the user is gazing, where the target interaction device and the electronic device belong to a target account; Send a target operation instruction to the target interaction device through the target account.

2. The method according to claim 1, wherein The determining of the target interaction device at which the user is gazing includes: Obtain the gazing angle information of the user's eyes and the image captured by the camera; Map the gazing angle information to a target area in the image according to the relative position relationship between the camera and the user's eyes; Determine the object included in the target area as the target interaction device at which the user is gazing.

3. The method according to claim 2, wherein When the camera follows the movement of the user's head, the relative position relationship is the static relative position relationship between the pre-calibrated camera and the user's eyes; or, When the camera does not follow the movement of the user's head, the relative position relationship is the dynamic relative position relationship between the temporarily calibrated camera and the user's eyes.

4. The method according to claim 2, wherein The calibration method of the relative position relationship includes at least one of the following: When both the camera and the user's eyes capture a QR code, obtain the first gazing angle information of the user's eyes for the QR code and the calibration image containing the QR code captured by the camera, determine the distance and pose of the camera relative to the QR code according to the position of the QR code in the calibration image, and determine the relative position relationship according to the distance and pose and the first gazing angle information; or, When either the camera or the user's eyes have captured a calibration object, obtain the description information of the calibration object and feedback the description information to the other party of the camera and the user's eyes, and when the other party captures the calibration object that matches the description information, obtain the second gazing angle information of the user's eyes for the calibration object and the distance and pose of the camera relative to the calibration object, so as to determine the relative position relationship according to the distance and pose and the second gazing angle information.

5. The method according to claim 1, wherein The method further includes: obtaining the identity information of the user, and determining whether the user has the management authority for the target account according to the identity information; The sending of the target operation instruction to the target interaction device through the target account includes: when it is determined that the user has the management authority for the target account, sending the target operation instruction to the target interaction device through the target account.

6. The method according to claim 1, wherein The sending of the target operation instruction to the target interaction device through the target account includes: When the target interaction device is determined, sending a pre-defined default instruction to the target interaction device through the target account, and / or, Generating a corresponding target operation instruction in response to a target operation request initiated by the user, and sending the target operation instruction to the target interaction device.

7. The method according to claim 1, characterized in that, Sending a target operation instruction to the target interaction device through the target account includes: When a communication connection is established between the target interaction device and the electronic device, sending the target operation instruction to the target interaction device; or, When the target interaction device and the electronic device are respectively in communication connection with a control center, sending the target operation instruction to the control center so that the control center forwards the target operation instruction to the target interaction device.

8. A device linkage device based on directional interaction, characterized in that, The device is applied to an electronic device, and the device includes: A device determination unit, configured to determine a target interaction device gazed at by a user, where the target interaction device and the electronic device belong to a target account; An instruction sending unit, configured to send a target operation instruction to the target interaction device through the target account.

9. An electronic device, characterized in that, including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method according to any one of claims 1 to 7.

Images