Method and apparatus for controlling eye gaze communication
The gaze communication control method and device facilitate rapid and convenient wireless communication by detecting objects and establishing connections through gaze recognition, addressing the inconvenience of existing user intervention methods.
Patent Information
- Application Number
- KR1020210043137
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-02
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2041-04-02
AI Technical Summary
Existing methods for wireless communication require user intervention, such as hand interaction or screen selection, making it inconvenient and time-consuming to connect with a communication target.
A gaze communication control method and device that uses cameras to detect objects and extract wireless connection information, allowing users to establish connections without hand interaction by recognizing objects through gaze and mapping them to gaze positions, generating a communication list, and connecting to desired devices using wireless connection information.
Enables quick and convenient wireless communication with desired targets by eliminating the need for hand interaction, utilizing gaze-based object recognition and connection technology.
Smart Images

Figure 112021038853857-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method and apparatus for controlling eye-line communication, and more specifically, to a method and apparatus for controlling eye-line communication that can connect to or control a communication device located in the direction of a user's gaze without user intervention. Background Technology
[0002] In order for a user to establish wireless communication or a wireless call with a communication target, the user reads the information required for the wireless connection and then establishes the connection. To obtain the information necessary for the wireless connection, a method is used in which the user uses their hand to access the device interface or touches a button.
[0003] As such, since user intervention is required to connect with the communication target, it can be inconvenient for the user, and a lot of time is required to connect with the communication target.
[0004] Meanwhile, a gaze communication method has been proposed that allows users to select a communication target by looking at it, thereby minimizing user intervention. However, since the proposed gaze communication method also requires user intervention because the communication target must be selected via a screen, there is a need for a method in gaze communication that allows the user to establish a connection with a communication target without using their hands at all. The problem to be solved
[0005] The problem that the present invention aims to solve is to provide a line-of-sight communication control method and device that allows a user to quickly and conveniently establish a connection with a communication target without using their hands at all. means of solving the problem
[0006] According to one embodiment of the present invention, a gaze communication control method is provided for communicating with a communication target desired by a user in a gaze communication control device. The gaze communication control method comprises the steps of: extracting objects existing in the user's surroundings and information of each object using surrounding images captured as the user observes the surroundings; mapping each object to a gaze position; extracting wireless connection information of a communication device in the user's surroundings; generating a communication list having records of object information and wireless connection information of a communication device corresponding to each object identifier for each object identifier; recognizing an object from an image of the direction of the user's gaze; and attempting to establish a connection using wireless connection information existing in a record corresponding to the object identifier within the communication list.
[0007] The step of recognizing the object may include recognizing the object from an image of the gaze direction captured through a first camera, extracting a recognition area of the object, and calculating the center coordinate value of the pupil from an image of the user's pupil captured through a second camera, and the attempting step may include checking whether the center coordinate value of the pupil is included within the recognition area.
[0008] The step of extracting information of each of the above objects may include the step of calculating the recognition area of each of the above objects and the center position of each of the above objects.
[0009] The above attempting step may include, when the identifier of the object is a person identifier, a step of calculating a distance value between each device identifier and the object corresponding to the person identifier for a plurality of device identifiers owned by the object, and a step of checking whether wireless connection information corresponding to the device identifier exists in the communication list in order of the device identifiers with the smallest distance values.
[0010] The above-mentioned calculating step may include a step of calculating the distance value based on the center position of the object corresponding to each device identifier and the center position of the object corresponding to the person identifier.
[0011] The above-described line-of-sight communication control method may further include the step of repeating the steps of extracting objects existing around the user and information of each object, and extracting wireless attribute information of communication devices around the user, when a communication target desired by the user does not exist among the plurality of device identifiers.
[0012] The above attempting step may include a step of verifying whether the identifier of the object is the communication target desired by the user.
[0013] The above-mentioned eye-line communication control device can be implemented as a glasses type.
[0014] According to another embodiment of the present invention, a gaze communication control device is provided that enables a user to connect with a desired communication target without using their hands. The gaze communication control device includes an object recognition unit, a wireless connection information analysis unit, and a connection control unit. The object recognition unit recognizes an object from an image in the direction of the user's gaze. The wireless connection information analysis unit generates a communication list for objects recognized from surrounding images captured as the user observes the surroundings, wherein the communication list contains records of object information and wireless connection information of a communication device corresponding to the object identifier for each object identifier, and searches the communication list for wireless connection information stored in correspondence with the identifier of the object recognized from the image in the direction of the gaze. The connection control unit establishes a connection using the searched wireless connection information.
[0015] The above-described eye-tracking communication control device may further include an eye tracking unit that calculates the center coordinate value of the pupil from the user's eye image and estimates the position of the gaze the user is looking at using the center coordinate value, and the wireless connection information analysis unit can check whether the gaze position exists within the recognition area of the object.
[0016] The above-described eye communication control device may further include an eye image acquisition unit that captures an image of the user's pupil and transmits the pupil image to the eye tracking unit.
[0017] The above-described gaze communication control device may further include an object image acquisition unit that captures an image of the direction of the user's gaze and transmits the captured image to the object recognition unit.
[0018] The above-described line-of-sight communication control device may further include a wireless connection information acquisition unit that extracts wireless connection information of a communication device existing around the user, and the object recognition unit may recognize objects existing around the user using the user's surrounding image and extract the object information.
[0019] When the identifier of an object recognized from the image of the line of sight is a person identifier, the above wireless connection information analysis unit calculates the distance value between each device and the object for multiple devices owned by the object, and checks whether wireless connection information corresponding to the identifier of the device exists in the communication list in order of the devices with the smallest distance values.
[0020] The above wireless access information analysis unit can calculate the distance value based on the center location of the object corresponding to each device and the center location of the object corresponding to the person identifier. Effects of the invention
[0021] According to an embodiment of the present invention, the user can connect to and control a communication target more quickly and conveniently without using their hands at all, through the user's gaze. Brief explanation of the drawing
[0022] FIG. 1 is a diagram showing the configuration of a line-of-sight communication control device according to one embodiment of the present invention. Figure 2 is a diagram showing an example of a wireless connection information acquisition unit illustrated in Figure 1. Figure 3 is a diagram showing an example of how the line-of-sight communication control device illustrated in Figure 1 is implemented. FIG. 4 is a diagram showing an example of object detection and gaze tracking according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of acquiring wireless connection information in a wireless connection information acquisition unit according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of information analysis in an information analysis unit according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a method for scanning and storing object information and wireless connection information of a communication device according to an embodiment of the present invention. FIGS. 8 and FIGS. 9 are flowcharts illustrating in detail a line-of-sight communication control method according to an embodiment of the present invention. FIG. 10 is a flowchart explaining the process for connecting to the corresponding device in order of decreasing distance value shown in FIG. 9. FIG. 11 is a drawing showing a line-of-sight communication control device according to another embodiment of the present invention. Specific details for implementing the invention
[0023] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.
[0024] Throughout the specification and claims, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0025] Now, a line-of-sight communication control method and apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.
[0026] FIG. 1 is a diagram showing the configuration of a line-of-sight communication control device according to one embodiment of the present invention, and FIG. 2 is a diagram showing an example of a wireless connection information acquisition unit illustrated in FIG. 1.
[0027] Referring to FIG. 1, the eye-line communication control device (100) includes an object image acquisition unit (110), an eye-line image acquisition unit (120), an object recognition unit (130), an eye-line tracking unit (140), a wireless connection information acquisition unit (150), a wireless connection information analysis unit (160), and a connection control unit (170). Additionally, the eye-line communication control device (100) may further include a database (DB) (180).
[0028] The object image acquisition unit (110) captures an image in the direction of the user's gaze and transmits the captured image to the object recognition unit (130). The image captured by the object image acquisition unit (110) is used for detecting objects. For convenience of explanation below, the image captured by the object image acquisition unit (110) is referred to as an object image.
[0029] The gaze image acquisition unit (120) captures the user's pupil image in real time and transmits the captured pupil image to the gaze tracking unit (140).
[0030] The object image acquisition unit (110) and the gaze image acquisition unit (120) may each include a camera (not shown in the drawing) that captures an object image and an eye image. The camera of the object image acquisition unit (110) may be positioned so that its shooting direction faces the direction of the gaze of the user. Thus, assuming that the user is looking at an object, the camera of the object image acquisition unit (110) acquires an image of the object in front of the user.
[0031] The object recognition unit (130) recognizes all objects within the object image and stores information of the recognized objects in the DB (180). Here, the objects may include, for example, people, communication devices such as mobile phones or computers, etc. Additionally, the object information may include information about the object's recognition area, the center location of the recognition area, and the identification (ID) of the recognized object. Additionally, the object information may further include the ID of another recognized object located closest to the object.
[0032] The object recognition unit (130) stores information of the object in the DB (180) corresponding to the object ID.
[0033] The gaze tracking unit (140) tracks the user's gaze position from the pupil image. The gaze tracking unit (140) can detect the center of the pupil from the pupil image and extract the center coordinate value of the pupil. When the gaze tracking unit (140) extracts the center coordinate value of the pupil, it can track the movement of the pupil through the center coordinate value of the pupil and calculate the user's gaze position. When the eye scans the surrounding environment or gazes at a specific object, the gaze tracking unit (140) extracts the center coordinate value of the pupil from the pupil image and tracks the movement of the gaze over time.
[0034] In addition, the eye tracking unit (140) can map the extracted center coordinate values of the pupil to the objects recognized from the object image in the object recognition unit (130) to identify the object of interest at the location the user is looking at.
[0035] The wireless connection information acquisition unit (150) scans a peripheral communication device to acquire wireless connection information of the peripheral communication device. The wireless connection information may include communication attribute information and owner information necessary for communication and connection. The wireless connection information acquisition unit (150) stores the acquired wireless connection information of the peripheral communication device in the DB (180). The wireless connection information acquisition unit (150) can store the acquired wireless connection information of the peripheral communication device in correspondence with the same object ID within the DB (180). That is, the wireless connection information acquisition unit (150) can store the corresponding wireless connection information in correspondence with the same object ID as the ID of the acquired peripheral communication device within the DB (180). Additionally, if an object ID corresponding to the owner information of the acquired peripheral communication device exists within the DB (180), the wireless connection information acquisition unit (150) stores the corresponding wireless connection information in correspondence with the object ID corresponding to the owner information.
[0036] Referring to FIG. 2, the wireless connection information acquisition unit (150) may include, for example, a Wi-Fi information scanner (152) and a Bluetooth information scanner (154). In the embodiment, Wi-Fi information and Bluetooth information were described as examples of wireless connection information, but the unit is not limited thereto.
[0037] The Wi-Fi information scanner (152) collects Wi-Fi communication attribute information through a USB (Universal serial bus) I / O interface. The Wi-Fi communication attribute information may include an IP (Internet protocol) address, an AP (Access point) ID, owner information, signal strength, etc. The Bluetooth information scanner (154) collects Bluetooth communication attribute information. The Bluetooth communication attribute information may include a Bluetooth device address, a Bluetooth device ID, and owner information, etc.
[0038] Again, looking at FIG. 1, the wireless connection information analysis unit (160) analyzes the positional relationships of objects recognized by the object recognition unit (130) and the wireless connection information of the surrounding communication device.
[0039] When a user looks at a communication target, the object recognition unit (130) recognizes an object ID from an object image obtained from the direction the user is looking, and the gaze tracking unit (140) calculates the gaze position from the user's pupil image. If the gaze position is within the object's recognition area, the wireless connection information analysis unit (160) checks whether the recognized object ID exists in the DB (180). The wireless connection information analysis unit (160) checks whether wireless connection information stored in the DB (180) corresponding to the object ID exists, and transmits the communication target ID and wireless connection information to the connection control unit (170).
[0040] The connection control unit (170) connects to the communication target using the wireless connection information of the estimated communication target. At this time, the connection control unit (170) can perform a process of verifying to the user whether the estimated communication target is the device desired by the user.
[0041] Figure 3 is a diagram showing an example of how the line-of-sight communication control device illustrated in Figure 1 is implemented.
[0042] As shown in FIG. 3, the eye-line communication control device (100) can be implemented in glasses (200).
[0043] The gaze image acquisition unit (120) may include a small camera and a reflective prism. The small camera may be mounted on the eyeglass lens, and the reflective prism may be attached to the surface of the eyeglass lens. When a user wears the glasses and looks at a specific location, the small camera captures an image of the pupil reflected by the reflective prism. At this time, to compensate for the loss of light intensity of the pupil image caused by the reflective prism, a Total Reflection Mirror (TRM) may be used instead of the reflective prism. The TRM may be attached to the surface of the eyeglass lens and may be mounted on the glasses in a structure separate from the eyeglass lens.
[0044] The object image acquisition unit (110) and the gaze image acquisition unit (120) may be mounted on the bridge portion (not shown in the drawing) connecting the left and right lenses of the glasses. The wireless connection information acquisition unit (150) may be mounted at an appropriate location on the temple portion of the glasses. The object recognition unit (130), the gaze tracking unit (140), the wireless connection information analysis unit (160), and the connection control unit (170) may each be mounted separately on the components constituting the glasses (200), but may also be mounted inside the glasses as a single computing device (210).
[0045] The eye-line communication control device (100) can be implemented in wearable products such as hats, goggles, headbands, and helmets in addition to glasses. The eye-line communication control device (100) can capture an image of the user's pupils and can be implemented in a position where the image can be captured according to the user's gaze direction. In the embodiment, an eye-line communication control device (100) of the glasses type is assumed and described.
[0046] The user initially wears a glasses-type eye-tracking communication control device (100) and observes the surroundings. Then, the eye-tracking image acquisition unit (120) captures an image of the user's pupils in real time, and the object image acquisition unit (110) captures an image of the surroundings at the position of the user's gaze.
[0047] FIG. 4 is a diagram showing an example of object detection and gaze tracking according to an embodiment of the present invention.
[0048] Referring to FIG. 4, the user wears a glasses-type eye-tracking communication control device (100) and observes the surroundings up to 360 degrees.
[0049] The object image acquisition unit (110) captures an object image at a viewing angle set in the direction of the user's gaze and transmits the captured object image to the object recognition unit (130).
[0050] The gaze image acquisition unit (120) captures the user's pupil image in real time and transmits the captured pupil image to the gaze tracking unit (140).
[0051] The gaze tracking unit (140) detects the pupil from the eye image and estimates the user's gaze direction and gaze position by extracting the center coordinate value of the pupil.
[0052] The object recognition unit (130) recognizes an object existing within an object image captured by the object image acquisition unit (110), and extracts an object ID, an object recognition area, and a center position of the object within the recognition area.
[0053] The object recognition unit (130) maps the recognized object to the gaze position, extracts the object ID, the object recognition area, and the center position of the object within the recognition area, and stores the object information including the object ID, the object recognition area, and the center position of the object in the DB (180). The object recognition area can be represented, for example, as rectangular area information based on the position coordinates of the top left and the bottom right.
[0054] For example, the object recognition unit (130) can recognize a smartphone and a person from the object image (400) shown in FIG. 4. The object recognition unit (130) extracts the ID of the recognized smartphone, "Smart-phone," the recognition area of the smartphone, and the center position of the smartphone. It also extracts the identifier of the recognized person, "Betty," the recognition area of the person, and the center position of the person. The object information extracted in this way is stored in the DB (180).
[0055] FIG. 5 is a diagram showing an example of acquiring wireless connection information in a wireless connection information acquisition unit according to an embodiment of the present invention.
[0056] Referring to FIG. 5, when a user wears a glasses-type eye-tracking communication control device (100) and observes the surrounding environment up to 360 degrees, the wireless connection information acquisition unit (150) scans the wireless connection information of the surrounding communication device.
[0057] For example, the wireless connection information acquisition unit (150) collects Wi-Fi communication attribute information and Bluetooth communication attribute information. The communication attribute information collected at this time may include a device ID, the owner ID of the device, a Wi-Fi address, a Bluetooth device address, etc.
[0058] FIG. 6 is a diagram showing an example of information analysis in a wireless connection information analysis unit according to an embodiment of the present invention.
[0059] Referring to FIG. 6, the wireless connection information analysis unit (160) analyzes the gaze position estimated by the gaze tracking unit (140) and the information of the object recognized by the object recognition unit (130) to determine the communication device corresponding to the communication target, and extracts wireless attribute information of the communication device corresponding to the communication target from the DB (180).
[0060] In addition, the wireless connection information analysis unit (160) can determine the communication device corresponding to the line of sight position by analyzing the relationship between the recognized object and the communication device.
[0061] Next, the connection control unit (170) can connect to or control the communication device by using the wireless attribute information of the communication device corresponding to the communication target.
[0062] FIG. 7 is a flowchart illustrating a method for scanning and storing object information and wireless connection information of a communication device according to an embodiment of the present invention.
[0063] Referring to FIG. 7, the user wears a glasses-type eye-tracking communication control device (100) and observes the surrounding environment up to 360 degrees.
[0064] The object image acquisition unit (110) captures an object image of a field of view set in the direction of the user's gaze and stores the captured object image (S710). At the same time, the wireless connection information acquisition unit (150) scans a peripheral communication device and extracts wireless attribute information of the peripheral communication device (S720).
[0065] The object recognition unit (130) recognizes objects from the acquired object image (S730), extracts the object ID of the recognized object, and calculates the recognition area and the center position of the object for each object ID (S740).
[0066] The object recognition unit (130) stores object information, including a recognition area and a center position of an object, for each object ID in a first file of the DB (180) (S750). After mapping the recognized object to a gaze position, the object recognition unit (130) can store object information for each object ID in the first file of the DB (180).
[0067] The wireless connection information acquisition unit (150) stores wireless attribute information for each ID of the peripheral communication device and / or owner ID in the second file of the DB (180) (S760).
[0068] The information analysis unit (160) generates a communication list for each object ID, each record containing communication speed information corresponding to the same object ID as the object information (S770), and stores and manages the generated communication list in the DB (180) (S780).
[0069] Next, when a user looks at a communication device to communicate with, a method for connecting to the communication device in the line-of-sight communication control device (100) is explained in detail with reference to FIGS. 8 and FIGS. 9.
[0070] FIGS. 8 and FIGS. 9 are flowcharts illustrating in detail a line-of-sight communication control method according to an embodiment of the present invention.
[0071] Referring to FIG. 8, when a user looks at a communication device to communicate with, the object image acquisition unit (110) of the gaze communication control device (100) captures an object image in the direction of the user's gaze, and the object recognition unit (130) recognizes an object from the object image and extracts the object ID and recognition area information of the recognized object (S802).
[0072] In addition, the gaze image acquisition unit (120) of the gaze communication control device (100) captures an image of the user's pupil, and the gaze tracking unit (140) detects the pupil from the pupil image and extracts the center coordinate value of the pupil, thereby estimating the user's gaze position.
[0073] The wireless attribute information analysis unit (160) extracts wireless attribute information of the recognized object when the center coordinate value of the pupil is included in the recognition area information. To do this, the wireless attribute information analysis unit (160) checks whether the object ID is a device ID corresponding to the device name (S804).
[0074] The wireless attribute information analysis unit (160) checks whether wireless attribute information exists in the record corresponding to the device ID (S806).
[0075] If wireless attribute information analysis unit (160) has wireless attribute information in a record corresponding to a device ID, it transmits the wireless attribute information of the device ID to the connection control unit (170).
[0076] The connection control unit (170) checks whether the device ID is a communication device desired by the user (S808), and if the device ID is a communication device desired by the user, outputs a device connection message, connects to the device using the wireless attribute information of the device ID, and controls the device (S810).
[0077] Referring to FIG. 9, the wireless attribute information analysis unit (160) checks if the object ID is not a device ID corresponding to a device name, or if there is no wireless attribute information in the record corresponding to the device ID, whether the object ID is a person ID representing a person name (S902).
[0078] In the wireless attribute information analysis unit (160), if the object ID is a person ID, it checks whether the person ID exists in the DB (180).
[0079] The wireless attribute information analysis unit (160) checks from the DB (180) whether there is a device ID corresponding to a person ID, and checks whether there are multiple device IDs corresponding to a person ID (S904).
[0080] If there are multiple device IDs corresponding to a person ID, the wireless attribute information analysis unit (160) extracts a center position from a record corresponding to a person ID for each device ID and extracts a center position from a record corresponding to a device ID, and calculates a distance value between the center position of the person ID and the center position of each device ID (S906).
[0081] The wireless attribute information analysis unit (160) rearranges all device IDs using matching keys in order of smallest distance value based on the distance values calculated for multiple device IDs (S908).
[0082] The wireless attribute information analysis unit (160) performs a process for connecting to the corresponding communication device in order of smallest distance value (S910). The process for connecting to the corresponding communication device in order of smallest distance value is explained with reference to FIG. 10.
[0083] Meanwhile, the wireless attribute information analysis unit (160) checks whether wireless attribute information exists in the record corresponding to the device ID if there is one device ID corresponding to the person ID (S912) (S914).
[0084] If the information analysis unit (160) has wireless attribute information in a record corresponding to a device ID, it transmits the wireless attribute information of the device ID to the connection control unit (170).
[0085] The connection control unit (170) checks whether the device ID is a communication device desired by the user (S916), and if the device ID is a communication device desired by the user, outputs a device connection message, connects to the device using the wireless attribute information of the device ID, and controls the device (S918).
[0086] Meanwhile, the wireless connection information analysis unit (160) can control the object recognition unit (130) and the wireless connection information acquisition unit (150) so that object re-recognition and peripheral communication device re-scanning are performed if there is no device ID owned by a person ID in the DB (180), or if there are multiple device IDs owned by a person ID in the DB (180) but no communication device desired by the user.
[0087] The object recognition unit (130) performs an object re-recognition operation, and the wireless connection information acquisition unit (150) scans the peripheral communication device again (S920).
[0088] FIG. 10 is a flowchart explaining the process for connecting to the corresponding device in order of decreasing distance value shown in FIG. 9.
[0089] Referring to FIG. 10, the wireless attribute information analysis unit (160) sets n=1 (S1002).
[0090] The wireless attribute information analysis unit (160) checks whether n is greater than m (S1004). m represents the number of device IDs when there are multiple device IDs corresponding to a person ID.
[0091] The wireless attribute information analysis unit (160) checks whether wireless attribute information exists in the record corresponding to the nth device ID if n is less than m (S1004). That is, since the device IDs are rearranged in order of smallest distance value, n=1 represents the device ID with the smallest distance value.
[0092] If wireless attribute information analysis unit (160) has wireless attribute information in a record corresponding to the nth device ID, it transmits the wireless attribute information of the device ID to the connection control unit (170).
[0093] The connection control unit (170) checks whether the device ID is a communication device desired by the user (S1008), and if the device ID is a communication device desired by the user, outputs a device connection message, connects to the device using the wireless attribute information of the device ID, and controls the device (S1010).
[0094] Meanwhile, if the device ID is not the communication device desired by the user, or if wireless attribute information does not exist in the record corresponding to the nth device ID, the wireless connection information analysis unit (160) updates n to n+1 (S1012), and repeats steps (S1004~1010) for the next device ID in sequence.
[0095] In this way, the wireless attribute information analysis unit (160) can determine the communication device desired by the user without user intervention.
[0096] If multiple device IDs corresponding to a person ID are not all communication devices desired by the user, the wireless connection information analysis unit (160) can control the object recognition unit (130) and the wireless connection information acquisition unit (150) so that object re-recognition and peripheral communication device re-scanning are performed.
[0097] FIG. 11 is a drawing showing a line-of-sight communication control device according to another embodiment of the present invention.
[0098] Referring to FIG. 11, the line-of-sight communication control device (1100) may represent a computing device in which the line-of-sight communication control method according to the embodiment of the present invention described above is implemented.
[0099] The line-of-sight communication control device (1100) may include at least one of a processor (1110), a memory (1120), an input interface device (1130), an output interface device (1140), and a storage device (1150). Each component may be connected by a common bus (1160) to communicate with each other. Additionally, each component may be connected via an individual interface or an individual bus centered around the processor (1110), rather than via the common bus (1160).
[0100] The processor (1110) can be implemented in various types such as an Application Processor (AP), a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), etc., and may be any semiconductor device that executes instructions stored in memory (1120) or a storage device (1150). The processor (1110) can execute program instructions stored in at least one of memory (1120) and the storage device (1150). Such a processor (1110) may be configured to implement the functions and methods described above based on FIGS. 1 to 10. For example, the processor (1110) may be configured to perform at least some of the functions of the object recognition unit (130), eye tracking unit (140), wireless connection information acquisition unit (150), wireless connection information analysis unit (160), and connection control unit (170) described in FIG. 1.
[0101] The memory (1120) and storage device (1150) may include various forms of volatile or non-volatile storage media. For example, the memory (1120) may include ROM (read-only memory) (1121) and RAM (random access memory) (1122). In an embodiment of the present invention, the memory (1120) may be located inside or outside the processor (1110), and the memory (1120) may be connected to the processor (1110) through various known means. The memory (1120) or storage device (1150) may include the DB (180) described in FIG. 1.
[0102] The input interface device (1130) is configured to provide data (e.g., object image, pupil image, etc.) to the processor (1110), and the output interface device (1140) is configured to output data (e.g., inferred device ID, device connection result) from the processor (1110).
[0103] In addition, at least some of the line-of-sight communication control methods according to embodiments of the present invention may be implemented as a program or software executed on a computing device, and the program or software may be stored on a computer-readable medium.
[0104] In addition, at least some of the line-of-sight communication control methods according to embodiments of the present invention may be implemented in hardware that can be electrically connected to a computing device.
[0105] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.
Claims
Claim 1 A method for controlling eye-tracking communication to communicate with a communication target desired by a user in an eye-tracking communication control device comprises: a step of extracting information of each object including objects existing in the surroundings of the user, a recognition area and a center location of each object, using surrounding images captured as the user observes the surroundings; a step of mapping each object to a gaze location; a step of extracting wireless connection information including owner information of a communication device in the surroundings of the user; a step of creating a communication list having records of object information and wireless connection information of a communication device corresponding to each object identifier, wherein if an object identifier corresponding to the owner information of the wireless connection information of the communication device exists, the wireless connection information is stored as a record corresponding to the object identifier corresponding to the owner information; a step of recognizing an object from an image of the direction of gaze the user is looking at; and a step of attempting to connect using wireless connection information existing in a record corresponding to the object identifier in the communication list. The step of recognizing an object includes a step of recognizing an object from an image of the direction of gaze captured through a first camera and extracting a recognition area of the object, and a step of calculating the center coordinate value of the pupil from an image of the user's pupil captured through a second camera. The attempting step is to check whether the center coordinate value of the pupil is included within the recognition area, and if the center coordinate value of the pupil is included within the recognition area, to attempt a connection using wireless connection information existing in a record corresponding to the identifier of the object in the communication list.A method for controlling line-of-sight communication, comprising the step of, when the identifier of the recognized object is a person identifier, calculating a distance value for each of a plurality of device identifiers corresponding to the person identifier based on the center position of the object corresponding to the device identifier and the center position of the object corresponding to the person identifier, and attempting to connect by checking whether wireless connection information corresponding to the device identifier exists in the communication list in order of smallest distance value. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 A line-of-sight communication control method according to claim 1, further comprising the step of repeating the steps of extracting objects existing around the user and information of each object, and extracting wireless connection information of a communication device around the user, when a communication target desired by the user does not exist among the plurality of device identifiers. Claim 7 A method for controlling eye-line communication according to claim 1, wherein the attempting step further includes a step of verifying whether the identifier of the object is the communication target desired by the user. Claim 8 In claim 1, the above-mentioned eye-line communication control device is an eye-line communication control method implemented as a glasses type. Claim 9 A gaze communication control device that enables a user to connect with a desired communication target without using their hands comprises: an object recognition unit that recognizes an object from an image of the gaze direction the user is looking at; for objects recognized from surrounding images captured as the user observes the surroundings, a communication list is generated having records of object information and wireless connection information of a communication device corresponding to the object identifier for each object identifier, wherein the object information includes the object's recognition area and center location, and the wireless connection information includes owner information; the wireless connection information is stored as a record corresponding to the object identifier corresponding to the owner information of the wireless connection information of the communication device; a wireless connection information analysis unit that searches the wireless connection information stored in the communication list corresponding to the identifier of the object recognized from the image of the gaze direction; and a connection control unit that connects using the searched wireless connection information. The object recognition unit recognizes an object from the image of the gaze direction captured through a first camera and extracts the recognition area of the object, and calculates the center coordinate value of the pupil from the image of the user's pupil captured through a second camera. The wireless connection information analysis unit determines whether the center coordinate value of the pupil is within the recognition area. A line-of-sight communication control device that checks whether it is included, and if the center coordinate value of the pupil is included within the recognition area, attempts to connect using wireless connection information existing in a record corresponding to the identifier of the object in the communication list, and if the identifier of the recognized object is a person identifier, calculates a distance value for each of the plurality of devices corresponding to the person identifier based on the center position of the object corresponding to the device and the center position of the object corresponding to the person identifier, and attempts to connect by checking whether wireless connection information corresponding to the identifier of the device exists in the communication list in order of smallest distance value. Claim 10 A gaze communication control device according to claim 9, further comprising a gaze tracking unit that estimates the gaze position of the user using the center coordinate values. Claim 11 A gaze communication control device according to claim 10, further comprising a gaze image acquisition unit that captures an image of the user's pupil and transmits the pupil image to the gaze tracking unit. Claim 12 A gaze communication control device according to claim 9, further comprising an object image acquisition unit that captures an image of the direction of the user's gaze and transmits the captured image to the object recognition unit. Claim 13 In claim 9, the line-of-sight communication control device further includes a wireless connection information acquisition unit that extracts wireless connection information of a communication device existing in the vicinity of the user, and the object recognition unit recognizes objects existing in the vicinity of the user using the user's surrounding image and extracts the object information. Claim 14 delete Claim 15 delete
Citation Information
Patent Citations
Method and apparatus For Connecting Devices Using Eye-tracking
KR1020150007643A
Method and apparatus for recognizing object using multi-sensing
KR1020180046586A
Artificial intelligence device that can be controlled according to user gaze
KR1020190085895A
