System and method for artificial intelligence agent considering user's gaze

Abstract

The present disclosure relates to a method for operating an artificial intelligence agent in consideration of a gaze, wherein the position of a virtual object may be configured, the virtual object may be output, a user's gaze information may be acquired, it may be confirmed whether the virtual object's activation condition is satisfied in consideration of the user's gaze information, and the user's utterance may be processed without a wake word if the virtual object's activation condition is satisfied as a result of confirmation.

Description

Artificial intelligence agent system and method considering the user's perspective

[0001] The present invention relates to an artificial intelligence agent system and method that considers a user's gaze in an extended reality (XR) device.

[0002] Most current voice assistant systems require the user to utter a wake word before actually uttering the command, and then the actual command to the voice assistant. For example, in "Hi Bixby, what's the weather today?", the "Hi Bixby" part is not the actual command, but rather the wake word to initiate input to the voice assistant.

[0003] Interactions using wake words can feel unnatural because they require the system to use these utterances, which can be omitted in real-life conversations between people.

[0004] Therefore, there are several prior technologies to resolve the inconvenience of users using wake words.

[0005] For example, a system that cuts off the beginning and end of the user's conversation and sends the corresponding sentence to the server to classify the user's intent when the wake word is spoken in the middle of a conversation, or a system that continuously keeps the microphone on without the wake word and collects the user's speech and then determines the user's intent based on this.

[0006] However, these existing technologies require continuous system resource allocation to analyze user speech. For example, analyzing user conversations requires continuous computations from automatic speech recognition (ASR) modules and natural language understanding (NLU) modules. However, continuously running these modules for discontinuous voice assistant use requires continuous consumption of system resources. Furthermore, in addition to system efficiency issues, the user's voice data is processed into data, including unnecessary parts for analysis, not just the utterances used for commands, which can also raise privacy concerns.

[0007] The above information may be provided as background art to aid in understanding the present disclosure. No claim is made regarding whether any of the above is applicable as prior art related to the present disclosure.

[0008] An embodiment of the present invention seeks to provide an artificial intelligence agent system and method that takes into account a user's gaze.

[0009] A method for operating an artificial intelligence agent by considering a gaze according to one embodiment may include: setting a position of a virtual object; outputting the virtual object; acquiring gaze information of a user; checking whether an activation condition of the virtual object is satisfied by considering the gaze information of the user; and processing the user's speech without a wake word if the activation condition of the virtual object is satisfied as a result of the checking.

[0010] At this time, the operation of checking whether the activation condition of the virtual object is met by considering the gaze information of the user can be performed by checking whether the activation condition of the virtual object is met based on the user looking at the virtual object by considering the gaze information of the user and the location of the virtual object.

[0011] At this time, the method of operating the artificial intelligence agent by considering the gaze may further include an action of confirming the location of the user, and the action of setting the location of the virtual object may include an action of setting the location of the virtual object by considering the location of the user.

[0012] At this time, the method of operating the artificial intelligence agent by considering the line of sight may further include an operation of confirming the location of the user; and an operation of analyzing the area where the user is located using an image input through a camera, and the operation of setting the location of the virtual object may include an operation of setting the location of the virtual object by considering the location of the user and information of the analyzed area.

[0013] At this time, the method of operating the artificial intelligence agent by considering the gaze further includes an operation of confirming the location of the user, and the operation of confirming whether the activation condition of the virtual object is satisfied by considering the gaze information of the user may be determined based on the user being within a preset distance of the virtual object and the user looking at the virtual object by considering the gaze information of the user, the location of the user, and the location of the virtual object, and whether the activation condition of the virtual object is satisfied.

[0014] At this time, the operation of setting the position of the virtual object further includes the operation of setting the field of view (FOV) of the virtual object together with the position of the virtual object, and the operation of outputting the virtual object outputs the virtual object by displaying the field of view of the virtual object or the eyes of the virtual object based on the output of the virtual object, and the operation of checking whether the activation condition of the virtual object is met by considering the gaze information of the user can check whether the activation condition of the virtual object is met based on the user and the virtual object looking at each other by considering the gaze information of the user, the position of the virtual object, and the field of view of the virtual object.

[0015] At this time, the method for operating an artificial intelligence agent by considering the gaze further includes an operation of confirming the position of the user, and the operation of setting the position of the virtual object further includes an operation of setting the field of view (FOV) of the virtual object together with the position of the virtual object, and the operation of outputting the virtual object outputs the virtual object by displaying the field of view or the eyes of the virtual object based on the output of the virtual object, and the operation of checking whether the activation condition of the virtual object is met by considering the gaze information of the user can check whether the activation condition of the virtual object is met based on the fact that the user is present within a preset distance of the virtual object and the user and the virtual object are looking at each other by considering the gaze information of the user, the position of the virtual object, and the field of view of the virtual object.

[0016] At this time, the method of operating the artificial intelligence agent by considering the gaze may further include an operation of adjusting the virtual object so that the virtual object satisfies the activation condition, based on the wake word input when the result of the verification does not satisfy the activation condition of the virtual object.

[0017] At this time, the action of outputting the virtual object may output the virtual object in the form of a preset character and output the virtual object so as to perform the action pattern of the preset character.

[0018] According to one embodiment, an artificial intelligence agent system considering gaze may include: a first sensor for sensing a user's eyes; a display unit including a display on which a virtual object is output; a virtual object setting unit including a circuit for setting a position of the virtual object; a virtual object visualization unit including a circuit for outputting the virtual object through the display unit; a user sensing unit including a circuit for obtaining gaze information of the user through the first sensor; a virtual object activation unit including a circuit for determining whether an activation condition of the virtual object is met by considering the gaze information of the user; a speech collection unit including a circuit for collecting the user's speech without a wake word when the virtual object activation unit determines that the activation condition of the virtual object is met; and a speech processing unit including a circuit for processing the user's speech collected by the speech collection unit.

[0019] At this time, the virtual object activation unit can determine that the activation condition of the virtual object is met based on the user looking at the virtual object by considering the user's gaze information and the location of the virtual object.

[0020] At this time, the artificial intelligence agent system considering the gaze further includes a second sensor for confirming the location, the user sensing unit confirms the location of the user through the second sensor, and the virtual object setting unit can set the location of the virtual object by considering the location of the user.

[0021] At this time, the artificial intelligence agent system considering the gaze further includes a second sensor for confirming the location; and a third sensor for collecting an image of the direction in which the user is looking, wherein the user sensing unit confirms the location of the user through the second sensor and analyzes the area in which the user is located using the image, and the virtual object visualization unit can set the location of the virtual object by considering the location of the user and information of the analyzed area.

[0022] At this time, the artificial intelligence agent system considering gaze further includes a second sensor for confirming a location, and the user sensing unit confirms the location of the user through the second sensor, and the virtual object activation unit can confirm that the user exists within a preset distance of the virtual object and that the user is looking at the virtual object based on consideration of the user's gaze information, the user's location, and the location of the virtual object, and that the activation condition of the virtual object is met.

[0023] At this time, the virtual object visualization unit sets the field of view (FOV) of the virtual object together with the location of the virtual object, and based on the output of the virtual object, displays the field of view of the virtual object or the eyes of the virtual object to output the virtual object, and the virtual object activation unit can determine that the activation condition of the virtual object is met based on the user and the virtual object looking at each other by considering the gaze information of the user, the location of the virtual object, and the field of view of the virtual object.

[0024] At this time, the artificial intelligence agent system considering the gaze further includes a second sensor for confirming the location, and the user sensing unit confirms the location of the user through the second sensor, the virtual object visualization unit sets the field of view (FOV) of the virtual object together with the location of the virtual object, and based on the output of the virtual object, displays the field of view of the virtual object or the eyes of the virtual object to output the virtual object, and the virtual object activation unit can determine that the user is present within a preset distance of the virtual object and that the user and the virtual object are looking at each other, and that the activation condition of the virtual object is satisfied based on the user's gaze information, the location of the virtual object, and the field of view of the virtual object.

[0025] At this time, the virtual object visualization unit can adjust the virtual object so that it meets the activation condition according to the wake word input through the utterance collection unit in a state where the virtual object activation unit confirms that the virtual object does not meet the activation condition.

[0026] At this time, the virtual object setting unit can set the virtual object in the form of a preset character and set the virtual object to perform the behavior pattern of the preset character.

[0027] An artificial intelligence agent system considering gaze according to one embodiment may include a processor, which may individually or collectively include a first sensor for sensing a user's eyes; a display unit including a display on which a virtual object is output; and a processing circuit configured to set a position of the virtual object, output the virtual object through the display unit, acquire gaze information of the user through the first sensor, determine whether an activation condition of the virtual object is satisfied by considering the gaze information of the user, and process the user's utterance without a wake word if the activation condition of the virtual object is satisfied as a result of the determination.

[0028] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0029] FIG. 1 is a flowchart illustrating an exemplary process for operating an artificial intelligence agent by considering a user's gaze in an artificial intelligence agent system according to various embodiments.

[0030] FIG. 2 is a flowchart illustrating an exemplary process of operating an artificial intelligence agent by considering the user's location and the user's gaze in an artificial intelligence agent system according to various embodiments.

[0031] FIG. 3 is a diagram illustrating an example of using a virtual object in an artificial intelligence agent system according to various embodiments.

[0032] FIG. 4a is a diagram illustrating an example of displaying the status of a virtual object in an artificial intelligence agent system according to various embodiments.

[0033] FIG. 4b is a diagram illustrating an example of an area in which interaction with a virtual object can be performed in an artificial intelligence agent system according to various embodiments.

[0034] FIG. 5 is a diagram illustrating an example of a case in which a wake word can be omitted in an artificial intelligence agent system according to various embodiments.

[0035] FIG. 6 is a diagram illustrating an example of a case in which a wake word can be omitted in an artificial intelligence agent system according to various embodiments.

[0036] FIG. 7 is a diagram illustrating an example of setting the location of a virtual object by analyzing the area where a user is located in an artificial intelligence agent system according to various embodiments.

[0037] FIG. 8 is a diagram illustrating an example of a case where multiple users are targeted in an artificial intelligence agent system according to various embodiments.

[0038] FIG. 9 is a block diagram illustrating an exemplary configuration of an artificial intelligence agent system according to various embodiments.

[0039] FIG. 10 is a block diagram of an exemplary electronic device within a network environment according to various embodiments.

[0040] FIG. 11 is a diagram illustrating an exemplary configuration and structure of an electronic device implemented in the form of wearable augmented reality glasses according to various embodiments.

[0041] Below, various embodiments are described in detail with reference to the attached drawings. However, the various embodiments may be modified in various ways, and the scope of the patent application is not intended to be limited by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0042] The terminology used in the various embodiments is for the purpose of description only and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0044] Additionally, when describing with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof may be omitted. When describing an embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description may be omitted.

[0045] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the embodiments. These terms are merely intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

[0046] Components included in one embodiment and components with common functions will be described using the same names in other embodiments. Unless otherwise stated, descriptions made in one embodiment may also apply to other embodiments, and specific descriptions may be omitted to the extent of overlap.

[0047] Hereinafter, an artificial intelligence agent system and method that take into account a user's gaze according to various embodiments of the present invention will be described in detail with reference to the attached FIGS. 1 to 11.

[0048] FIG. 1 is a flowchart illustrating an exemplary process for operating an artificial intelligence agent by considering a user's gaze in an artificial intelligence agent system according to various embodiments.

[0049] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0050] According to an embodiment, operations 110 to 150 may be understood to be performed by a processor (e.g., processor (910) of FIG. 9) of an artificial intelligence agent system (e.g., artificial intelligence agent system (900) of FIG. 9).

[0051] Referring to Figure 1, the artificial intelligence agent system sets the location of a virtual object (110). The artificial intelligence agent system can set a preset (designated) area of ​​the display as the location of the virtual object.

[0052] As another example, an AI agent system can collect images corresponding to the user's field of view through a camera, analyze the images to determine the user's location, and then determine the location of a virtual object based on the analyzed information. For example, if the AI ​​agent system analyzes the user's field of view image and determines that the user is in the kitchen, it can place a virtual object on the kitchen counter. Alternatively, if the user is in the living room, the AI ​​agent system can place a virtual object next to the TV or sofa.

[0053] For example, virtual objects can be visualized in a specific location, such as a specific area. This location can be a space set by the user (e.g., next to the bed, next to the sofa, next to the TV) or the last visualized location, independent of the user's line of sight.

[0054] Meanwhile, in the 110-step operation, the AI ​​agent system can set the field of view (FOV) of the virtual object. The field of view of the virtual object can be used to determine whether to interact with the user.

[0055] The AI ​​agent system then outputs a virtual object to a set location (120). At this time, the AI ​​agent system can display the virtual object's field of view or eyes to indicate to the user where the virtual object is looking. This allows the user to check whether the virtual object is ready and to interact with it.

[0056] FIG. 3 is a diagram illustrating an example of using a virtual object in an artificial intelligence agent system according to various embodiments.

[0057] Referring to FIG. 3, an artificial intelligence agent system (320) in the form of an extended reality (XR) device can output a virtual object (330) located at a certain distance to a user (310). When the user (320) looks at the virtual object (330) without a wake word and starts speaking, the artificial intelligence agent system (320) can wait to receive the speech when the user (320) looks at the virtual object (330).

[0058] FIG. 4a is a diagram illustrating an example of displaying the status of a virtual object in an artificial intelligence agent system according to various embodiments.

[0059] Referring to FIG. 4a, the artificial intelligence agent system can indicate where a virtual object is facing by showing its eyes to the virtual object.

[0060] For example, a virtual object can have the states of user gaze (410), gaze elsewhere (420), eyes closed (430), doing something else (440), and turning away (450).

[0061] If the artificial intelligence agent system can immediately receive the user's speech, it can display the status of the virtual object as user gaze (410), and if not, it can display the status as gaze elsewhere (420), eyes closed (430), doing something else (440), and turning away (450).

[0062] Meanwhile, when outputting a virtual object in the 120 motion, the AI ​​agent system can output the virtual object in the form of a preset character and output the virtual object to perform the preset character's behavioral pattern. For example, if the virtual object is set to a puppy, the puppy-shaped virtual object can follow the user's movements and move to a preset location depending on where the user stays for a certain period of time and sit or lie down. As another example, if the virtual object is set to a parrot, the parrot-shaped virtual object can fly around the user as the user moves and move to a preset location depending on where the user stays for a certain period of time and sit in a high place (e.g., on top of a TV, a clock, or in the living room). The form of the virtual object is not limited to animals, and can be set to characters or heroes from movies or animations, and can be set to imitate each character's unique movements.

[0063] FIG. 4b is a diagram illustrating an example of an area in which interaction with a virtual object can be performed in an artificial intelligence agent system according to various embodiments.

[0064] Referring to FIG. 4b, the artificial intelligence agent system may display a distance (470) at which interaction with a virtual object (330) is possible, and may also display field of view information corresponding to the gaze direction of the virtual object (330).

[0065] At this time, if the virtual object (330) is set to be activated only within the field of view of the virtual object (330) while being located at a certain distance (470) from the virtual object (330), this activation area (480) can be displayed in a different color or shade.

[0066] Referring again to Figure 1, the artificial intelligence agent system obtains the user's gaze information (130).

[0067] The artificial intelligence agent system checks whether the activation conditions of the virtual object are met (140).

[0068] In the 140th movement, the artificial intelligence agent system can consider the user's gaze information and the location of the virtual object and determine that the activation condition of the virtual object is met (corresponds) when the user is looking at the virtual object.

[0069] As another example, the artificial intelligence agent system may consider the user's gaze information, the location of the virtual object, and the field of view of the virtual object, as in the example of Figure 5 below, and determine that the activation condition of the virtual object is met (corresponds) when the user and the virtual object are looking at each other (facing each other).

[0070] FIG. 5 is a diagram illustrating an example of a case in which a wake word can be omitted in an artificial intelligence agent system according to various embodiments.

[0071] Referring to FIG. 5, the artificial intelligence agent system may consider the user's gaze information, the location of the virtual object, and the field of view of the virtual object, and determine that the activation condition of the virtual object is met (corresponds) when the field of view of the user (310) and the virtual object (330) overlap and look at each other (face each other).

[0072] Referring back to Figure 1, if the activation condition of the virtual object is met as a result of the verification of operation 140, the AI ​​agent system processes the user's utterance without a wake word (150). In other words, the AI ​​agent system can determine the start point of detecting the utterance using the activation condition of the virtual object instead of a wake word.

[0073]

[0074] FIG. 2 is a flowchart illustrating an exemplary process of operating an artificial intelligence agent by considering the user's location and the user's gaze in an artificial intelligence agent system according to various embodiments.

[0075] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0076] According to an embodiment, it can be understood that operations 210 to 290 are performed in a processor (e.g., processor (910) of FIG. 9) of an artificial intelligence agent system (e.g., artificial intelligence agent system (900) of FIG. 9).

[0077] Referring to Figure 2, the artificial intelligence agent system confirms the user's location (210).

[0078] The AI ​​agent system can collect images corresponding to the user's field of view through a camera, analyze information about the area where the user is located using the images (220), and set the location of a virtual object by considering the user's location and the information about the analyzed area (230). For example, if the AI ​​agent system analyzes the image of the user's field of view and determines that the area where the user is located is the kitchen, the AI ​​agent system can place a virtual object on a kitchen counter at a certain distance from the user. As another example, if the area where the user is located is the living room and the user is sitting on a sofa, the AI ​​agent system can place a virtual object next to a TV or next to the sofa at a certain distance from the sofa.

[0079] FIG. 7 is a diagram illustrating an example of setting the location of a virtual object by analyzing the area where a user is located in an artificial intelligence agent system according to various embodiments.

[0080] Referring to FIG. 7, the artificial intelligence agent system analyzes the user's field of view image to identify a gas range, kitchen utensils, a refrigerator, a sink, and a faucet, and if it detects that the area where the user is located is a kitchen, it can place a virtual object (330) on the kitchen counter.

[0081] Action 220 is an optional action that can be omitted. If action 220 is omitted, the artificial intelligence agent system can check the user's location in action 210 and, referring to FIG. 2, set the virtual object to be located within a certain distance from the user's location in action 230.

[0082] Meanwhile, in the 210-step operation, the AI ​​agent system can set the field of view (FOV) of the virtual object. The field of view of the virtual object can be used to determine whether to interact with the user.

[0083] Referring back to Figure 2, the AI ​​agent system outputs a virtual object to a set location (240). At this time, the AI ​​agent system can display the virtual object's field of view or eyes to indicate to the user where the virtual object is looking. This allows the user to determine whether the virtual object is ready and to interact with it.

[0084] When outputting a virtual object in the 240 motion, the artificial intelligence agent system can output the virtual object in the form of a preset character and output the virtual object to perform the behavior pattern of the preset character.

[0085] The artificial intelligence agent system obtains the user's gaze information (250).

[0086] The artificial intelligence agent system checks whether the activation conditions of the virtual object are met (260).

[0087] In the 260 motion, the artificial intelligence agent system can determine that the activation condition of the virtual object is met (corresponds to) when the user is within a preset distance of the virtual object and the user is looking at the virtual object by considering the user's location, the user's gaze information, and the location of the virtual object.

[0088] As another example, the AI ​​agent system may consider the user's location, the user's gaze information, the location of the virtual object, and the field of view of the virtual object to determine that the user is within a preset distance of the virtual object and that the user and the virtual object are looking at each other (facing each other), and thus the activation condition of the virtual object is met (meeting).

[0089] FIG. 6 is a diagram illustrating an example of a case in which a wake word can be omitted in an artificial intelligence agent system according to various embodiments.

[0090] Referring to FIG. 6, the artificial intelligence agent system considers the user's location, the user's gaze information, the location of the virtual object, and the field of view of the virtual object to determine that the radius of the user (310) and the radius of the virtual object (330) are centered and exist within a preset distance, and if the gazes of the user (310) and the virtual object (330) overlap and look at each other (face each other), it can determine that the activation condition of the virtual object is met (conforms).

[0091] Referring again to Figure 2, if the activation condition of the virtual object is met as a result of the verification of operation 250, the AI ​​agent system processes the user's utterance without a wake word (290). In other words, the AI ​​agent system can determine the start point of detecting the utterance using the activation condition of the virtual object instead of a wake word.

[0092] If the activation condition of the virtual object is not satisfied as a result of the confirmation of the 260 operation, the artificial intelligence agent system can check whether a wake word is received (270).

[0093] When the wake word is received as a result of the confirmation of action 270, the artificial intelligence agent system adjusts the virtual object to satisfy the activation condition (280) and moves on to action 290.

[0094] For example, in the 280 motion, if the virtual object is located far from the user or is not looking at the user, the artificial intelligence agent system can move the virtual object to a certain distance from the user's location and adjust the virtual object to look at the user.

[0095] If the wake word is not received as a result of the confirmation of action 270, the artificial intelligence agent system can return to action 210 and repeat the series of processes.

[0096] Actions 270 and 280 described in Fig. 2 are optional actions and can be omitted.

[0097]

[0098] Meanwhile, the artificial intelligence agent system can also be used in a multi-user environment.

[0099] FIG. 8 is a diagram illustrating an example of a case where multiple users are targeted in an artificial intelligence agent system according to various embodiments.

[0100] Referring to FIG. 8, the artificial intelligence agent system can notify another user, the first user (310), that the virtual object (330) is busy by having the virtual object (330) look at the second user (810) while the virtual object (330) receives a speech from the second user (810) and performs an interaction to process the speech.

[0101] The first user (310) can notify the AI ​​agent system that the next user is waiting by waiting for the virtual object (330) to not look at the second user (810) or to look at the first user (310) or by uttering a wait word in order to use the AI ​​agent.

[0102] When the artificial intelligence agent system receives a wake word from the first user (310), it can terminate the interaction with the second user (810) and notify the first user (310) that it is ready to process the utterance by having the virtual object (330) look at the first user (310).

[0103]

[0104] FIG. 9 is a block diagram illustrating an exemplary configuration of an artificial intelligence agent system according to various embodiments.

[0105] Referring to FIG. 9, the artificial intelligence agent system (900) may be configured to include a processor (e.g., including a processing circuit) (910), a communication unit (e.g., including a communication circuit) (920), a display unit (e.g., including a display) (930), a memory (940), and a sensor unit (e.g., including a sensor) (950).

[0106] The communication unit (920) may include various communication circuits, and may communicate with an intelligent server that processes spoken voice and responds with the processing result as a communication interface device including a receiver and a transmitter.

[0107] The display unit (930) may include a display and may display status information (or indicators), limited numbers and characters, moving pictures, and still pictures generated during the operation of the artificial intelligence agent system (900). In addition, the display unit (930) may display virtual objects under the control of the virtual object visualization unit (912).

[0108] The memory (940) stores an operating system, application programs, and storage data for controlling the overall operation of the artificial intelligence agent system (900). Furthermore, the memory (940) can store configuration information for virtual objects. The configuration information for the virtual objects may include information on the location, shape, and behavioral patterns of the virtual objects.

[0109] The sensor unit (950) may include a first sensor (951), a second sensor (952), and a third sensor (953).

[0110] The first sensor (951) can detect the user's eyes to determine the user's gaze direction. The first sensor (951) may be an infrared camera that photographs the user's eyes.

[0111] The second sensor (952) can determine the user's location. The second sensor (952) may be a GPS sensor or a device that determines the user's location using a triangulation technique to determine the distance between Wi-Fi or base stations.

[0112] The third sensor (953) can collect images in the direction the user is looking. The third sensor (953) may be a camera.

[0113] The processor (910) may include a virtual object setting unit (911), a virtual object visualization unit (912), a user sensing unit (913), a virtual object activation unit (914), a speech collection unit (915), and a speech processing unit (916).

[0114] The virtual object setting unit (911) can set the location of the virtual object.

[0115] The virtual object setting unit (911) can set a preset area of ​​the display unit (930) as the location of the virtual object.

[0116] The virtual object setting unit (911) can set the location of the virtual object by considering the user's location confirmed through the second sensor (952). For example, the artificial intelligence agent system can confirm the user's location and set the virtual object to be located within a certain distance from the user's location.

[0117] As another example, the virtual object setting unit (911) may collect an image corresponding to the user's field of view through the third sensor (953), analyze information about the area where the user is located using the image, and set the location of the virtual object by considering the information of the analyzed area. For example, if the AI ​​agent system analyzes the image of the user's field of view and determines that the area where the user is located is the kitchen, the AI ​​agent system may place the virtual object on the kitchen counter. As another example, if the area where the user is located is the living room, the AI ​​agent system may place the virtual object next to the TV or the sofa.

[0118] As another example, the virtual object setting unit (911) can collect an image corresponding to the user's field of view through the third sensor (953), analyze information about the area where the user is located using the image, and set the location of the virtual object by considering the user's location and information about the analyzed area.

[0119] The virtual object setting unit (911) can set the virtual object in the form of a preset character and set it to perform the behavioral pattern of the preset character. For example, if the virtual object is set as a puppy, the puppy-shaped virtual object can move around the user as the user moves and move to a preset location and sit or lie down depending on where the user stays for a certain period of time. As another example, if the virtual object is set as a parrot, the parrot-shaped virtual object can fly around the user as the user moves and move to a preset location depending on where the user stays for a certain period of time and sit in a high place (e.g., on top of a TV, on top of a clock, on top of the living room). The form of the virtual object is not limited to animals, and can be set as a character or hero from a movie or animation, and can be set to imitate the unique movements of each character.

[0120] The user sensing unit (913) can obtain the user's gaze information through the first sensor (951).

[0121] The user sensing unit (913) can confirm the user's location through the second sensor (952).

[0122] The user sensing unit (913) can analyze the area where the user is located using the image collected through the third sensor (953).

[0123] The virtual object visualization unit (912) can output a virtual object through the display unit (930).

[0124] The virtual object visualization unit (912) can set the location of the virtual object by considering the user's location and information of the analyzed area.

[0125] The virtual object visualization unit (912) sets the field of view (FOV) of the virtual object along with the location of the virtual object, and when outputting the virtual object, it can display and output the field of view or eyes of the virtual object.

[0126] The virtual object activation unit (914) can check whether the activation conditions of the virtual object are met by considering the user's gaze information.

[0127] The virtual object visualization unit (912) can determine that the activation condition of the virtual object is met when the user is looking at the virtual object by considering the user's gaze information and the location of the virtual object.

[0128] The virtual object visualization unit (912) can determine that the activation condition of the virtual object is met when the user and the virtual object are looking at each other by considering the user's gaze information, the location of the virtual object, and the field of view of the virtual object.

[0129] The virtual object visualization unit (912) can determine that the activation condition of the virtual object is met when the user is within a preset distance of the virtual object and the user is looking at the virtual object by considering the user's gaze information, the user's location, and the location of the virtual object.

[0130] The virtual object visualization unit (912) can determine that the activation condition of the virtual object is met if the user is within a preset distance of the virtual object and the user and the virtual object are looking at each other by considering the user's gaze information, the location of the virtual object, and the field of view of the virtual object.

[0131] If a wake word is input through the speech collection unit (915) when the virtual object activation unit (914) verifies that the virtual object does not meet the activation conditions, the virtual object visualization unit (912) can adjust the virtual object so that it meets the activation conditions. For example, if the virtual object is located far from the user or is not looking at the user, the virtual object visualization unit (912) can move the virtual object to a certain distance from the user's location and adjust the virtual object so that it looks at the user.

[0132] The speech collection unit (915) can collect the user's speech without a wake word if the virtual object activation condition is met as a result of the confirmation by the virtual object activation unit (914).

[0133] The speech processing unit (916) can process the user's speech collected by the speech collection unit (915). For example, the speech processing unit (916) can convert the speech into text data and output a processing result based on the text data as a response.

[0134] The speech processing unit (916) may be configured to be included in the artificial intelligence agent system (900), but may also be implemented as an externally located intelligent device, and may respond with the processing result through communication via the communication unit (920).

[0135]

[0136] The artificial intelligence agent system (900) of FIG. 9 may be configured in the form of an electronic device (1001) in a network environment as in FIG. 10 below, or may be configured in the form of wearable augmented reality glasses (1100) as in FIG. 11.

[0137] FIG. 10 is a block diagram of an exemplary electronic device (1001) within a network environment (1000) according to various embodiments.

[0138] Referring to FIG. 10, in a network environment (1000), an electronic device (1001) may communicate with an electronic device (1002) via a first network (1098) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1004) or a server (1008) via a second network (1099) (e.g., a long-range wireless communication network). According to an embodiment, the electronic device (1001) may communicate with the electronic device (1004) via the server (1008). According to an embodiment, the electronic device (1001) may include a processor (1020), a memory (1030), an input module (1050), an audio output module (1055), a display module (1060), an audio module (1070), a sensor module (1076), an interface (1077), a connection terminal (1078), a haptic module (1079), a power management module (1088), a battery (1089), a communication module (1090), a subscriber identification module (1096), or an antenna module (1097). In various embodiments, the electronic device (1001) may omit at least one of these components (e.g., the connection terminal (1078)), or may have one or more other components added. In various embodiments, some of these components (e.g., the sensor module (1076), the camera module (1080), or the antenna module (1097)) may be integrated into one component (e.g., the display module (1060)).

[0139] The processor (1020) may include various processing circuits and multiple processors. For example, as used in this disclosure, including in the claims, the term "processor" may include various processing circuits including at least one processor, wherein one or more of the at least one processor may be configured to perform various functions described herein, individually and / or collectively, in a distributed manner. When "processor," "at least one processor," and "one or more processors" are described herein as being configured to perform multiple functions, these terms include, but are not limited to, situations where one processor performs some of the recited functions and another processor performs other of the recited functions, and situations where a single processor may perform all of the recited functions. Furthermore, the at least one processor may include a combination of processors that perform various recited / disclosed functions, for example, in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions. The processor (1020) may, for example, execute software (e.g., a program (1040)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1001) connected to the processor (1020) and perform various data processing or operations. According to an embodiment, as at least a part of the data processing or operations, the processor (1020) may store commands or data received from other components (e.g., a sensor module (1076) or a communication module (1090)) in a volatile memory (1032), process the commands or data stored in the volatile memory (1032), and store result data in a non-volatile memory (1034).According to an embodiment, the processor (1020) may include a main processor (1021) (e.g., a central processing unit or an application processor) or an auxiliary processor (1023) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1021). For example, when the electronic device (1001) includes the main processor (1021) and the auxiliary processor (1023), the auxiliary processor (1023) may be configured to use less power than the main processor (1021) or to be specialized for a given function. The auxiliary processor (1023) may be implemented separately from the main processor (1021) or as a part thereof.

[0140] The auxiliary processor (1023) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1060), the sensor module (1076), or the communication module (1090)) of the electronic device (1001), for example, on behalf of the main processor (1021) while the main processor (1021) is in an inactive (e.g., sleep) state, or together with the main processor (1021) while the main processor (1021) is in an active (e.g., application execution) state. In an embodiment, the auxiliary processor (1023) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1080) or a communication module (1090)). In an embodiment, the auxiliary processor (1023) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1001) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1008)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0141] Meanwhile, the processor (1020) may correspond to the processor (910) of FIG. 9.

[0142] The memory (1030) can store various data used by at least one component (e.g., the processor (1020) or the sensor module (1076)) of the electronic device (1001). The data can include, for example, software (e.g., the program (1040)) and input data or output data for commands related thereto. The memory (1030) can include volatile memory (1032) or non-volatile memory (1034).

[0143] The program (1040) may be stored as software in memory (1030) and may include, for example, an operating system (1042), middleware (1044), or an application (1046).

[0144] The input module (1050) can receive commands or data to be used in a component of the electronic device (1001) (e.g., a processor (1020)) from an external source (e.g., a user) of the electronic device (1001). The input module (1050) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0145] The audio output module (1055) can output audio signals to the outside of the electronic device (1001). The audio output module (1055) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0146] The display module (1060) can visually provide information to an external party (e.g., a user) of the electronic device (1001). The display module (1060) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In an embodiment, the display module (1060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0147] The audio module (1070) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to an embodiment, the audio module (1070) can acquire sound through the input module (1050), output sound through the sound output module (1055), or an external electronic device (e.g., electronic device (1002)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1001).

[0148] The sensor module (1076) can detect the operating status (e.g., power or temperature) of the electronic device (1001) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to an embodiment, the sensor module (1076) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, a Hall sensor, or an illuminance sensor.

[0149] Meanwhile, the sensor module (1076) may include a sensor unit (950) including a first sensor (951), a second sensor (952), and a third sensor (953) of FIG. 9.

[0150] Additionally, the sensor module (1076) may further include a camera module capable of capturing still images and moving images. In this case, the camera module may include one or more lenses, image sensors, image signal processors, or flashes.

[0151] The interface (1077) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1001) with an external electronic device (e.g., the electronic device (1002)). In an embodiment, the interface (1077) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0152] For example, the electronic device (1001) can transmit a video signal to an external electronic device through a connection terminal (1078). The electronic device (1001) can transmit a video signal for the external electronic device to output a video to a display module (1060) of the external electronic device.

[0153] The connection terminal (1078) may be for outputting a video signal or for outputting an audio signal. For example, the connection terminal (1078) may be for outputting a video signal and an audio signal simultaneously. For example, the electronic device (1001) may output a video signal and an audio signal to an external electronic device through an interface such as HDMI, DP, or Thunderbolt from the connection terminal (1078) that outputs a video signal and an audio signal simultaneously.

[0154] The connection terminal (1078) may include a connector through which the electronic device (1001) may be physically connected to an external electronic device (e.g., the electronic device (1002)). According to an embodiment, the connection terminal (1078) may include, for example, an HDMI connector, a DP connector, a Thunderbolt connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0155] The haptic module (1079) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to an embodiment, the haptic module (1079) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0156] The power management module (1088) can manage power supplied to the electronic device (1001). According to an embodiment, the power management module (1088) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0157] A battery (1089) may power at least one component of the electronic device (1001). According to an embodiment, the battery (1089) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0158] The communication module (1090) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1001) and an external electronic device (e.g., electronic device (1002), electronic device (1004), or server (1008)), and the performance of communication through the established communication channel. The communication module (1090) may operate independently from the processor (1020) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module (1090) may include a wireless communication module (1092) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1094) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module can communicate with an external electronic device (1004) via a first network (1098) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1099) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1092) can verify or authenticate the electronic device (1001) within a communication network such as the first network (1098) or the second network (1099) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1096).

[0159] The wireless communication module (1092) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1092) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1092) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1092) may support various requirements specified in the electronic device (1001), an external electronic device (e.g., the electronic device (1004)), or a network system (e.g., the second network (1099)). According to an embodiment, the wireless communication module (1092) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0160] The antenna module (1097) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In an embodiment, the antenna module (1097) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In an embodiment, the antenna module (1097) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1098) or the second network (1099), may be selected from the plurality of antennas by, for example, the communication module (1090). A signal or power may be transmitted or received between the communication module (1090) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1097).

[0161] According to various embodiments, the antenna module (1097) may form a mmWave antenna module. According to embodiments, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0162] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0163] According to an embodiment, a command or data may be transmitted or received between the electronic device (1001) and an external electronic device (1004) via a server (1008) connected to a second network (1099). Each of the external electronic devices (1002 or 1004) may be the same or a different type of device as the electronic device (1001). According to an embodiment, all or part of the operations executed in the electronic device (1001) may be executed in one or more of the external electronic devices (1002, 1004, or 1008). For example, when the electronic device (1001) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1001) may, instead of or in addition to executing the function or service on its own, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1001). The electronic device (1001) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1001) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (1004) may include an Internet of Things (IoT) device. The server (1008) may be an intelligent server utilizing machine learning and / or a neural network.According to an embodiment, an external electronic device (1004) or server (1008) may be included in the second network (1099). The electronic device (1001) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0164]

[0165] FIG. 11 is a diagram illustrating an exemplary configuration and structure of an electronic device implemented in the form of wearable augmented reality glasses according to various embodiments.

[0166] Referring to FIG. 11, an electronic device (1100) may be worn on a user's face and may provide the user with images related to an augmented reality service and / or a virtual reality service.

[0167] In an embodiment, the electronic device (1100) includes a first display (1105), a second display (1110), a first screen display unit (1115a), a second screen display unit (1115b), an input optical member (1120), a first transparent member (1125a), a second transparent member (1125b), a lighting unit (1130a, 1130b), a first PCB (1135a), a second PCB (1135b), a first hinge (1140a), a second hinge (1140b), a first camera (1145a, 1145b, 1145c, 1145d), a plurality of microphones (e.g., a first microphone (1150a), a second microphone (1150b), a third microphone (1150c)), a plurality of speakers (e.g., a first speaker (1155a), a second It may include a speaker (1155b)), a battery (1160), a second camera (1175a, 1175b), a third camera (1165), and a visor (1170a, 1170b).

[0168] In an embodiment, the displays (e.g., the first display (1105) and the second display (1110)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the display is made of one of the liquid crystal display, the digital mirror display, or the silicon liquid crystal display, the electronic device (1100) may include a light source that irradiates light to the screen output area of ​​the display. In another embodiment, when the display is capable of generating light on its own, for example, when it is made of one of the organic light emitting diode or the micro LED, the electronic device (1100) may provide a good quality virtual image to the user even without including a separate light source. In an embodiment, if the display is implemented with an organic light-emitting diode or micro LED, a light source is unnecessary, so the electronic device (1100) can be made lighter. In the following, a display capable of generating light on its own is referred to as a self-luminous display, and the description is based on the assumption of a self-luminous display.

[0169] Displays according to various embodiments of the present document (e.g., the first display (1105) and the second display (1110)) may be composed of at least one micro LED (micro light emitting diode). For example, the micro LED can express red (R), green (G), and blue (B) by self-luminescence, and has a small size (e.g., 100 μm or less), so that one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display includes the micro LED, it can provide a high resolution without a backlight unit (BLU).

[0170] Not limited thereto, one pixel may include R, G, and B, and one chip may be implemented with multiple pixels including R, G, and B.

[0171] In an embodiment, the display (e.g., the first display (1105) and the second display (1110)) may include a display area composed of pixels for displaying a virtual image and light-receiving pixels (e.g., photo sensor pixels) disposed between the pixels for receiving light reflected from the eye, converting it into electrical energy, and outputting it.

[0172] In an embodiment, the electronic device (1100) can detect a user's gaze direction (e.g., eye movement) through light-receiving pixels. For example, the electronic device (1100) can detect and track a gaze direction for the user's right eye and a gaze direction for the user's left eye through one or more light-receiving pixels constituting a first display (1105) and one or more light-receiving pixels constituting a second display (1110). The electronic device (1100) can determine the position of the center of the virtual image based on the gaze directions of the user's right and left eyes (e.g., the direction in which the pupils of the user's right and left eyes are gazing) detected through one or more light-receiving pixels.

[0173] In an embodiment, light emitted from displays (e.g., a first display (1105) and a second display (1110)) may pass through a lens (not shown) and a waveguide to reach a first screen display unit (1115a) formed on a first transparent member (1125a) positioned to face the user's right eye and a second screen display unit (1115b) formed on a second transparent member (1125b) positioned to face the user's left eye. For example, light emitted from displays (e.g., a first display (1105) and a second display (1110)) may pass through a waveguide to be reflected by a grating area formed on an input optical member (1120) and screen display units (1115a, 1115b) and transmitted to a user's eye. The first transparent member (1125a) and / or the second transparent member (1125b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent.

[0174] In an embodiment, a lens (not shown) may be positioned in front of a display (e.g., a first display (1105) and a second display (1110)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0175] In an embodiment, the screen display unit (1115a, 1115b) or the transparent member (e.g., the first transparent member (1125a), the second transparent member (1125b)) may include a lens including a waveguide, a reflective lens.

[0176] In an embodiment, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to an embodiment, light incident on one end of the waveguide may be propagated within the display waveguide by the nano-patterns and provided to the user. In an embodiment, a waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element, for example, a reflective mirror. In an embodiment, the waveguide may guide light emitted from the display (1105, 1110) to the user's eye by using at least one diffractive element or reflective element included in the waveguide.

[0177] According to various embodiments, the diffractive element may include an input optical member (1120) / output optical member (not shown). For example, the input optical member (1120) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display (e.g., a first display (1105) and a second display (1110)) (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (1125a), a second transparent member (1125b)) of a screen display unit (1115a, 1115b). The output grating region can act as an outlet to diffract (or reflect) light transmitted to the transparent member of the waveguide (e.g., the first transparent member (1125a), the second transparent member (1125b)) toward the user's eyes.

[0178] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0179] In an embodiment, light emitted from a display (1105, 1110) may be guided along an optical path to a waveguide via an input optical member (1120). Light traveling within the waveguide may be guided toward a user's eyes via an output optical member. The screen display (1115a, 1115b) may be determined based on the light emitted toward the user's eyes.

[0180] In an embodiment, the first camera (1145a, 1145b, 1145c, 1145d) may include a camera used for 6 degrees of freedom (6DoF), 6DoF head tracking, hand detection and tracking, gesture and / or spatial recognition. For example, the first camera (1145a, 1145b, 1145c, 1145d) may include a GS (global shutter) camera to detect head and hand movements and track movements.

[0181] For example, the first camera (1145a, 1145b, 1145c, 1145d) may be a stereo camera for head tracking and spatial recognition, and cameras of the same specifications and performance may be applied. The first camera (1145a, 1145b, 1145c, 1145d) may be a GS camera with excellent performance (e.g., image drag) for detecting and tracking fine movements such as rapid hand movements and fingers.

[0182] According to various embodiments, the first camera (1145a, 1145b, 1145c, 1145d) may be a rolling shutter (RS) camera. The first camera (1145a, 1145b, 1145c, 1145d) may perform SLAM functions through spatial recognition and depth shooting for 6 DoF. The first camera (1145a, 1145b, 1145c, 1145d) may perform a user gesture recognition function.

[0183] In an embodiment, the second camera (1175a, 1175b) may be used for detecting and tracking the pupil. The second camera (1175a, 1175b) may be referred to as an ET (eye tracking) camera. The second camera (1175a, 1175b) may track the user's gaze direction. The electronic device (1100) may consider the user's gaze direction and position the center of the virtual image projected on the screen display unit (1115a, 1115b) according to the direction in which the user's pupil is looking.

[0184] The second camera (1175a, 1175b) for tracking the gaze direction may be a GS camera that can detect the pupil and track rapid eye movements. The second cameras (1175a, 1175b) may be installed for the left and right eyes, respectively, and the second cameras (1175a, 1175b) for the left and right eyes may be cameras with identical performance and specifications.

[0185] In an embodiment, the third camera (1165) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (1165) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an OIS (optical image stabilizer). However, the third camera (1165) is not limited thereto, and may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0186] In an embodiment, at least one sensor (e.g., a gyro sensor, an acceleration sensor, a geomagnetic sensor, a touch sensor, an ambient light sensor, and / or a gesture sensor), and a first camera (1145a, 1145b, 1145c, 1145d) can perform at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or space recognition, and slam function through depth shooting.

[0187] In another embodiment, the first camera (1145a, 1145b, 1145c, 1145d) may be used separately as a camera for head tracking and a camera for hand tracking.

[0188] In an embodiment, the lighting units (1130a, 1130b) may have different uses depending on the attachment location. For example, the lighting units (1130a, 1130b) may be attached together with the first camera (1145a, 1145b, 1145c, 1145d) mounted around a hinge (e.g., a first hinge (1140a), a second hinge (1140b)) connecting a frame and a temple or around a bridge connecting the frames. When shooting with a GS camera, the lighting units (1130a, 1130b) may be used as a means of supplementing the surrounding brightness. For example, the lighting units (1130a, 1130b) may be used when it is difficult to detect a subject to be shot due to a dark environment or mixing or reflecting light from multiple light sources.

[0189] In an embodiment, a PCB (e.g., a first PCB (1135a), a second PCB (1135b)) may include a processor (not shown), a memory (not shown), and a communication module (not shown) that control components of the electronic device (1100).

[0190] A communication module (not shown) may support the establishment of a direct (e.g., wired) or wireless communication channel between the electronic device (1100) and an external electronic device, and the performance of communication through the established communication channel. The PCB may transmit electrical signals to components of the electronic device (1100).

[0191] A communication module (not shown) may include one or more communication processors that operate independently from the processor and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (not shown) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules (not shown) may communicate with an external electronic device via a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0192] The wireless communication module can support 5G networks and next-generation communication technologies beyond 4G networks, such as new radio access technology (NR). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module can support high-frequency bands (e.g., mmWave bands) to achieve high data rates, for example. The wireless communication module may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.

[0193] The electronic device (1100) may further include an antenna module (not shown). The antenna module may transmit or receive signals or power to or from an external location of the electronic device (1100) (e.g., an external electronic device). In one embodiment, the antenna module may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (1135a), a second PCB (1135b)). In one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0194] In an embodiment, a plurality of microphones (e.g., a first microphone (1150a), a second microphone (1150b), and a third microphone (1150c)) can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) by the electronic device (1100).

[0195] In an embodiment, a plurality of speakers (e.g., a first speaker (1155a), a second speaker (1155b)) can output audio data received from a communication module or stored in a memory.

[0196] In an embodiment, one or more batteries (1160) may be included and may supply power to components constituting the electronic device (1100).

[0197] In an embodiment, the visor (1170a, 1170b) can adjust the amount of external light transmitted to the user's eyes according to the transmittance. The visor (1170a, 1170b) can be positioned in front or behind the screen display unit (1115a, 1115b). The front of the screen display unit (1115a, 1115b) can mean the direction opposite to the user wearing the electronic device (1100), and the back can mean the direction toward the user wearing the electronic device (1100). The visor (1170a, 1170b) can protect the screen display unit (1115a, 1115b) and adjust the amount of external light transmitted.

[0198] For example, the visor (1170a, 1170b) may include an electrochromic element that changes color according to the applied power to control the transmittance. Electrochromism is a phenomenon in which the color changes due to an oxidation-reduction reaction caused by the applied power. The visor (1170a, 1170b) can control the transmittance of external light by utilizing the color change of the electrochromic element.

[0199] For example, the visor (1170a, 1170b) may include a control module and an electrochromic element. The control module may control the electrochromic element to adjust the transmittance of the electrochromic element.

[0200]

[0201] According to one embodiment, a method for operating an artificial intelligence agent by considering a gaze may include: setting a position of a virtual object; outputting the virtual object; acquiring gaze information of a user; determining whether an activation condition of the virtual object is satisfied by considering the gaze information of the user; and processing the user's speech without a wake word if the activation condition of the virtual object is satisfied as a result of the determination.

[0202] According to one embodiment, the operation of checking whether the activation condition of the virtual object is met by considering the gaze information of the user may be performed by checking whether the activation condition of the virtual object is met based on the user looking at the virtual object by considering the gaze information of the user and the location of the virtual object.

[0203] According to one embodiment, a method for operating an artificial intelligence agent by taking into account a gaze may further include an action of determining a location of the user, and the action of setting a location of the virtual object may include an action of setting a location of the virtual object by taking into account the location of the user.

[0204] According to one embodiment, a method for operating an artificial intelligence agent by considering a gaze may further include an operation of confirming the location of the user; and an operation of analyzing an area where the user is located by using an image input through a camera, and the operation of setting the location of the virtual object may include an operation of setting the location of the virtual object by considering the location of the user and information of the analyzed area.

[0205] According to one embodiment, a method for operating an artificial intelligence agent by considering a gaze further includes an operation of confirming the location of the user, and the operation of confirming whether the activation condition of the virtual object is satisfied by considering the gaze information of the user may be determined based on the user being present within a preset distance of the virtual object and the user looking at the virtual object by considering the gaze information of the user, the location of the user, and the location of the virtual object, and whether the activation condition of the virtual object is satisfied.

[0206] According to one embodiment, the operation of setting the position of the virtual object further includes the operation of setting the field of view (FOV) of the virtual object together with the position of the virtual object, and the operation of outputting the virtual object outputs the virtual object by displaying the field of view of the virtual object or the eyes of the virtual object based on the output of the virtual object, and the operation of checking whether the activation condition of the virtual object is met by considering the gaze information of the user can check whether the activation condition of the virtual object is met based on the user and the virtual object looking at each other by considering the gaze information of the user, the position of the virtual object, and the field of view of the virtual object.

[0207] According to one embodiment, a method for operating an artificial intelligence agent by considering a gaze further includes an operation of confirming the position of the user, and the operation of setting the position of the virtual object further includes an operation of setting a field of view (FOV) of the virtual object together with the position of the virtual object, and the operation of outputting the virtual object displays and outputs the field of view of the virtual object or the eyes of the virtual object based on the output of the virtual object, and the operation of confirming whether an activation condition of the virtual object is met by considering the gaze information of the user may be determined to be met based on the presence of the user within a preset distance of the virtual object and the user and the virtual object looking at each other by considering the gaze information of the user, the position of the virtual object, and the field of view of the virtual object.

[0208] According to one embodiment, a method for operating an artificial intelligence agent by considering gaze may further include an operation of adjusting the virtual object so that the virtual object satisfies an activation condition, based on the wake word input when the wake word is not input in a state where the virtual object does not satisfy an activation condition as a result of the verification.

[0209] According to one embodiment, the operation of outputting the virtual object may output the virtual object in the form of a preset character and output the virtual object so as to perform a behavior pattern of the preset character.

[0210] According to one embodiment, an artificial intelligence agent system considering gaze may include: a first sensor for sensing a user's eyes; a display unit including a display on which a virtual object is output; a virtual object setting unit including a circuit for setting a position of the virtual object; a virtual object visualization unit including a circuit for outputting the virtual object through the display unit; a user sensing unit including a circuit for obtaining gaze information of the user through the first sensor; a virtual object activation unit including a circuit for determining whether an activation condition of the virtual object is met by considering the gaze information of the user; a speech collection unit including a circuit for collecting the user's speech without a wake word when the virtual object activation unit determines that the activation condition of the virtual object is met; and a speech processing unit including a circuit for processing the user's speech collected by the speech collection unit.

[0211] According to one embodiment, the virtual object activation unit can determine that the activation condition of the virtual object is met based on the user looking at the virtual object by considering the user's gaze information and the location of the virtual object.

[0212] According to one embodiment, the artificial intelligence agent system considering gaze further includes a second sensor for confirming a location, the user sensing unit confirms the location of the user through the second sensor, and the virtual object setting unit can set the location of the virtual object by considering the location of the user.

[0213] According to one embodiment, an artificial intelligence agent system considering gaze further includes a second sensor for determining a location; and a third sensor for collecting an image of a direction in which the user is looking, wherein the user sensing unit determines the location of the user through the second sensor and analyzes an area in which the user is located using the image, and the virtual object visualization unit can set the location of the virtual object by considering the location of the user and information of the analyzed area.

[0214] According to one embodiment, the artificial intelligence agent system considering gaze further includes a second sensor for confirming a location, and the user sensing unit confirms the location of the user through the second sensor, and the virtual object activation unit can confirm that the user is within a preset distance of the virtual object and that the user is looking at the virtual object based on consideration of the gaze information of the user, the location of the user, and the location of the virtual object, and that the activation condition of the virtual object is satisfied.

[0215] According to one embodiment, the virtual object visualization unit sets a field of view (FOV) of the virtual object together with the location of the virtual object, and based on the output of the virtual object, displays the field of view of the virtual object or the eyes of the virtual object to output the virtual object, and the virtual object activation unit can determine that the activation condition of the virtual object is met based on the user and the virtual object looking at each other by considering the gaze information of the user, the location of the virtual object, and the field of view of the virtual object.

[0216] According to one embodiment, an artificial intelligence agent system considering a gaze further includes a second sensor for confirming a location, wherein the user sensing unit confirms the location of the user through the second sensor, the virtual object visualization unit sets a field of view (FOV) of the virtual object together with the location of the virtual object, and based on the output of the virtual object, displays the field of view of the virtual object or the eyes of the virtual object to output the virtual object, and the virtual object activation unit can determine that the activation condition of the virtual object is met based on the user's gaze information, the location of the virtual object, and the field of view of the virtual object, and the user and the virtual object are looking at each other.

[0217] According to one embodiment, the virtual object visualization unit can adjust the virtual object to meet the activation condition according to the wake word input through the utterance collection unit when the virtual object activation unit confirms that the virtual object does not meet the activation condition.

[0218] According to one embodiment, the virtual object setting unit can set the virtual object in the form of a preset character and set the virtual object to perform a behavior pattern of the preset character.

[0219] According to one embodiment, an artificial intelligence agent system considering gaze may include a processor, which may individually or collectively include a first sensor for sensing a user's eyes; a display unit including a display on which a virtual object is output; and a processing circuit configured to set a position of the virtual object, output the virtual object through the display unit, obtain gaze information of the user through the first sensor, determine whether an activation condition of the virtual object is satisfied by considering the gaze information of the user, and process the user's utterance without a wake word if the activation condition of the virtual object is satisfied as a result of the determination.

[0220]

[0221] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, home appliances, and the like. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0222] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it should be understood that the component may be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0223] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, or any combination thereof, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0224] Various embodiments of the present document may be implemented as software (e.g., a program (1040)) including one or more instructions stored in a storage medium (e.g., an internal memory (1036) or an external memory (1038)) readable by a machine (e.g., an electronic device (1001)). For example, a processor (e.g., a processor (1020)) of the machine (e.g., an electronic device (1001)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium is a tangible device and may not contain signals (e.g., electromagnetic waves), but the term does not distinguish between cases where data is stored semi-permanently on the storage medium and cases where it is stored temporarily.

[0225] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0226] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0227] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0228] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0229] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0230] While the present disclosure has been described and illustrated with reference to various embodiments, it is to be understood that these various embodiments are illustrative and not limiting. Furthermore, those skilled in the art will appreciate that various modifications in construction and detail are possible without departing from the full scope and spirit of the present disclosure, including the scope of the following claims and their equivalents. Furthermore, it will be appreciated that any of the embodiments described herein may be utilized in conjunction with any other embodiment described herein.

Claims

1. In a method of operating an artificial intelligence agent by considering the gaze, An action that sets the position of a virtual object; An action of outputting the above virtual object; An action to obtain information about the user's gaze; An operation for checking whether the activation condition of the virtual object is met by considering the gaze information of the user; and An action to process the user's speech without a wake word, based on the verification result meeting the activation condition of the virtual object. A method including:

2. In paragraph 1, The operation of checking whether the activation condition of the virtual object is met by considering the above user's gaze information is as follows. Considering the user's gaze information and the location of the virtual object, it is determined that the user is looking at the virtual object and that the activation condition of the virtual object is met. method.

3. In any one of paragraphs 1 and 2, Action to check the location of the above user Including more, The action of setting the position of the above virtual object is: An action to set the location of the virtual object by considering the location of the user. A method including:

4. In any one of paragraphs 1 to 3, Action to determine the location of the above user; and An action that analyzes the area where the user is located using an image input through a camera. Including more, The action of setting the position of the above virtual object is: An action to set the location of the virtual object by considering the location of the user and information of the analyzed area. A method including:

5. In any one of paragraphs 1 to 4, Action to check the location of the above user Including more, The operation of checking whether the activation condition of the virtual object is met by considering the above user's gaze information is as follows. Considering the user's gaze information, the user's location, and the location of the virtual object, it is determined that the user exists within a preset distance of the virtual object and that the user is looking at the virtual object, and that the activation condition of the virtual object is met. method.

6. In any one of paragraphs 1 to 5, The action of setting the position of the above virtual object is: An action to set the field of view (FOV) of the virtual object together with the location of the virtual object. Including more, The action of outputting the above virtual object is: Based on the output of the virtual object, the virtual object is output by displaying the field of view of the virtual object or the eyes of the virtual object, The operation of checking whether the activation condition of the virtual object is met by considering the above user's gaze information is as follows. Based on the user's gaze information, the location of the virtual object, and the field of view of the virtual object, it is determined that the user and the virtual object are looking at each other and that the activation condition of the virtual object is met. method.

7. In any one of paragraphs 1 to 6, Action to check the location of the above user Including more, The action of setting the position of the above virtual object is: An action to set the field of view (FOV) of the virtual object together with the location of the virtual object. Including more, The action of outputting the above virtual object is: Based on the output of the virtual object, the virtual object is output by displaying the field of view of the virtual object or the eyes of the virtual object, The operation of checking whether the activation condition of the virtual object is met by considering the above user's gaze information is as follows. Considering the user's gaze information, the location of the virtual object, and the field of view of the virtual object, it is confirmed that the user exists within a preset distance of the virtual object, and that the user and the virtual object are looking at each other, and that the activation condition of the virtual object is met. method.

8. In any one of paragraphs 1 to 7, An operation of adjusting the virtual object so that it satisfies the activation condition according to the wake word input in a state where the activation condition of the virtual object is not satisfied as a result of the verification. How to include more.

9. In any one of paragraphs 1 to 8, The action of outputting the above virtual object is: Outputting the above virtual object in the form of a preset character, and outputting the virtual object so that it performs the behavior pattern of the preset character. method.

10. In an artificial intelligence agent system that takes into account gaze, A first sensor that senses the user's eyes; A display section including a display on which a virtual object is output; A virtual object setting unit including a circuit for setting the position of the virtual object; A virtual object visualization unit including a circuit that outputs the virtual object through the display unit; A user sensing unit including a circuit that obtains the user's gaze information through the first sensor; A virtual object activation unit including a circuit that checks whether the activation condition of the virtual object is met by considering the gaze information of the user; A speech collection unit including a circuit that collects the user's speech without a wake word, according to the result of verification by the virtual object activation unit that the activation condition of the virtual object is met; and A speech processing unit including a circuit for processing the user's speech collected from the speech collection unit An artificial intelligence agent system comprising:

11. In paragraph 10, The above virtual object activation part, Considering the user's gaze information and the location of the virtual object, it is determined that the user is looking at the virtual object and that the activation condition of the virtual object is met. Artificial intelligence agent system.

12. In any one of paragraphs 10 to 11, A second sensor to check the location Including more, The above user sensing unit, The user's location is confirmed through the second sensor, The above virtual object setting section is: Setting the location of the virtual object by considering the location of the user Artificial intelligence agent system.

13. In any one of paragraphs 10 to 12, a second sensor for determining location; and A third sensor that collects images in the direction the user is looking Including more, The above user sensing unit, The location of the user is confirmed through the second sensor, and the area where the user is located is analyzed using the image. The above virtual object visualization section, Setting the location of the virtual object by considering the location of the user and the information of the analyzed area. Artificial intelligence agent system.

14. In any one of paragraphs 10 to 13, A second sensor to check the location Including more, The above user sensing unit, The user's location is confirmed through the second sensor, The above virtual object activation part, Considering the user's gaze information, the user's location, and the location of the virtual object, it is determined that the user exists within a preset distance of the virtual object and that the user is looking at the virtual object, and that the activation condition of the virtual object is met. Artificial intelligence agent system.

15. In an artificial intelligence agent system that takes into account gaze, A first sensor that senses the user's eyes; A display section including a display on which a virtual object is output; and A processor including a processing circuit, individually or collectively, configured to set the position of the virtual object, output the virtual object through the display unit, obtain the user's gaze information through the first sensor, check whether the user's gaze information meets an activation condition of the virtual object by considering the user's gaze information, and process the user's utterance without a wake word if the activation condition of the virtual object is met as a result of the check. An artificial intelligence agent system comprising:

Images