Apparatus and method for operating artificial intelligence agent by considering visual information

Abstract

The present disclosure relates to an apparatus and a method for operating an artificial intelligence agent by considering visual information. The method for operating an artificial intelligence agent, according to one embodiment, may comprise: receiving an image from a camera; selecting at least one speech processing model from among a plurality of speech processing models on the basis of the image; loading the selected speech processing model; and processing the speech of a user by using the selected speech processing model.

Description

Device and method for operating an artificial intelligence agent by considering visual information

[0001] The following embodiments relate to a technology for providing an artificial intelligence agent in an extended reality (XR) device.

[0002] In the extended reality (XR) environment, the complexity of speech increases due to the addition of spatial information and environmental information, which is different from the method of giving commands to artificial intelligence agents such as existing voice assistants. When training models to be applied to the automatic speech recognition module (ASR module) and natural language understanding module (NLU module) to process all speech, there is a problem of increased cost.

[0003] According to one embodiment, a method of operating an artificial intelligence agent may include: receiving an image from a camera; selecting at least one voice processing model from among a plurality of voice processing models based on the image; loading the selected voice processing model; and processing a user's speech using the selected voice processing model.

[0004] According to one embodiment, an artificial intelligence agent device may include an image collection unit that receives an image from a camera; a dynamic model selection unit that selects at least one voice processing model from among a plurality of voice processing models based on the image and loads the selected voice processing model; and an artificial intelligence agent processing unit that processes a user's speech using the selected voice processing model.

[0005] FIG. 1 is a flowchart illustrating a process of providing an artificial intelligence agent by considering visual information in an artificial intelligence agent device according to one embodiment.

[0006] FIG. 2 is a flowchart illustrating an example of selecting a voice processing model based on visual information in an artificial intelligence agent device according to one embodiment.

[0007] FIG. 3 is a flowchart illustrating an example of selecting a voice processing model based on visual information and the user's location in an artificial intelligence agent device according to one embodiment.

[0008] FIG. 4 is a flowchart illustrating an example of selecting a voice processing model based on visual information, previous visual information, and the user's location in an artificial intelligence agent device according to one embodiment.

[0009] FIG. 5 is a diagram illustrating an example of providing an artificial intelligence agent by considering visual information in an artificial intelligence agent device according to one embodiment.

[0010] FIG. 6 is a diagram illustrating an example of displaying a domain selected based on visual information in an artificial intelligence agent device according to one embodiment.

[0011] FIG. 7 is a diagram illustrating an example of selecting a desired domain from among multiple domains through a gesture in an artificial intelligence agent device according to one embodiment.

[0012] FIG. 8 is a diagram illustrating an example of a plurality of domains being searched in an artificial intelligence agent device according to one embodiment.

[0013] FIG. 9 is a diagram illustrating an example of selecting a domain by considering visual information that changes over time in an artificial intelligence agent device according to one embodiment.

[0014] FIG. 10 is a diagram illustrating another example of selecting a domain by considering visual information that changes over time in an artificial intelligence agent device according to one embodiment.

[0015] FIG. 11 is a diagram illustrating the configuration of an artificial intelligence agent device that takes visual information into account according to one embodiment.

[0016] FIG. 12 is a block diagram of an electronic device within a network environment according to one embodiment.

[0017] FIG. 13 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0018] FIG. 14 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0019] FIG. 15 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to one embodiment.

[0020] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0021] The terms used in the examples are for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood to not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0022] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiments pertain. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0023] In addition, when describing with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted. When describing embodiments, if a detailed description of a related known technology is judged to unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

[0024] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the embodiments. These terms are only intended to distinguish the components from other components, and the nature, order, or sequence of the components are not limited by the terms. When a component is described as being "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but another component may also be "connected," "coupled," or "connected" between each component.

[0025] Components included in one embodiment and components with common functions will be described using the same names in other embodiments. Unless otherwise stated, the descriptions given in one embodiment may also apply to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

[0026] Hereinafter, a device and method for operating an artificial intelligence agent by considering visual information according to an embodiment of the present invention will be described in detail with reference to the attached FIGS. 1 to 15.

[0027] FIG. 1 is a flowchart illustrating a process of providing an artificial intelligence agent by considering visual information in an artificial intelligence agent device according to one embodiment.

[0028] Referring to Figure 1, the AI ​​agent device receives an image from a camera (110). In step 110, the image is an image of the direction the user is looking, and when the AI ​​agent device receives the image, it can also check gaze information indicating the object the user is looking at in the image.

[0029] Furthermore, the AI ​​agent device can select at least one voice processing model from among multiple voice processing models based on the image (120). Step 120 can be repeated at preset time intervals. Furthermore, step 120 can also be performed if the user's location remains unchanged for a preset period of time. More specific examples of step 120 will be described later with reference to FIGS. 2 through 4.

[0030] In addition, the artificial intelligence agent device can load the selected speech processing model (130). In step 130, the artificial intelligence agent device can load the selected speech processing model into at least one of an automatic speech recognition module (ASR module), a natural language understanding module (NLU module), a natural language generator module (NLG module), and a text-to-speech module (TTS module).

[0031] In step 130, the AI ​​agent device may determine the loading point as the point at which the object is recognized, the point at which the object is a certain distance away, the point at which a domain to be loaded is selected based on sequentially received information, or the point at which a user request is detected. At this time, the domain may include at least one speech processing model.

[0032] For example, when a user is walking down the street wearing an extended reality (XR) device and objects such as a coffee shop or restaurant come into view, the AI ​​agent device can output a guidance pop-up or audio message to the user asking, "Would you like to load the domain of Coffee Shop A?" If the user accepts the loading request, the AI ​​agent device can load the domain.

[0033] And, the artificial intelligence agent device can process the user's speech using the selected speech processing model (140).

[0034] For example, in step 140, if the artificial intelligence agent device receives an abbreviated utterance from the user, “Order me two Americanos,” while the domain of Coffee Shop A is loaded, it can analyze the utterance into an unabbreviated utterance, “Order me two Americanos at the OOO store of Coffee Shop A,” based on the domain information of Coffee Shop A and the location information of the user, and output and display the response result on the screen.

[0035] And, the artificial intelligence agent device can check whether the speech processing has failed (150).

[0036] If the utterance processing fails as a result of the 150-step verification, the artificial intelligence agent device can reselect a utterance-based utterance processing model from among multiple utterance processing models (160).

[0037] For example, if an artificial intelligence agent device analyzes an image received through a camera and loads the domain of Coffee A, and previously loaded domains such as "TV," "Music," "Weather," and "Calendar" are loaded, and a user inputs an utterance unrelated to the previously loaded domain, such as "Launch the OOO game," the utterance response may fail.

[0038] At this time, the AI ​​agent device determines the object name that can be recognized within the utterance "OOO game" and, if a domain related to the OOO game exists, can reselect and load the voice processing model included in the related domain. If a domain related to the OOO game does not exist, the AI ​​agent device can output feedback to the user indicating that it cannot respond.

[0039] And, the artificial intelligence agent device can process the user's speech using the reselected speech processing model (170).

[0040] FIG. 5 is a diagram illustrating an example of providing an artificial intelligence agent by considering visual information in an artificial intelligence agent device according to one embodiment.

[0041] Referring to FIG. 5, when an artificial intelligence agent device receives an image corresponding to the user's visual information received while the user is moving on the road, it analyzes the image and obtains extracted information such as "Coffee A" corresponding to a meaningful object name through a signboard (510) located on a building.

[0042] And, the artificial intelligence agent device can visually display (520) that the domain corresponding to “coffee A” has been selected.

[0043] And, the artificial intelligence agent device receives information from the user.

[0044] When a wake word, for example, "Bixby" is input, an agent for voice recognition is executed, and when an utterance (530) before or after the utterance for which the wake word is recognized, for example, "Bixby, order me an Americano Sezan" or "Order me an Americano Sezan, Bixby," is received, it is confirmed that the store of "Coffee A" is "the OOO store" based on the user's location, and the user's utterance is judged as "Order me an Americano Sezan at the OOO store of Coffee A", and the utterance can be processed.

[0045] Meanwhile, although not shown in FIG. 1, the artificial intelligence agent device can cancel the selection of the selected domain and delete the selected voice processing model loaded corresponding to the selected domain if the selected domain is not used for a preset period of time.

[0046] Additionally, if the input image does not contain information related to the selected domain for a preset period of time, the artificial intelligence agent device may cancel the selection of the selected domain and delete the selected voice processing model loaded in response to the selected domain.

[0047]

[0048] FIG. 2 is a flowchart illustrating an example of selecting a voice processing model based on visual information in an artificial intelligence agent device according to one embodiment.

[0049] Referring to Figure 2, the artificial intelligence agent device analyzes an image to extract multiple pieces of information (210). Here, the multiple pieces of information may represent meaningful object names.

[0050] At step 210, the AI ​​agent device can analyze the image to determine meaningful object names. For example, it can parse text information contained in the received image to gather meaningful object information. More specifically, the AI ​​agent device can obtain meaningful object information by parsing text information displayed on signs, menus, posters, etc. contained in images received through the camera.

[0051] Alternatively, the AI ​​agent device can analyze the image and determine the object name, such as a bus stop, TV, or air conditioner, based on a pre-trained object recognition model, and use the object name as object information.

[0052] Additionally, AI agent devices can utilize object detection and object recognition. Object detection is the process of finding instances of objects in an image, enabling the device to not only identify objects in the image but also determine their respective locations.

[0053] Additionally, AI agent devices can recognize various objects using deep learning models such as convolutional neural networks (CNNs) for object recognition using deep learning, or machine learning techniques such as support vector machines (SVMs). Representative examples include the Viola-Jones algorithm.

[0054] And, the artificial intelligence agent device can select at least one domain among multiple domains based on the extracted information (220).

[0055] And, the artificial intelligence agent device can select at least one voice processing model among multiple voice processing models based on the selected domain (230).

[0056] And, the artificial intelligence agent device can display visual information corresponding to the selected domain (240).

[0057] In step 240, if there are multiple selected domains, the artificial intelligence agent device can check the user's gaze, check the selected domain corresponding to the user's gaze among the multiple selected domains, and display the selected domain corresponding to the user's gaze.

[0058] Alternatively, when there are multiple selected domains, the artificial intelligence agent device can identify the user's gaze, identify a selected domain corresponding to the user's gaze among the multiple selected domains, and display visual information corresponding to each of the multiple selected domains, and display the selected domain corresponding to the user's gaze so as to be distinguished from other selected domains.

[0059] FIG. 6 is a diagram illustrating an example of displaying a domain selected based on visual information in an artificial intelligence agent device according to one embodiment.

[0060] Referring to FIG. 6, the artificial intelligence agent device analyzes the received image (600) to obtain information corresponding to meaningful object names, such as “Coffee A”, “Coffee B”, “Coffee C”, and “Coffee D”, and selects a domain corresponding to each of “Coffee A”, “Coffee B”, “Coffee C”, and “Coffee D”.

[0061] The artificial intelligence agent device can check the user's gaze (610), confirm that the user is looking at "Coffee B", and display (620) in the form of an icon that the domain corresponding to "Coffee B" has been finally selected.

[0062] The artificial intelligence agent device can also display (630) in the form of icons the domains for “Coffee A” and “Coffee C” located around the finally selected “Coffee B” as adjacent domains with a high probability of selection.

[0063] Additionally, the artificial intelligence agent device can display (640) the domain for “Coffee D” existing on the image (600) in the form of an icon, even though it is far from the user’s line of sight.

[0064] FIG. 7 is a diagram illustrating an example of selecting a desired domain from among multiple domains through a gesture in an artificial intelligence agent device according to one embodiment.

[0065] Referring to FIG. 7, the artificial intelligence agent device can obtain information on induction, cooking surfaces, etc. using information extracted from an image input in step 710, and select a domain related to cooking and a domain capable of controlling kitchen appliances based on this.

[0066] When the artificial intelligence agent device receives a user's gesture in step 720, it can display the selected domains based on an image to the user if there are multiple selected domains according to the gesture, and determine the domain to be finally applied among the domains selected by the user's touch.

[0067] FIG. 8 is a diagram illustrating an example of a plurality of domains being searched in an artificial intelligence agent device according to one embodiment.

[0068] Referring to FIG. 8, when an artificial intelligence agent device receives an image (820) in the direction a user (810) is looking, it can analyze the image (820) to extract information. At this time, the user (810) may be a user wearing an artificial intelligence agent in the form of an XR.

[0069] If the extracted information (830) includes information on a TV, speaker, or air conditioner, the artificial intelligence agent device can select the IOT control domain (841), the music domain (842), and the TV domain (843).

[0070] Meanwhile, the artificial intelligence agent device may create a single domain (e.g., a living room domain) by combining the IOT control domain (841), the music domain (842), and the TV domain (843). For example, if the artificial intelligence agent device detects that a user is present at a location for a predetermined period of time or more than a predetermined number of times and uses all of the IOT control domain (841), the music domain (842), and the TV domain (843) more than a predetermined number of times, the artificial intelligence agent device may create a living room domain by combining the IOT control domain (841), the music domain (842), and the TV domain (843).

[0071]

[0072] FIG. 3 is a flowchart illustrating an example of selecting a voice processing model based on visual information and the user's location in an artificial intelligence agent device according to one embodiment.

[0073] Referring to Figure 3, the artificial intelligence agent device can analyze an image to extract multiple pieces of information (310). Step 310 can extract multiple pieces of information, similar to step 210.

[0074] In addition, the AI ​​agent device can identify the user's location (320). In step 320, the AI ​​agent device can calculate location information of the user using a global positioning system (GPS), room information in the home using ultra-wideband (UWB) technology, or distance information between an object and the user. For example, the AI ​​agent device can identify object names such as "Coffee Shop A" or "Bus Stop" in an image received through a camera and calculate the distance between the object and the user.

[0075] Furthermore, the AI ​​agent device can select at least one domain from among multiple domains based on the extracted information and the user's location (330). For example, in step 330, if meaningful object names "Coffee A" and "Bus Stop" are determined based on information extracted from an image received through the camera, the AI ​​agent device can select the Coffee A domain and the Bus Route Map domain as the domains to be loaded.

[0076] Additionally, in step 330, the AI ​​agent device may select a domain based on the extracted information and the user's location, such as a change in region / country, such as moving to another country. For example, if the user travels from Korea to the United States or Europe, the AI ​​agent device may select a domain such as "translation domain" if the country location has changed based on location information and the language information analyzed from the image input through the camera has also changed from Korean to English. As another example, if the user's location has changed from Seoul to Jeju Island or Gangwon Province, the AI ​​agent device may select a domain such as "travel information domain" or "Jeju Island travel domain."

[0077] And, the artificial intelligence agent device can select at least one voice processing model among multiple voice processing models based on the selected domain (340).

[0078] Additionally, the artificial intelligence agent device can display visual information corresponding to the selected domain (350). Step 350 can display visual information similar to step 240.

[0079]

[0080] FIG. 4 is a flowchart illustrating an example of selecting a voice processing model based on visual information, previous visual information, and the user's location in an artificial intelligence agent device according to one embodiment.

[0081] Referring to FIG. 4, the artificial intelligence agent device can analyze an image to extract multiple pieces of information and verify the extracted information (410). Step 410 can extract multiple pieces of information, similar to step 210.

[0082] Additionally, the artificial intelligence agent device can identify the user's location (420). Step 420 can identify the user's location, similar to step 320.

[0083] And, the artificial intelligence agent device can check the extracted previous information, which is information extracted from the previous image used when selecting the previous voice processing model (430).

[0084] And, the artificial intelligence agent device can select at least one domain among a plurality of domains based on the extracted previous information, the extracted information, and the user's location (440).

[0085] At step 440, the artificial intelligence agent device can select a domain based on sequentially incoming information such as extracted previous information and extracted information.

[0086] An example of selecting a domain based on sequentially incoming information is described with reference to FIGS. 9 and 10.

[0087] FIG. 9 is a diagram illustrating an example of selecting a domain by considering visual information that changes over time in an artificial intelligence agent device according to one embodiment.

[0088] Referring to FIG. 9, the artificial intelligence agent device identifies object names such as “Coffee A”, “Coffee B”, and “Bus Stop” as meaningful information in the analysis content (922) of the image (920) received at a first time as the user (910) moves, identifies object names such as “Coffee B” and “Bus Stop” as meaningful information in the analysis content (932) of the image (930) received at a second time, identifies object names such as “Coffee B”, “Table”, “Chair”, and “Menu” as meaningful information in the analysis content (942) of the image (940) received at a third time, and if the user (910) confirms that the location information remains within a certain area, the domain to be loaded can be determined as the domain of Coffee B.

[0089] FIG. 10 is a diagram illustrating another example of selecting a domain by considering visual information that changes over time in an artificial intelligence agent device according to one embodiment.

[0090] Referring to FIG. 10, the artificial intelligence agent device identifies object names such as “Coffee A”, “Coffee B”, and “Bus Stop” as meaningful information in the analysis content (1022) of the image (1020) received at a first time as the user (1010) moves, identifies object names such as “Coffee B” and “Bus Stop” as meaningful information in the analysis content (1032) of the image (1030) received at a second time, identifies object names such as “Bus Stop” as meaningful information in the analysis content (1042) of the image (1040) received at a third time, and if the user (1010) confirms that the location information remains within a certain area, the device can determine the domain to be loaded as a domain related to a bus route map.

[0091] And, the artificial intelligence agent device can select at least one voice processing model among multiple voice processing models based on the selected domain (450).

[0092] Additionally, the artificial intelligence agent device can display visual information corresponding to the selected domain (460). Step 460 can display visual information similar to step 240.

[0093]

[0094] FIG. 11 is a diagram illustrating the configuration of an artificial intelligence agent device that takes visual information into account according to one embodiment.

[0095] Referring to FIG. 11, the artificial intelligence agent device (1100) may include a processor (1110), a communication unit (1130), a display unit (1140), a domain database (1150), a camera (1160), a location sensor (1170), and a microphone (1180).

[0096] The communication unit (1130) is a communication interface device including a receiver and a transmitter, and can communicate with an intelligent server that processes spoken voice and responds with the processing result.

[0097] The display unit (1140) displays status information (or indicators), limited numbers and characters, moving pictures, and still pictures generated during the operation of the artificial intelligence agent device (1100). In addition, the display unit (1140) can display a selected domain under the control of the processor (1110). At this time, the selected domain can be displayed as text or in the form of an icon.

[0098] The domain database (1150) can store information about at least one voice processing model corresponding to each of a plurality of domains.

[0099] The camera (1160) can capture an image in the direction the user is looking.

[0100] The location sensor (1170) can confirm the current location. The location sensor (1170) can confirm the location using a global positioning system (GPS) or an ultra-wideband (UWB) technology.

[0101] The microphone (1180) can process external acoustic signals into electrical voice data.

[0102] The processor (1110) may include an image collection unit (1111), a location collection unit (1112), a dynamic model selection unit (1113), a domain analysis unit (1114), and an artificial intelligence agent processing unit (1120).

[0103] The image collection unit (1111) can receive an image from the camera (1160). The image is an image of the direction the user is looking. When receiving an image, the image collection unit (1111) can also check gaze information indicating the object the user is looking at in the image.

[0104] The location collection unit (1112) can confirm the user's location. The user's location can be confirmed through the location sensor (1170).

[0105] The dynamic model selection unit (1113) can select at least one speech processing model among a plurality of speech processing models based on an image.

[0106] The dynamic model selection unit (1113) can repeat the operation of selecting a speech processing model based on an image at preset time intervals. If the user's location does not change for a preset time, the dynamic model selection unit (1113) can perform the operation of selecting a speech processing model based on an image.

[0107] The dynamic model selection unit (1113) can analyze an image to extract a plurality of pieces of information, select at least one domain from among a plurality of domains based on the extracted information, and select at least one voice processing model from among a plurality of voice processing models based on the selected domain.

[0108] The dynamic model selection unit (1113) can analyze an image to extract multiple pieces of information, identify the user's location, select at least one domain from among multiple domains based on the extracted information and the user's location, and select at least one voice processing model from among multiple voice processing models based on the selected domain. At this time, the dynamic model selection unit (1113) can search for a domain through a domain database (1150).

[0109] The dynamic model selection unit (1113) analyzes an image to extract multiple pieces of information, verifies the extracted information, verifies the user's location, verifies the extracted previous information, which is information extracted from a previous image used when selecting a previous voice processing model, selects at least one domain from among a plurality of domains based on the extracted previous information, the extracted information, and the user's location, and selects at least one voice processing model from among a plurality of voice processing models based on the selected domain.

[0110] The dynamic model selection unit (1113) can display visual information corresponding to the selected domain on the display unit (1140).

[0111] The dynamic model selection unit (1113) can load the selected domain.

[0112] Specifically, the dynamic model selection unit (1113) can apply at least one speech processing model corresponding to the selected domain to at least one of an automatic speech recognition module (ASR module), a natural language understanding module (NLU module), a natural language generator module (NLG module), and a text to speech module (TTS module).

[0113] The domain analysis unit (1114) can select a speech processing model based on speech when the speech processing model selected based on the image in the dynamic model selection unit (1113) cannot process speech under the control of the dynamic model selection unit (1113).

[0114] The domain analysis unit (1114) can determine an object name that can be known in the utterance, select a domain related to the object name, and select at least one speech processing model corresponding to the selected domain.

[0115] The artificial intelligence agent processing unit (1120) can process the user's speech according to the selected domain.

[0116] The artificial intelligence agent processing unit (1120) may include an automatic speech recognition module (ASR module) (1121), a natural language understanding module (NLU module) (1122), a natural language generator module (NLG module) (1123), a text-to-speech conversion module (TTS module) (1124), and an image processing module (1125).

[0117] The ASR module (1121) can convert received voice input into text data.

[0118] The NLU module (1122) can identify the user's intention using text data of voice input.

[0119] The NLG module (1123) can change the specified information into text format.

[0120] The TTS module (1124) can change information in text form into information in voice form.

[0121] The image processing module (1125) can analyze the received image.

[0122] The artificial intelligence agent processing unit (1120) may be configured to be included in the artificial intelligence agent device (1100), but may also be implemented as an intelligent device located externally, and may respond with the processing result through communication via the communication unit (1130).

[0123]

[0124] Meanwhile, the artificial intelligence agent device (1200) of FIG. 11 may be configured in the form of an electronic device (1201) in a network environment as in FIG. 12 below, or may be configured in the form of wearable augmented reality glasses (1300) as in FIG. 13.

[0125] FIG. 12 is a block diagram of an electronic device (1201) within a network environment (1200) according to one embodiment.

[0126] Referring to FIG. 12, in a network environment (1200), an electronic device (1201) may communicate with an electronic device (1202) via a first network (1298) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1204) or a server (1208) via a second network (1299) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1201) may communicate with the electronic device (1204) via the server (1208). According to one embodiment, the electronic device (1201) may include a processor (1220), a memory (1230), an input module (1250), an audio output module (1255), a display module (1260), an audio module (1270), a sensor module (1276), an interface (1277), a connection terminal (1278), a haptic module (1279), a power management module (1288), a battery (1289), a communication module (1290), a subscriber identification module (1296), or an antenna module (1297). In some embodiments, the electronic device (1201) may omit at least one of these components (e.g., the connection terminal (1278)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1276), camera module (1280), or antenna module (1297)) may be integrated into a single component (e.g., display module (1260)).

[0127] The processor (1220) may control at least one other component (e.g., a hardware or software component) of the electronic device (1201) connected to the processor (1220) by executing, for example, software (e.g., a program (1240)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1220) may store commands or data received from other components (e.g., a sensor module (1276) or a communication module (1290)) in a volatile memory (1232), process the commands or data stored in the volatile memory (1232), and store result data in a non-volatile memory (1234). According to one embodiment, the processor (1220) may include a main processor (1221) (e.g., a central processing unit or an application processor) or a secondary processor (1223) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1221). For example, when the electronic device (1201) includes the main processor (1221) and the secondary processor (1223), the secondary processor (1223) may be configured to use less power than the main processor (1221) or to be specialized for a given function. The secondary processor (1223) may be implemented separately from the main processor (1221) or as a part thereof.

[0128] The auxiliary processor (1223) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1260), the sensor module (1276), or the communication module (1290)) of the electronic device (1201), for example, on behalf of the main processor (1221) while the main processor (1221) is in an inactive (e.g., sleep) state, or together with the main processor (1221) while the main processor (1221) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1223) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1280) or a communication module (1290)). In one embodiment, the auxiliary processor (1223) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1201) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1208)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0129] Meanwhile, the processor (1220) can perform the operation of the processor (1110) of FIG. 11.

[0130] The memory (1230) can store various data used by at least one component (e.g., the processor (1220) or the sensor module (1276)) of the electronic device (1201). The data can include, for example, software (e.g., the program (1240)) and input data or output data for commands related thereto. The memory (1230) can include a volatile memory (1232) or a non-volatile memory (1234).

[0131] The memory (1230) may be configured to include the domain database (1150) of FIG. 11.

[0132] The program (1240) may be stored as software in memory (1230) and may include, for example, an operating system (1242), middleware (1244), or an application (1246).

[0133] The input module (1250) can receive commands or data to be used in a component of the electronic device (1201) (e.g., a processor (1220)) from an external source (e.g., a user) of the electronic device (1201). The input module (1250) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0134] The audio output module (1255) can output audio signals to the outside of the electronic device (1201). The audio output module (1255) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0135] The display module (1260) can visually provide information to an external party (e.g., a user) of the electronic device (1201). The display module (1260) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1260) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0136] The audio module (1270) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1270) can acquire sound through the input module (1250), output sound through the sound output module (1255), or an external electronic device (e.g., electronic device (1202)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1201).

[0137] The sensor module (1276) can detect the operating status (e.g., power or temperature) of the electronic device (1201) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1276) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, a Hall sensor, or an illuminance sensor.

[0138] Additionally, the sensor module (1276) may further include a camera module capable of capturing still images and moving images. In this case, the camera module may include one or more lenses, image sensors, image signal processors, or flashes.

[0139] The interface (1277) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1201) with an external electronic device (e.g., the electronic device (1202)). In one embodiment, the interface (1277) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0140] For example, the electronic device (1201) can transmit a video signal to an external electronic device through a connection terminal (1278). The electronic device (1201) can transmit a video signal for the external electronic device to output a video to a display module (1260) of the external electronic device.

[0141] The connection terminal (1278) may be for outputting a video signal or for outputting an audio signal. For example, the connection terminal (1278) may be for outputting a video signal and an audio signal simultaneously. For example, the electronic device (1201) may output a video signal and an audio signal to an external electronic device through an interface such as HDMI, DP, or Thunderbolt from the connection terminal (1278) that outputs a video signal and an audio signal simultaneously.

[0142] The connection terminal (1278) may include a connector through which the electronic device (1201) may be physically connected to an external electronic device (e.g., the electronic device (1202)). According to one embodiment, the connection terminal (1278) may include, for example, an HDMI connector, a DP connector, a Thunderbolt connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0143] The haptic module (1279) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1279) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0144] The power management module (1288) can manage power supplied to the electronic device (1201). According to one embodiment, the power management module (1288) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0145] A battery (1289) may power at least one component of the electronic device (1201). In one embodiment, the battery (1289) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0146] The communication module (1290) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1201) and an external electronic device (e.g., electronic device (1202), electronic device (1204), or server (1208)), and the performance of communication through the established communication channel. The communication module (1290) may operate independently from the processor (1220) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1290) may include a wireless communication module (1292) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1294) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1204) via a first network (1298) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1299) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1292) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1296) to verify or authenticate the electronic device (1201) within a communication network such as the first network (1298) or the second network (1299).

[0147] The wireless communication module (1292) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1292) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1292) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1292) can support various requirements specified in the electronic device (1201), an external electronic device (e.g., the electronic device (1204)), or a network system (e.g., the second network (1299)). According to one embodiment, the wireless communication module (1292) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0148] The antenna module (1297) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1297) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1297) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1298) or the second network (1299), may be selected from the plurality of antennas by, for example, the communication module (1290). A signal or power may be transmitted or received between the communication module (1290) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1297).

[0149] According to various embodiments, the antenna module (1297) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0150] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0151] According to one embodiment, commands or data may be transmitted or received between the electronic device (1201) and an external electronic device (1204) via a server (1208) connected to a second network (1299). Each of the external electronic devices (1202 or 1204) may be the same or a different type of device as the electronic device (1201). According to one embodiment, all or part of the operations executed in the electronic device (1201) may be executed in one or more of the external electronic devices (1202, 1204, or 1208). For example, when the electronic device (1201) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1201) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1201). The electronic device (1201) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1201) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1204) may include an Internet of Things (IoT) device. The server (1208) may be an intelligent server utilizing machine learning and / or a neural network.In one embodiment, an external electronic device (1204) or server (1208) may be included within the second network (1299). The electronic device (1201) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0152]

[0153] FIG. 13 is a diagram illustrating the structure of an electronic device implemented in the form of wearable augmented reality glasses according to one embodiment.

[0154] Referring to FIG. 13, an electronic device (1300) may be worn on a user's face and may provide the user with images related to an augmented reality service and / or a virtual reality service.

[0155] In one embodiment, the electronic device (1300) includes a first display (1305), a second display (1310), a first screen display unit (1315a), a second screen display unit (1315b), an input optical member (1320), a first transparent member (1325a), a second transparent member (1325b), a lighting unit (1330a, 1330b), a first PCB (1335a), a second PCB (1335b), a first hinge (1340a), a second hinge (1340b), a first camera (1345a, 1345b, 1345c, 1345d), a plurality of microphones (e.g., a first microphone (1350a), a second microphone (1350b), a third microphone (1350c)), a plurality of speakers (e.g., a first speaker (1355a), a second It may include a speaker (1355b)), a battery (1360), a second camera (1375a, 1375b), a third camera (1365), and a visor (1370a, 1370b).

[0156] In one embodiment, the displays (e.g., the first display (1305) and the second display (1310)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the displays are made of one of the liquid crystal display, the digital mirror display, or the silicon liquid crystal display, the electronic device (1300) may include a light source that irradiates light onto a screen output area of ​​the display. In another embodiment, when the displays are capable of generating light on their own, for example, when they are made of one of the organic light emitting diodes or the micro LED, the electronic device (1300) may provide a good quality virtual image to the user even without including a separate light source. In one embodiment, if the display is implemented with an organic light-emitting diode or micro LED, a light source is unnecessary, and thus the electronic device (1300) can be made lighter. In the following, a display capable of generating light on its own is referred to as a self-luminous display, and the description is based on the assumption of a self-luminous display.

[0157] Displays according to various embodiments of the present disclosure (e.g., the first display (1305) and the second display (1310)) may be composed of at least one micro LED (micro light emitting diode). For example, the micro LED can express red (R), green (G), and blue (B) by self-luminescence, and is small in size (e.g., 100 μm or less), so that one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display is composed of the micro LED, it can provide a high resolution without a backlight unit (BLU).

[0158] Not limited thereto, one pixel may include R, G, and B, and one chip may be implemented with multiple pixels including R, G, and B.

[0159] In one embodiment, the display (e.g., the first display (1305) and the second display (1310)) may be configured with a display area composed of pixels for displaying a virtual image and light-receiving pixels (e.g., photo sensor pixels) arranged between the pixels for receiving light reflected from the eye, converting it into electrical energy, and outputting it.

[0160] In one embodiment, the electronic device (1300) can detect the direction of the user's gaze (e.g., eye movement) through light-receiving pixels. For example, the electronic device (1300) can detect and track the direction of the user's gaze with respect to the right eye and the direction of the user's gaze with respect to the left eye through one or more light-receiving pixels constituting the first display (1305) and one or more light-receiving pixels constituting the second display (1310). The electronic device (1300) can determine the position of the center of the virtual image based on the direction of the user's gaze with respect to the right and left eyes (e.g., the direction in which the pupils of the user's right and left eyes are gazing) detected through one or more light-receiving pixels.

[0161] In one embodiment, light emitted from displays (e.g., a first display (1305) and a second display (1310)) may pass through a lens (not shown) and a waveguide to reach a first screen display unit (1315a) formed on a first transparent member (1325a) positioned to face the user's right eye and a second screen display unit (1315b) formed on a second transparent member (1325b) positioned to face the user's left eye. For example, light emitted from displays (e.g., a first display (1305) and a second display (1310)) may pass through a waveguide to be reflected by a grating area formed on an input optical member (1320) and the screen display units (1315a, 1315b) and transmitted to the user's eyes. The first transparent member (1325a) and / or the second transparent member (1325b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent.

[0162] In one embodiment, a lens (not shown) may be positioned in front of a display (e.g., a first display (1305) and a second display (1310)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0163] In one embodiment, the screen display (1315a, 1315b) or the transparent member (e.g., the first transparent member (1325a), the second transparent member (1325b)) may include a lens including a waveguide, a reflective lens.

[0164] In one embodiment, the waveguide may be made of glass, plastic, or polymer, and may include nano-patterns formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the waveguide may be propagated within the display waveguide by the nano-patterns and provided to the user. In one embodiment, a waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element, for example, a reflective mirror. In one embodiment, the waveguide may guide light emitted from the display (1305, 1310) to the user's eyes by using at least one diffractive element or reflective element included in the waveguide.

[0165] According to various embodiments, the diffractive element may include an input optical member (1320) / output optical member (not shown). For example, the input optical member (1320) may mean an input grating area, and the output optical member (not shown) may mean an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from a display (e.g., a first display (1305) and a second display (1310)) (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (1350a), a second transparent member (1350b)) of a screen display unit (1315a, 1315b). The output grating region can act as an outlet to diffract (or reflect) light transmitted to the transparent member of the waveguide (e.g., the first transparent member (1350a), the second transparent member (1350b)) toward the user's eyes.

[0166] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0167] In one embodiment, light emitted from a display (1305, 1310) may be guided along an optical path through an input optical member (1320) into a waveguide. Light traveling within the waveguide may be guided toward a user's eyes through an output optical member. The screen display (1315a, 1315b) may be determined based on the light emitted toward the user's eyes.

[0168] In one embodiment, the first camera (1345a, 1345b, 1345c, 1345d) may include a camera used for 6 degrees of freedom (6DoF), 6DoF head tracking, hand detection and tracking, gesture and / or spatial recognition. For example, the first camera (1345a, 1345b, 1345c, 1345d) may include a global shutter (GS) camera to detect and track head and hand movements.

[0169] For example, the first camera (1345a, 1345b, 1345c, 1345d) may be a stereo camera for head tracking and spatial recognition, and cameras of the same specifications and performance may be used. The first camera (1345a, 1345b, 1345c, 1345d) may be a GS camera with excellent performance (e.g., image drag) for detecting and tracking fine movements such as rapid hand movements and fingers.

[0170] According to various embodiments, the first camera (1345a, 1345b, 1345c, 1345d) may be a rolling shutter (RS) camera. The first camera (1345a, 1345b, 1345c, 1345d) may perform SLAM functions through spatial recognition and depth shooting for 6 DoF. The first camera (1345a, 1345b, 1345c, 1345d) may perform a user gesture recognition function.

[0171] In one embodiment, the second camera (1375a, 1375b) may be used to detect and track the pupil. The second camera (1375a, 1375b) may be referred to as an ET (eye tracking) camera. The second camera (1375a, 1375b) may track the user's gaze direction. The electronic device (1300) may consider the user's gaze direction and position the center of the virtual image projected on the screen display unit (1315a, 1315b) according to the direction in which the user's pupil is looking.

[0172] The second camera (1375a, 1375b) for tracking the gaze direction may be a GS camera that can detect the pupil and track rapid eye movements. The second cameras (1375a, 1375b) may be installed for the left and right eyes, respectively, and the second cameras (1375a, 1375b) for the left and right eyes may be cameras with identical performance and specifications.

[0173] In one embodiment, the third camera (1365) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (1365) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an OIS (optical image stabilizer). However, the third camera (1365) is not limited thereto, and may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0174] In one embodiment, at least one sensor (e.g., a gyroscope sensor, an acceleration sensor, a magnetometer sensor, a touch sensor, an ambient light sensor, and / or a gesture sensor), and a first camera (1345a, 1345b, 1345c, 1345d) can perform at least one of head tracking for 6DoF, pose estimation & prediction, gesture and / or spatial recognition, and slam function with depth shooting.

[0175] In another embodiment, the first camera (1345a, 1345b, 1345c, 1345d) may be used separately as a camera for head tracking and a camera for hand tracking.

[0176] In one embodiment, the lighting units (1330a, 1330b) may have different uses depending on the attachment location. For example, the lighting units (1330a, 1330b) may be attached together with the first camera (1345a, 1345b, 1345c, 1345d) mounted around a hinge (e.g., a first hinge (1340a), a second hinge (1340b)) connecting a frame and a temple, or around a bridge connecting the frames. When shooting with a GS camera, the lighting units (1330a, 1330b) may be used as a means of supplementing the ambient brightness. For example, the lighting units (1330a, 1330b) may be used when it is difficult to detect a subject to be shot due to a dark environment or mixing or reflecting light from multiple light sources.

[0177] In one embodiment, a PCB (e.g., a first PCB (1335a), a second PCB (1335b)) may include a processor (not shown), a memory (not shown), and a communication module (not shown) that control components of the electronic device (1300).

[0178] A communication module (not shown) may support the establishment of a direct (e.g., wired) or wireless communication channel between the electronic device (1300) and an external electronic device, and the performance of communication through the established communication channel. The PCB may transmit electrical signals to components constituting the electronic device (1300).

[0179] A communication module (not shown) may include one or more communication processors that operate independently from the processor and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (not shown) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules (not shown) may communicate with an external electronic device via a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0180] The wireless communication module can support 5G networks and next-generation communication technologies beyond 4G networks, such as new radio access technology (NR). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module can support high-frequency bands (e.g., mmWave bands) to achieve high data rates, for example. The wireless communication module may support various technologies to secure performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.

[0181] The electronic device (1300) may further include an antenna module (not shown). The antenna module may transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (1335a), a second PCB (1335b)). In one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0182] In one embodiment, multiple microphones (e.g., a first microphone (1350a), a second microphone (1350b), and a third microphone (1350c)) can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) by the electronic device (1300).

[0183] In one embodiment, multiple speakers (e.g., first speaker (1355a), second speaker (1355b)) can output audio data received from a communication module or stored in a memory.

[0184] In one embodiment, one or more batteries (1360) may be included and may supply power to components that make up the electronic device (1300).

[0185] In one embodiment, the visor (1370a, 1370b) can adjust the amount of external light transmitted to the user's eyes according to the transmittance. The visor (1370a, 1370b) can be positioned in front or behind the screen display unit (1315a, 1315b). The front of the screen display unit (1315a, 1315b) can mean the direction opposite to the user wearing the electronic device (1300), and the back can mean the direction toward the user wearing the electronic device (1300). The visor (1370a, 1370b) can protect the screen display unit (1315a, 1315b) and adjust the amount of external light transmitted.

[0186] For example, the visor (1370a, 1370b) may include an electrochromic element that changes color according to the applied power to control the transmittance. Electrochromism is a phenomenon in which the color changes due to an oxidation-reduction reaction caused by the applied power. The visor (1370a, 1370b) can control the transmittance of external light by utilizing the color change of the electrochromic element.

[0187] For example, the visor (1370a, 1370b) may include a control module and an electrochromic element. The control module may control the electrochromic element to adjust the transmittance of the electrochromic element.

[0188]

[0189] In addition, the artificial intelligence agent device (1100) of FIG. 11 may be implemented as the integrated intelligence system (10) of FIG. 14, and when the artificial intelligence agent processing unit (1120) of the artificial intelligence agent device (1100) is implemented as an external configuration, the artificial intelligence agent processing unit (1120) may be implemented as the integrated intelligence system (10) of FIG. 14.

[0190] FIG. 14 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0191] Referring to FIG. 14, an integrated intelligence system (10) of one embodiment may include a user terminal (1400), an intelligent server (1500), and a service server (1600).

[0192] The user terminal (1400) of one embodiment may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0193] According to the illustrated embodiment, the user terminal (1400) may include a communication interface (1410), a microphone (1420), a speaker (1430), a display (1440), a memory (1450), or a processor (1460). The above-listed components may be operatively or electrically connected to each other.

[0194] The communication interface (1410) of one embodiment may be configured to connect to an external device and transmit and receive data. The microphone (1420) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. The speaker (1430) of one embodiment may output the electrical signal as sound (e.g., voice). The display (1440) of one embodiment may be configured to display an image or video. The display (1440) of one embodiment may also display a graphical user interface (GUI) of a running app (or application program).

[0195] In one embodiment, the memory (1450) may store a client module (1451), a software development kit (SDK) (1453), and multiple apps (1455). The client module (1451) and the SDK (1453) may constitute a framework (or solution program) for performing general functions. In addition, the client module (1451) or the SDK (1453) may constitute a framework for processing voice input.

[0196] In one embodiment, the memory (1450) may be a program for performing a specified function, wherein the plurality of apps (1455) may include a first app (1455_1) and a second app (1455_3). In one embodiment, each of the plurality of apps (1455) may include a plurality of operations for performing a specified function. For example, the apps may include an alarm app, a message app, and / or a schedule app. In one embodiment, the plurality of apps (1455) may be executed by the processor (1460) to sequentially execute at least some of the plurality of operations.

[0197] The processor (1460) of one embodiment can control the overall operation of the user terminal (1400). For example, the processor (1460) can be electrically connected to a communication interface (1410), a microphone (1420), a speaker (1430), and a display (1440) to perform a specified operation.

[0198] The processor (1460) of one embodiment may also execute a program stored in the memory (1450) to perform a designated function. For example, the processor (1460) may execute at least one of the client module (1451) or the SDK (1453) to perform the following operations for processing voice input. The processor (1460) may control the operations of multiple apps (1455), for example, through the SDK (1453). The following operations described as operations of the client module (1451) or the SDK (1453) may be operations executed by the processor (1460).

[0199] The client module (1451) of one embodiment can receive a voice input. For example, the client module (1451) can receive a voice signal corresponding to a user utterance detected through a microphone (1420). The client module (1451) can transmit the received voice input to the intelligent server (1500). The client module (1451) can transmit status information of the user terminal (1400) together with the received voice input to the intelligent server (1500). The status information can be, for example, execution status information of an app.

[0200] The client module (1451) of one embodiment may receive a result corresponding to the received voice input. For example, the client module (1451) may receive a result corresponding to the received voice input if the intelligent server (1500) can produce a result corresponding to the received voice input. The client module (1451) may display the received result on the display (1440).

[0201] In one embodiment, the client module (1451) can receive a plan corresponding to the received voice input. The client module (1451) can display the results of executing multiple operations of the app according to the plan on the display (1440). For example, the client module (1451) can sequentially display the results of executing multiple operations on the display. The user terminal (1400) can, for example, display only some of the results of executing multiple operations (e.g., the result of the last operation) on the display.

[0202] According to one embodiment, the client module (1451) may receive a request from the intelligent server (1500) to obtain information necessary to produce a result corresponding to a voice input. According to one embodiment, the client module (1451) may transmit the necessary information to the intelligent server (1500) in response to the request.

[0203] The client module (1451) of one embodiment can transmit result information of executing multiple operations according to a plan to the intelligent server (1500). The intelligent server (1500) can use the result information to confirm that the received voice input has been processed correctly.

[0204] The client module (1451) of one embodiment may include a voice recognition module. According to one embodiment, the client module (1451) may recognize voice inputs that perform limited functions through the voice recognition module. For example, the client module (1451) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0205] An intelligent server (1500) of one embodiment may receive information related to a user voice input from a user terminal (1400) via a communication network. According to one embodiment, the intelligent server (1500) may convert data related to the received voice input into text data. According to one embodiment, the intelligent server (1500) may generate a plan for performing a task corresponding to the user voice input based on the text data.

[0206] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN), a recurrent neural network (RNN), or a combination of the above, or another AI system. In one embodiment, the plan may be selected from a set of predefined plans, or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0207] An intelligent server (1500) of one embodiment may transmit results according to a generated plan to a user terminal (1400), or transmit the generated plan to the user terminal (1400). According to one embodiment, the user terminal (1400) may display results according to the plan on a display. According to one embodiment, the user terminal (1400) may display results of executing an operation according to the plan on a display.

[0208] An intelligent server (1500) of one embodiment may include a front end (1510), a natural language platform (1520), a capsule database (1530), an execution engine (1540), an end user interface (1550), a management platform (1560), a big data platform (1570), or an analytic platform (1580).

[0209] The front end (1510) of one embodiment can receive a voice input from a user terminal (1400). The front end (1510) can transmit a response corresponding to the voice input.

[0210] According to one embodiment, the natural language platform (1520) may include an automatic speech recognition module (ASR module) (1521), a natural language understanding module (NLU module) (1523), a planner module (1525), a natural language generator module (NLG module) (1527), or a text to speech module (TTS module) (1529).

[0211] An automatic speech recognition module (1521) of one embodiment can convert a voice input received from a user terminal (1400) into text data. A natural language understanding module (1523) of one embodiment can use the text data of the voice input to determine the user's intention. For example, the natural language understanding module (1523) can perform syntactic analysis or semantic analysis to determine the user's intention. The natural language understanding module (1523) of one embodiment can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the voice input, and can match the meaning of the determined words to the intention to determine the user's intention.

[0212] In one embodiment, the planner module (1525) can generate a plan using the intent and parameters determined by the natural language understanding module (1523). According to one embodiment, the planner module (1525) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (1525) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (1525) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts of a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (1525) can determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (1525) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (1525) can determine the execution order of a plurality of actions based on the parameters required for the execution of the plurality of actions and the results output by the execution of the plurality of actions. Accordingly, the planner module (1525) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (1525) can generate the plan using information stored in a capsule database (1530) in which a set of relationships between concepts and actions is stored.

[0213] The natural language generation module (1527) of one embodiment can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (1529) of one embodiment can convert information in text format into information in speech format.

[0214] According to one embodiment, some or all of the functions of the natural language platform (1520) may also be implemented in the user terminal (1400).

[0215] The capsule database (1530) can store information on the relationships between multiple concepts and actions corresponding to multiple domains. According to one embodiment, a capsule can include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (1530) can store multiple capsules in the form of a concept action network (CAN). According to one embodiment, the multiple capsules can be stored in a function registry included in the capsule database (1530).

[0216] The capsule database (1530) may have a configuration corresponding to the domain database (1150) of FIG. 11, and a capsule of the capsule database (1530) may correspond to a domain of the domain database (1150).

[0217] The capsule database (1530) may include a strategy registry that stores strategy information necessary for determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a voice input. According to one embodiment, the capsule database (1530) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (1530) may include a layout registry that stores layout information of information output through the user terminal (1400). According to one embodiment, the capsule database (1530) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (1530) may include a dialog registry that stores information on dialogue (or interaction) with a user. The capsule database (1530) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans.The developer tool may include a dialog editor that creates a dialogue with the user. The developer tool may also include a follow-up editor that activates follow-up goals and allows editing of follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions. In one embodiment, the capsule database (1530) may also be implemented within the user terminal (1400).

[0218] The execution engine (1540) of one embodiment can produce a result using the generated plan. The end user interface (1550) can transmit the produced result to the user terminal (1400). Accordingly, the user terminal (1400) can receive the result and provide the received result to the user. The management platform (1560) of one embodiment can manage information used in the intelligent server (1500). The big data platform (1570) of one embodiment can collect user data. The analysis platform (1580) of one embodiment can manage the quality of service (QoS) of the intelligent server (1500). For example, the analysis platform (1580) can manage the components and processing speed (or efficiency) of the intelligent server (1500).

[0219] In one embodiment, a service server (1600) may provide a designated service (e.g., food ordering or hotel reservation) to a user terminal (1400). According to one embodiment, the service server (1600) may be a server operated by a third party. In one embodiment, the service server (1600) may provide information for generating a plan corresponding to the received voice input to the intelligent server (1500). The provided information may be stored in the capsule database (1530). In addition, the service server (1600) may provide result information according to the plan to the intelligent server (1500).

[0220] In the integrated intelligence system (10) described above, the user terminal (1400) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0221] In one embodiment, the user terminal (1400) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (1400) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0222] In one embodiment, the user terminal (1400) may perform a designated action, either alone or in conjunction with the intelligent server and / or service server, based on the received voice input. For example, the user terminal (1400) may execute an app corresponding to the received voice input and perform a designated action through the executed app.

[0223] In one embodiment, when a user terminal (1400) provides a service together with an intelligent server (1500) and / or a service server, the user terminal may detect user speech using the microphone (1420) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal may transmit the voice data to the intelligent server (1500) using a communication interface (1410).

[0224] According to one embodiment, an intelligent server (1500) may generate a plan for performing a task corresponding to a voice input received from a user terminal (1400), or a result of performing an operation according to the plan, in response to the voice input. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concepts may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0225] In one embodiment, the user terminal (1400) can receive the response using the communication interface (1410). The user terminal (1400) can output a voice signal generated within the user terminal (1400) to the outside using the speaker (1430), or can output an image generated within the user terminal (1400) to the outside using the display (1440).

[0226]

[0227] FIG. 15 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to one embodiment.

[0228] Referring to FIG. 15, the capsule database (e.g., capsule database (1530)) of the intelligent server (1500) can store capsules in the form of a CAN (concept action network). The capsule database can store operations for processing tasks corresponding to a user's voice input and parameters necessary for the operations in the form of a CAN (concept action network).

[0229] The capsule database may store a plurality of capsules (e.g., capsule(A)(1701), capsule(B)(1704)) corresponding to each of a plurality of domains (e.g., applications). In one embodiment, one capsule (e.g., capsule(A)(1701)) may correspond to one domain (e.g., location (geo), application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (1702) or CP 2 (1703)) for performing a function for a domain related to the capsule. In one embodiment, one capsule may include at least one operation (1710) and at least one concept (1720) for performing a specified function.

[0230] The above natural language platform (1520) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module (1525) of the natural language platform can generate a plan using capsules stored in a capsule database. For example, a plan (1707) can be generated using actions (17011, 17013) and concepts (17012, 17014) of capsule A (1710) and actions (17041) and concepts (17042) of capsule B (1704).

[0231]

[0232] According to one embodiment, a method of operating an artificial intelligence agent may include: receiving an image from a camera; selecting at least one voice processing model from a plurality of voice processing models based on the image; loading the selected voice processing model; and processing a user's speech using the selected voice processing model.

[0233] According to one embodiment, the operation of selecting at least one voice processing model from among a plurality of voice processing models based on the image may include: analyzing the image to extract a plurality of pieces of information; selecting at least one domain from among a plurality of domains based on the extracted information; and selecting at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0234] According to one embodiment, the operation of selecting at least one speech processing model among a plurality of speech processing models based on the image may be repeated at preset time intervals.

[0235] According to one embodiment, the operation of selecting at least one speech processing model from among a plurality of speech processing models based on the image may further include an operation of canceling the selection of the selected domain and deleting the selected speech processing model loaded corresponding to the selected domain if the selected domain is not used for a preset period of time.

[0236] According to one embodiment, the operation of selecting at least one speech processing model from among a plurality of speech processing models based on the image may further include an operation of canceling the selection of the selected domain and deleting the selected speech processing model loaded corresponding to the selected domain if the image input for a preset period of time does not include information related to the selected domain.

[0237] According to one embodiment, the operation of selecting at least one voice processing model from among a plurality of voice processing models based on the image may include: analyzing the image to extract a plurality of pieces of information; confirming the location of the user; selecting at least one domain from among a plurality of domains based on the extracted information and the location of the user; and selecting at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0238] According to one embodiment, the operation of selecting at least one voice processing model from among a plurality of voice processing models based on the image may be performed if the user's location does not change for a preset period of time.

[0239] According to one embodiment, the operation of selecting at least one voice processing model from among a plurality of voice processing models based on the image may include: an operation of analyzing the image to confirm extracted information, which is information obtained by extracting a plurality of pieces of information; an operation of confirming the location of the user; an operation of confirming extracted previous information, which is information extracted from a previous image used when selecting a previous voice processing model; an operation of selecting at least one domain from among a plurality of domains based on the extracted previous information, the extracted information, and the location of the user; and an operation of selecting at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0240] According to one embodiment, a method of operating an artificial intelligence agent may further include an action of displaying visual information corresponding to the selected domain.

[0241] According to one embodiment, the operation of displaying visual information corresponding to the selected domain may include, when there are multiple selected domains, an operation of confirming the user's gaze; an operation of confirming a selected domain corresponding to the user's gaze among the multiple selected domains; and an operation of displaying the selected domain corresponding to the user's gaze so as to be distinguished from other selected domains when displaying visual information corresponding to each of the multiple selected domains.

[0242] According to one embodiment, the operation of processing the user's speech using the selected speech processing model may further include an operation of reselecting a speech processing model based on the speech from among the plurality of speech processing models if the user's speech cannot be processed using the selected speech processing model; and an operation of processing the user's speech using the reselected speech processing model.

[0243] According to one embodiment, an artificial intelligence agent device may include an image collection unit that receives an image from a camera; a dynamic model selection unit that selects at least one voice processing model from among a plurality of voice processing models based on the image and loads the selected voice processing model; and an artificial intelligence agent processing unit that processes a user's speech using the selected voice processing model.

[0244] According to one embodiment, the dynamic model selection unit can select at least one speech processing model from among a plurality of speech processing models based on the image at preset time intervals.

[0245] According to one embodiment, the dynamic model selection unit may analyze the image to extract a plurality of pieces of information, select at least one domain from among a plurality of domains based on the extracted information, and select at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0246] According to one embodiment, the artificial intelligence agent device further includes a location collection unit that confirms the location of the user, and the dynamic model selection unit analyzes the image to extract a plurality of pieces of information, collects the location of the user through the location collection unit, selects at least one domain from among a plurality of domains based on the extracted information and the location of the user, and selects at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0247] The dynamic model selection unit of the artificial intelligence agent device can display visual information corresponding to the selected domain.

[0248] According to one embodiment, when displaying visual information corresponding to the selected domain, if there are multiple selected domains, the dynamic model selection unit may check the user's gaze, check a selected domain corresponding to the user's gaze among the multiple selected domains, and display visual information corresponding to each of the multiple selected domains so that the selected domain corresponding to the user's gaze is displayed so as to be distinguished from other selected domains.

[0249] According to one embodiment, the artificial intelligence agent device further includes a location collection unit that confirms the location of the user, and the dynamic model selection unit analyzes the image to confirm extracted information, which is information obtained by extracting a plurality of pieces of information, collects the location of the user through the location collection unit, confirms extracted previous information, which is information extracted from a previous image used when selecting a previous voice processing model, selects at least one domain from among a plurality of domains based on the extracted previous information, the extracted information, and the location of the user, and selects at least one voice processing model from among the plurality of voice processing models based on the selected domain.

[0250] According to one embodiment, the artificial intelligence agent device further includes a domain analysis unit that selects a related domain based on an utterance, and the artificial intelligence agent processing unit, if the user's utterance cannot be processed using the selected voice processing model, reselects a voice processing model corresponding to the selected domain through the domain analysis unit, and processes the user's utterance using the reselected voice processing model.

[0251]

[0252] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., singly or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0253] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems, and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0254] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the above. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0255] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a method of operating an artificial intelligence agent, The act of receiving an image from a camera; An operation of selecting at least one speech processing model among a plurality of speech processing models based on the above image; An operation of loading the above-mentioned selected voice processing model; and An action to process the user's speech using the above-mentioned selected voice processing model. A method including:

2. In paragraph 1 Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: An action of extracting multiple pieces of information by analyzing the above image; An operation of selecting at least one domain among a plurality of domains based on the extracted information; and An operation of selecting at least one speech processing model among the plurality of speech processing models based on the selected domain. A method including:

3. In any one of paragraphs 1 and 2, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: Repeat at preset time intervals method.

4. In any one of paragraphs 1 to 3, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: If the above-mentioned selected domain is not used for a preset period of time, the operation of canceling the selection of the above-mentioned selected domain and deleting the above-mentioned selected voice processing model loaded in response to the above-mentioned selected domain How to include more.

5. In any one of paragraphs 1 to 4, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: If the image input for a preset period of time does not include information related to the selected domain, the operation of canceling the selection of the selected domain and deleting the selected voice processing model loaded in response to the selected domain How to include more.

6. In any one of paragraphs 1 to 5, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: An action of extracting multiple pieces of information by analyzing the above image; An action to determine the location of the above user; An operation of selecting at least one domain among a plurality of domains based on the extracted information and the location of the user; and An operation of selecting at least one speech processing model among the plurality of speech processing models based on the selected domain. A method including:

7. In any one of paragraphs 1 to 6, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: If the user's location does not change for a preset period of time, method.

8. In any one of paragraphs 1 to 7, Based on the above image, the operation of selecting at least one speech processing model among multiple speech processing models is: An action of analyzing the above image and confirming the extracted information, which is information from which multiple pieces of information are extracted; An action to determine the location of the above user; An action to check the extracted previous information, which is information extracted from the previous image used when selecting the previous voice processing model; An operation of selecting at least one domain among a plurality of domains based on the extracted previous information, the extracted information, and the location of the user; and An operation of selecting at least one speech processing model among the plurality of speech processing models based on the selected domain. A method including:

9. In any one of paragraphs 1 to 8, An action to display visual information corresponding to the selected domain above. How to include more.

10. In any one of paragraphs 1 to 9, The action of displaying visual information corresponding to the above selected domain is: When there are multiple selected domains, an action to check the user's gaze; An operation of identifying a selected domain corresponding to the user's gaze among the plurality of selected domains; and When displaying visual information corresponding to each of the above multiple selected domains, an action is taken to display the selected domain corresponding to the user's gaze so as to be distinguished from other selected domains. A method including:

11. In any one of paragraphs 1 to 10, The operation of processing the user's speech using the above-mentioned selected voice processing model is as follows. If the user's speech cannot be processed using the selected speech processing model, an operation of reselecting a speech processing model based on the speech from among the plurality of speech processing models; and An action to process the user's speech using the above-mentioned reselected voice processing model. How to include more.

12. Image collection unit that receives images from the camera; A dynamic model selection unit that selects at least one speech processing model from among a plurality of speech processing models based on the image above and loads the selected speech processing model; and Artificial intelligence agent processing unit that processes the user's speech using the above-selected voice processing model An artificial intelligence agent device comprising:

13. In paragraph 12, The above dynamic model selection part, At least one speech processing model among a plurality of speech processing models is selected based on the image at preset time intervals. Artificial intelligence agent device.

14. In any one of paragraphs 12 to 13, The above dynamic model selection part, The above image is analyzed to extract a plurality of pieces of information, and at least one domain is selected from a plurality of domains based on the extracted information, and at least one voice processing model is selected from the plurality of voice processing models based on the selected domain. Artificial intelligence agent device.

15. In any one of paragraphs 12 to 14, Location collection unit that checks the location of the above user Including more, The above dynamic model selection part, The image is analyzed to extract multiple pieces of information, the location of the user is collected through the location collection unit, at least one domain is selected from multiple domains based on the extracted information and the location of the user, and at least one voice processing model is selected from multiple voice processing models based on the selected domain. Artificial intelligence agent device.

Images