Artificial intelligence server and method for providing information to user

Abstract

An embodiment of the present invention provides an artificial intelligence server that provides information to a user, comprising: a communication unit that communicates with a plurality of artificial intelligence devices arranged within a service area; and a processor that receives at least one of voice data of the user or terminal usage information of the user from at least one of the plurality of artificial intelligence devices, generates intention information of the user based on the received voice data or the received terminal usage information, generates state information of the user using the plurality of artificial intelligence devices, determines an information providing device among the plurality of artificial intelligence devices based on the generated state information of the user, generates output information to be output from the determined information providing device, and transmits a control signal to output the generated output information to the determined information providing device.

Description

Technology Field

[0001] The present invention relates to an artificial intelligence server and a method for providing information to a user. Specifically, the present invention relates to an artificial intelligence server and a method for providing information to a user using an artificial intelligence device deployed within a service area. Background Technology

[0002] The technology forming the background of the present invention is disclosed in Korean Published Patent Application No. 10-2019-0085895 (July 19, 2019). Recently, the widespread adoption of various artificial intelligence devices equipped with artificial intelligence has been active, and various services can be provided through artificial intelligence systems composed of artificial intelligence devices.

[0003] Artificial intelligence systems can be established within the home using various AI-equipped home appliances, autonomous vehicles, and AI speakers, while in places where many people gather, such as shopping malls, airports, and multiplexes, AI systems can be established using AI-equipped robots, kiosks, and digital signage.

[0004] However, currently, various artificial intelligence devices only interact with their respective fixed users and cannot operate organically to effectively provide information to multiple users. The problem to be solved

[0005] The present invention aims to provide an artificial intelligence server and a method thereof that identify a user's intention, determine an information providing device corresponding to the user's state among various artificial intelligence devices, and provide information suitable for the user's intention from the information providing device. means of solving the problem

[0006] One embodiment of the present invention provides an artificial intelligence server and a method thereof, which determine a user's intention based on information obtained from at least one of a plurality of artificial intelligence devices arranged in a service area, determine a user's state using a plurality of artificial intelligence devices, determine an information providing device among a plurality of artificial intelligence devices based on the user's state, generate output information to be output from the determined information providing device, and control the information providing device to output the output information. Effects of the invention

[0007] According to various embodiments of the present invention, a user can effectively receive information necessary for them from artificial intelligence devices placed in a service area, and each artificial intelligence device can prevent the provision of unnecessary information by outputting only the necessary information. Brief explanation of the drawing

[0008] FIG. 1 shows an AI device according to one embodiment of the present invention. FIG. 2 shows an AI server according to one embodiment of the present invention. FIG. 3 shows an AI system according to one embodiment of the present invention. FIG. 4 shows an AI device according to one embodiment of the present invention. FIG. 5 is an operation flowchart illustrating a method of providing information to a user according to one embodiment of the present invention. FIG. 6 is a diagram showing an example of generating user intention information according to one embodiment of the present invention. FIG. 7 is an operation flowchart illustrating a method of providing information to a user according to one embodiment of the present invention. FIGS. 8 to 11 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention. FIGS. 12 and 13 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention. FIGS. 14 to 16 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention. FIG. 17 is a drawing showing an example of providing information to a user according to one embodiment of the present invention. FIG. 18 is a drawing showing an example of providing information to a user according to one embodiment of the present invention. Specific details for implementing the invention

[0009] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components, regardless of drawing symbols, are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes 'module' and 'part' for components used in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0010] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0011] When it is stated that one component is 'connected' or 'connected' to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

[0013] Artificial Intelligence (AI)

[0014] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define and solve various problems addressed within the field of artificial intelligence. Machine learning is also defined as an algorithm that improves performance on a task through continuous experience.

[0015] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0016] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0017] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0018] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0019] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0020] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when the training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0021] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.

[0023] Robot

[0024] A robot can refer to a machine that automatically processes or operates a given task based on its own capabilities. In particular, a robot that has the ability to perceive its environment, make decisions on its own, and perform actions can be called an intelligent robot.

[0025] Robots can be classified into industrial, medical, household, military, etc., depending on their purpose of use or field.

[0026] A robot can perform various physical movements, such as moving robot joints, by being equipped with a drive unit including actuators or motors. Additionally, a mobile robot may include wheels, brakes, propellers, etc., in the drive unit, enabling it to drive on the ground or fly in the air.

[0028] Self-Driving

[0029] Autonomous driving refers to technology that drives itself, and an autonomous vehicle refers to a vehicle that drives without user intervention or with minimal user intervention.

[0030] For example, autonomous driving can include technologies such as maintaining the driving lane, automatically adjusting speed like adaptive cruise control, automatically driving along a set route, and automatically setting a route and driving once a destination is set.

[0031] Vehicles encompass vehicles equipped solely with internal combustion engines, hybrid vehicles equipped with both internal combustion engines and electric motors, and electric vehicles equipped solely with electric motors, and may include not only automobiles but also trains, motorcycles, etc.

[0032] In this case, an autonomous vehicle can be viewed as a robot with autonomous driving capabilities.

[0034] Extended Reality (XR)

[0035] Extended Reality is a collective term for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR technology provides real-world objects or backgrounds solely as CG images, AR technology provides virtual CG images superimposed on real-world images, and MR technology is a computer graphics technology that mixes and combines virtual objects with the real world.

[0036] MR technology is similar to AR technology in that it displays real-world objects and virtual objects together. However, there is a difference in that while virtual objects in AR technology are used to complement real-world objects, virtual objects and real-world objects are used as equals in MR technology.

[0037] XR technology can be applied to HMDs (Head-Mount Displays), HUDs (Head-Up Displays), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and devices to which XR technology is applied can be called XR devices.

[0039] FIG. 1 shows an AI device (100) according to one embodiment of the present invention.

[0040] In the following, the artificial intelligence device (100) may be referred to as a terminal.

[0041] The AI ​​device (100) can be implemented as a stationary device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0042] Referring to FIG. 1, the artificial intelligence device (100) may include a communication unit (110), an input unit (120), a learning processor (130), a sensing unit (140), an output unit (150), a memory (170), and a processor (180), etc.

[0043] The communication unit (110) can transmit and receive data with external devices, such as other AI devices (100a to 100e) or an AI server (200), using wired or wireless communication technology. For example, the communication unit (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0044] At this time, the communication technologies used by the communication unit (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0045] The input unit (120) can acquire various types of data.

[0046] At this time, the input unit (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input unit for receiving information from a user, etc. Here, the camera or microphone may be treated as a sensor, and the signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

[0047] The input unit (120) can obtain training data for model training and input data to be used when obtaining an output using a training model. The input unit (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input feature points as a preprocessing step for the input data.

[0048] The learning processor (130) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform an action.

[0049] At this time, the learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0050] At this time, the learning processor (130) may include memory integrated into or implemented in the AI ​​device (100). Alternatively, the learning processor (130) may be implemented using memory (170), external memory directly coupled to the AI ​​device (100), or memory maintained in an external device.

[0051] The sensing unit (140) can obtain at least one of internal information of the AI ​​device (100), surrounding environment information of the AI ​​device (100), and user information using various sensors.

[0052] At this time, the sensors included in the sensing unit (140) include a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, a radar, etc.

[0053] The output unit (150) can generate output related to sight, hearing, or touch.

[0054] At this time, the output unit (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0055] The memory (170) can store data that supports various functions of the AI ​​device (100). For example, the memory (170) can store input data, training data, training models, training history, etc. obtained from the input unit (120).

[0056] The processor (180) can determine at least one executable action of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm. The processor (180) can perform the determined action by controlling the components of the AI ​​device (100).

[0057] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the AI ​​device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0058] At this time, if the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0059] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0060] At this time, the processor (180) can obtain intent information corresponding to the user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0061] At this time, at least one of the STT engine or NLP engine may be composed of an artificial neural network in which at least a portion is learned according to a machine learning algorithm. Also, at least one of the STT engine or NLP engine may be learned by a learning processor (130), learned by a learning processor (240) of an AI server (200), or learned through distributed processing thereof.

[0062] The processor (180) may collect history information, including the operation details of the AI ​​device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information may be used to update a learning model.

[0063] The processor (180) can control at least some of the components of the AI ​​device (100) to run an application stored in memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination with each other to run the application.

[0065] FIG. 2 shows an AI server (200) according to one embodiment of the present invention.

[0066] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network. At this time, the AI ​​server (200) may be included as part of the configuration of the AI ​​device (100) and may perform at least part of the AI ​​processing together.

[0067] The AI ​​server (200) may include a communication unit (210), memory (230), a learning processor (240), and a processor (260), etc.

[0068] The communication unit (210) can transmit and receive data with external devices such as AI devices (100).

[0069] The memory (230) may include a model storage unit (231). The model storage unit (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through a learning processor (240).

[0070] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an AI device (100).

[0071] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0072] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0074] FIG. 3 shows an AI system (1) according to one embodiment of the present invention.

[0075] Referring to FIG. 3, the AI ​​system (1) is connected to a cloud network (10) at least one of an AI server (200), a robot (100a), an autonomous vehicle (100b), an XR device (100c), a smartphone (100d), or a home appliance (100e). Here, the robot (100a), the autonomous vehicle (100b), the XR device (100c), the smartphone (100d), or the home appliance (100e) to which AI technology is applied may be referred to as AI devices (100a to 100e).

[0076] A cloud network (10) may mean a network that constitutes part of a cloud computing infrastructure or exists within a cloud computing infrastructure. Here, the cloud network (10) may be configured using a 3G network, a 4G or LTE (Long Term Evolution) network or a 5G network, etc.

[0077] That is, each device (100a to 100e, 200) constituting the AI ​​system (1) can be connected to each other through a cloud network (10). In particular, each device (100a to 100e, 200) may communicate with each other through a base station, but may also communicate directly with each other without going through a base station.

[0078] The AI ​​server (200) may include a server that performs AI processing and a server that performs operations on big data.

[0079] The AI ​​server (200) is connected via a cloud network (10) to at least one of the AI ​​devices constituting the AI ​​system (1), such as a robot (100a), an autonomous vehicle (100b), an XR device (100c), a smartphone (100d), or a home appliance (100e), and can assist in at least some of the AI ​​processing of the connected AI devices (100a to 100e).

[0080] At this time, the AI ​​server (200) can train an artificial neural network according to a machine learning algorithm on behalf of the AI ​​devices (100a to 100e), and can directly store the training model or transmit it to the AI ​​devices (100a to 100e).

[0081] At this time, the AI ​​server (200) receives input data from the AI ​​devices (100a to 100e), infers a result value for the received input data using a learning model, and generates a response or control command based on the inferred result value and transmits it to the AI ​​devices (100a to 100e).

[0082] Alternatively, the AI ​​device (100a to 100e) may use a direct learning model to infer a result value for input data and generate a response or control command based on the inferred result value.

[0083] Hereinafter, various embodiments of AI devices (100a to 100e) to which the above-described technology is applied will be described. Here, the AI ​​devices (100a to 100e) illustrated in FIG. 3 can be seen as specific embodiments of the AI ​​device (100) illustrated in FIG. 1.

[0085] <AI+로봇>

[0086] The robot (100a) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, etc. by applying AI technology.

[0087] The robot (100a) may include a robot control module for controlling operation, and the robot control module may mean a software module or a chip that implements the same in hardware.

[0088] The robot (100a) can use sensor information obtained from various types of sensors to obtain state information of the robot (100a), detect (recognize) surrounding environment and objects, generate map data, determine movement path and driving plan, determine response to user interaction, or determine action.

[0089] Here, the robot (100a) can use sensor information obtained from at least one sensor among lidar, radar, and camera to determine a movement path and driving plan.

[0090] The robot (100a) can perform the above-mentioned actions using a learning model composed of at least one artificial neural network. For example, the robot (100a) can recognize the surrounding environment and objects using the learning model, and can determine actions using the recognized surrounding environment information or object information. Here, the learning model may be learned directly by the robot (100a) or learned from an external device such as an AI server (200).

[0091] At this time, the robot (100a) may perform an operation by generating a result using a direct learning model, but it may also perform an operation by transmitting sensor information to an external device such as an AI server (200) and receiving the result generated accordingly.

[0092] The robot (100a) can determine a movement path and a driving plan using at least one of map data, object information detected from sensor information, or object information obtained from an external device, and control a driving unit to drive the robot (100a) according to the determined movement path and driving plan.

[0093] Map data may include object identification information for various objects placed in the space where the robot (100a) moves. For example, map data may include object identification information for fixed objects such as walls and doors, and movable objects such as flowerpots and desks. In addition, the object identification information may include names, types, distances, locations, etc.

[0094] Additionally, the robot (100a) can perform actions or drive by controlling the drive unit based on the user's control / interaction. At this time, the robot (100a) can acquire intention information of interaction based on the user's actions or voice utterances, and can perform actions by determining a response based on the acquired intention information.

[0096] <AI+자율주행>

[0097] The autonomous vehicle (100b) can be implemented as a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying AI technology.

[0098] The autonomous vehicle (100b) may include an autonomous driving control module for controlling autonomous driving functions, and the autonomous driving control module may refer to a software module or a chip that implements the same in hardware. The autonomous driving control module may be included internally as a component of the autonomous vehicle (100b), but may also be configured and connected as separate hardware externally to the autonomous vehicle (100b).

[0099] The autonomous vehicle (100b) can use sensor information obtained from various types of sensors to obtain state information of the autonomous vehicle (100b), detect (recognize) surrounding environment and objects, generate map data, determine a travel path and driving plan, or determine an action.

[0100] Here, the autonomous vehicle (100b) can use sensor information obtained from at least one sensor among lidar, radar, and camera, just like the robot (100a), to determine a travel path and a driving plan.

[0101] In particular, the autonomous vehicle (100b) can recognize environments or objects in areas where the field of view is obscured or in areas beyond a certain distance by receiving sensor information from external devices, or by receiving information directly recognized from external devices.

[0102] The autonomous vehicle (100b) can perform the above-mentioned operations using a learning model composed of at least one artificial neural network. For example, the autonomous vehicle (100b) can recognize surrounding environments and objects using the learning model, and can determine a driving path using the recognized surrounding environment information or object information. Here, the learning model may be learned directly in the autonomous vehicle (100b) or learned from an external device such as an AI server (200).

[0103] At this time, the autonomous vehicle (100b) may perform operations by generating results using a direct learning model, but may also perform operations by transmitting sensor information to an external device such as an AI server (200) and receiving the results generated accordingly.

[0104] The autonomous vehicle (100b) can determine a movement path and a driving plan using at least one of map data, object information detected from sensor information or object information obtained from an external device, and control a driving unit to drive the autonomous vehicle (100b) according to the determined movement path and driving plan.

[0105] Map data may include object identification information for various objects placed in a space (e.g., a road) where the autonomous vehicle (100b) is driving. For example, the map data may include object identification information for fixed objects such as streetlights, rocks, and buildings, and movable objects such as vehicles and pedestrians. In addition, the object identification information may include names, types, distances, locations, etc.

[0106] Additionally, the autonomous vehicle (100b) can perform operations or drive by controlling the drive unit based on the user's control / interaction. At this time, the autonomous vehicle (100b) can acquire intention information of the interaction based on the user's actions or voice utterances, and can perform operations by determining a response based on the acquired intention information.

[0108] <AI+XR>

[0109] The XR device (100c) can be implemented as a Head-Mount Display (HMD), a Head-Up Display (HUD) equipped in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, digital signage, a vehicle, a stationary robot, or a mobile robot by applying AI technology.

[0110] The XR device (100c) can obtain information about surrounding space or real objects by analyzing 3D point cloud data or image data obtained through various sensors or from an external device to generate position data and attribute data for 3D points, and can render and output an XR object to be output. For example, the XR device (100c) can output an XR object containing additional information about a recognized object by associating it with the recognized object.

[0111] The XR device (100c) can perform the above-mentioned operations using a learning model composed of at least one artificial neural network. For example, the XR device (100c) can recognize real-world objects in 3D point cloud data or image data using the learning model and can provide information corresponding to the recognized real-world objects. Here, the learning model may be learned directly by the XR device (100c) or learned from an external device such as an AI server (200).

[0112] At this time, the XR device (100c) may perform an operation by generating a result using a direct learning model, but it may also perform an operation by transmitting sensor information to an external device such as an AI server (200) and receiving the result generated accordingly.

[0114] <AI+로봇+자율주행>

[0115] The robot (100a) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, etc. by applying AI technology and autonomous driving technology.

[0116] A robot (100a) equipped with AI technology and autonomous driving technology may refer to the robot itself having autonomous driving capabilities, or a robot (100a) that interacts with an autonomous driving vehicle (100b).

[0117] A robot (100a) with autonomous driving capabilities can be collectively referred to as a device that moves on its own along a given path without user control, or moves by determining its own path.

[0118] A robot (100a) and an autonomous vehicle (100b) having autonomous driving capabilities may use a common sensing method to determine one or more of a travel path or a driving plan. For example, a robot (100a) and an autonomous vehicle (100b) having autonomous driving capabilities may determine one or more of a travel path or a driving plan by using information sensed through a lidar, radar, or camera.

[0119] A robot (100a) interacting with an autonomous vehicle (100b) exists separately from the autonomous vehicle (100b) and can perform actions linked to the autonomous driving function inside the autonomous vehicle (100b) or linked to a user riding in the autonomous vehicle (100b).

[0120] At this time, the robot (100a) interacting with the autonomous vehicle (100b) can control or assist the autonomous driving function of the autonomous vehicle (100b) by acquiring sensor information on behalf of the autonomous vehicle (100b) and providing it to the autonomous vehicle (100b), or by acquiring sensor information and generating surrounding environment information or object information and providing it to the autonomous vehicle (100b).

[0121] Alternatively, a robot (100a) interacting with an autonomous vehicle (100b) may monitor a user riding in the autonomous vehicle (100b) or control the functions of the autonomous vehicle (100b) through interaction with the user. For example, if the robot (100a) determines that the driver is drowsy, it may activate the autonomous driving function of the autonomous vehicle (100b) or assist in controlling the drive unit of the autonomous vehicle (100b). Here, the functions of the autonomous vehicle (100b) controlled by the robot (100a) may include not only the autonomous driving function but also functions provided by a navigation system or an audio system equipped inside the autonomous vehicle (100b).

[0122] Alternatively, a robot (100a) interacting with an autonomous vehicle (100b) may provide information to or assist functions to the autonomous vehicle (100b) from outside the autonomous vehicle (100b). For example, the robot (100a) may provide traffic information, such as signal information, to the autonomous vehicle (100b), such as a smart traffic light, or may interact with the autonomous vehicle (100b) to automatically connect an electric charger to the charging port, such as an automatic electric charger for an electric vehicle.

[0124] <AI+로봇+XR>

[0125] The robot (100a) can be implemented as a guide robot, transport robot, cleaning robot, wearable robot, entertainment robot, pet robot, unmanned flying robot, drone, etc. by applying AI technology and XR technology.

[0126] A robot (100a) to which XR technology is applied may refer to a robot that is the subject of control / interaction within an XR image. In this case, the robot (100a) is distinguished from the XR device (100c) and can be interconnected with it.

[0127] When a robot (100a) that is the target of control / interaction within an XR image acquires sensor information from sensors including a camera, the robot (100a) or the XR device (100c) can generate an XR image based on the sensor information, and the XR device (100c) can output the generated XR image. Furthermore, the robot (100a) can operate based on a control signal input through the XR device (100c) or user interaction.

[0128] For example, the user can view an XR image corresponding to the viewpoint of the remotely linked robot (100a) through an external device such as an XR device (100c), and through interaction, can adjust the autonomous driving path of the robot (100a), control its movement or driving, or check information about surrounding objects.

[0130] <AI+자율주행+XR>

[0131] The autonomous vehicle (100b) can be implemented as a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying AI technology and XR technology.

[0132] An autonomous vehicle (100b) equipped with XR technology may refer to an autonomous vehicle equipped with means for providing XR images, or an autonomous vehicle that is the subject of control / interaction within the XR images. In particular, an autonomous vehicle (100b) that is the subject of control / interaction within the XR images may be distinguished from an XR device (100c) and may be interconnected with it.

[0133] An autonomous vehicle (100b) equipped with means for providing XR images can acquire sensor information from sensors including cameras and output an XR image generated based on the acquired sensor information. For example, the autonomous vehicle (100b) can provide an XR object corresponding to a real object or an object in the screen to the occupant by providing an XR image by outputting an XR image with a HUD.

[0134] At this time, when the XR object is displayed on the HUD, at least a portion of the XR object may be displayed so as to overlap with the actual object to which the occupant's gaze is directed. On the other hand, when the XR object is displayed on a display provided inside the autonomous vehicle (100b), at least a portion of the XR object may be displayed so as to overlap with an object on the screen. For example, the autonomous vehicle (100b) may display XR objects corresponding to objects such as lanes, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, buildings, etc.

[0135] When an autonomous vehicle (100b) that is the subject of control / interaction within an XR image acquires sensor information from sensors including a camera, the autonomous vehicle (100b) or the XR device (100c) can generate an XR image based on the sensor information, and the XR device (100c) can output the generated XR image. Furthermore, the autonomous vehicle (100b) can operate based on control signals input through an external device such as the XR device (100c) or user interaction.

[0137] FIG. 4 shows an AI device (100) according to one embodiment of the present invention.

[0138] Descriptions that overlap with Figure 1 are omitted.

[0139] Referring to FIG. 4, the input unit (120) may include a camera (Camera, 121) for inputting a video signal, a microphone (Microphone, 122) for receiving an audio signal, and a user input unit (User Input Unit, 123) for receiving information from a user.

[0140] Voice data or image data collected from the input unit (120) can be analyzed and processed into user control commands.

[0141] The input unit (120) is for inputting video information (or signal), audio information (or signal), data, or information input from a user, and for inputting video information, the AI ​​device (100) may be equipped with one or more cameras (121).

[0142] The camera (121) processes image frames, such as still images or video, obtained by an image sensor in video call mode or shooting mode. The processed image frames may be displayed on a display unit (Display Unit, 151) or stored in memory (170).

[0143] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the AI ​​device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0144] The user input unit (123) is for receiving information from a user, and when information is input through the user input unit (123), the processor (180) can control the operation of the AI ​​device (100) to correspond to the input information.

[0145] The user input unit (123) may include mechanical input means (or mechanical keys, such as buttons, dome switches, jog wheels, jog switches, etc. located on the front / rear or side of the AI ​​device (100)) and touch input means. As an example, the touch input means may consist of a virtual key, soft key, or visual key displayed on a touchscreen through software processing, or a touch key placed on a part other than the touchscreen.

[0146] The sensing unit (140) can be called a sensor unit.

[0147] The output unit (150) may include at least one of a display unit (Display Unit, 151), a sound output unit (Sound Output Unit, 152), a haptic module (Haptic Module, 153), and an optical output unit (Optical Output Unit, 154).

[0148] The display unit (151) displays (outputs) information processed by the AI ​​device (100). For example, the display unit (151) can display information on the execution screen of an application running on the AI ​​device (100), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0149] The display unit (151) can implement a touch screen by forming a layered structure with the touch sensor or by being formed as an integral unit. This touch screen functions as a user input unit (123) that provides an input interface between the AI ​​device (100) and the user, and at the same time, can provide an output interface between the terminal (100) and the user.

[0150] The sound output unit (152) can output audio data received from the communication unit (110) or stored in the memory (170) in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc.

[0151] The sound output unit (152) may include at least one of a receiver, a speaker, and a buzzer.

[0152] The haptic module (153) generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic module (153) can be vibration.

[0153] The light output unit (154) outputs a signal to indicate the occurrence of an event using the light of the light source of the AI ​​device (100). Examples of events occurring in the AI ​​device (100) may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

[0155] FIG. 5 is an operation flowchart illustrating a method of providing information to a user according to one embodiment of the present invention.

[0156] The artificial intelligence server (200) communicates with at least one external device, and the external device may include a mobile terminal, an IoT device, a kiosk, digital signage, etc.

[0157] In the following, external devices may refer to various devices including an artificial intelligence device (100), and may also be referred to as an artificial intelligence device (100) in that it has artificial intelligence functions through an artificial intelligence server (200).

[0158] Additionally, unless otherwise distinguished, the artificial intelligence device (100) is referred to as including the user terminal.

[0159] Referring to FIG. 5, the processor (260) of the artificial intelligence server (200) is Receiving at least one of the user's voice data or the user's terminal usage information (S501).

[0160] At this time, the processor (260) can receive user voice data corresponding to the user's spoken voice from at least one of a user terminal, an artificial intelligence device (100) placed in a service area, or a microphone placed in a service area through the communication unit (210).

[0161] The service area refers to an area where an artificial intelligence server (200) can provide information to a user using at least one artificial intelligence device (100), and may refer to an area where at least one artificial intelligence device (100) is installed.

[0162] At this time, the processor (260) can receive user terminal usage information from the user terminal or the artificial intelligence device (100) currently being used by the user through the communication unit (210).

[0163] Terminal usage information may include usage records of the user's user terminal or artificial intelligence device (100) and user identification information.

[0164] User identification information is information for identifying a specific user among multiple users, and may include user terminal information, user account information, user authentication information, or user recognition information stored in a terminal.

[0165] User recognition information may include user voiceprint recognition information, user facial recognition information, user iris recognition information, or user fingerprint recognition information. Specifically, user recognition information may include features used in voiceprint recognition, facial recognition, iris recognition, or fingerprint recognition.

[0166] At this time, the memory (260) of the artificial intelligence server (200) may store at least one of user recognition information, user identification information, user terminal information, and user account information that can distinguish each user.

[0167] At this time, the communication unit (210) can transmit and receive data with a user terminal, an artificial intelligence device (100), or an artificial intelligence device (100) through or directly via a base station using 5G communication technology.

[0168] And, the processor (260) of the artificial intelligence server (200) is Generate user intent information based on at least one of the user's voice data or the user's terminal usage information. (S503).

[0169] The user's voice data may include queries, requests, or commands explicitly spoken by the user, or conversations in which the user does not explicitly specify queries or requests.

[0170] For example, the user's voice data may include explicit commands or requests, such as "Tell me the way to XX Cafe," or conversations where questions or requests are not explicitly stated, such as "I need to go to XX Cafe."

[0171] If the voice data includes a query, request, or command explicitly spoken by the user, the processor (260) can generate user intent information based on the query, request, or command included in the voice data.

[0172] That is, as in the example above, if the user's voice data includes "Tell me the way to XX Cafe," the processor (260) can generate user intent information based on the voice data, such as "Move to XX Cafe" or "Check the route to XX Cafe."

[0173] At this time, the processor (260) can generate intent information by identifying the intent of an explicit query, request, or command included in the voice data using a natural language processing (NLP) technique or a natural language processing engine.

[0174] If the voice data contains conversation and does not contain explicit queries, requests, or commands, the processor (260) can analyze the conversation content included in the voice data to generate user intent information.

[0175] That is, as in the example above, if the user's voice data includes "I should go to XX Cafe," the processor (260) can generate user intent information based on the voice data, such as "move to XX Cafe" or "check the route to XX Cafe."

[0176] At this time, the processor (260) can extract keywords from the conversation included in the voice data and generate intent information that is highly related to the conversation included in the voice data based on the extracted keywords.

[0177] For example, if the user's voice data includes "I should go to XX Cafe," the processor (260) can extract keywords "XX Cafe" and "go" from the conversation and generate intent information such as "moving to XX Cafe" or "checking the route to XX Cafe" which is highly associated with the extracted keywords.

[0178] The processor (260) can generate intention information from the received voice data using a first intention information generation model.

[0179] The first intention information generation model includes an artificial neural network and can be trained using a machine learning algorithm or a deep learning algorithm.

[0180] At this time, the first intention information generation model can be learned through supervised learning.

[0181] The user's terminal usage information may include at least one of the user's usage records, search records, search results, and search result viewing records regarding the user's terminal or artificial intelligence device (100).

[0182] For example, if a user launches a map application or an internet browser and searches for "XX Cafe," and among the search results, views a search result for "XX Cafe" located in the "YY region," the terminal usage information may include information that the map application or internet browser was used, information that "XX Cafe" was searched for, and information that search results for "XX Cafe" located in the "YY region" were viewed.

[0183] The processor (260) can generate intention information from the received terminal usage information using a second intention information generation model.

[0184] The second intention information generation model includes an artificial neural network and can be trained using a machine learning algorithm or a deep learning algorithm.

[0185] At this time, the second intention information generation model can be learned through supervised learning.

[0186] That is, as in the example above, if the terminal usage information includes information that a map application or an internet browser was used, information that "XX Cafe" was searched for, and information that search results for "XX Cafe" located in "YY area" were viewed, the processor (260) can generate intention information such as "moving to XX Cafe in YY area" or "checking the route to XX Cafe in YY area" from the terminal usage information using a second intention information generation model.

[0187] And, the processor (260) of the artificial intelligence server (200) is Generate user status information (S505).

[0188] User status information may include at least one of the user's location, user's movement path, user's direction of movement, user's direction of gaze, or user's actions.

[0189] The processor (260) can generate user status information based on at least one of voice data, image data, or user terminal status information.

[0190] Image data for a user is image data including a user, and can be generated from at least one of a user terminal, an artificial intelligence device (100) placed in a service area, or a camera placed in a service area.

[0191] User terminal status information may include at least one of location information of the user terminal, operation status information of the user terminal, or input status information of the user terminal.

[0192] The processor (260) can recognize the user included in the image data using a user recognition model.

[0193] A user recognition model is a model that distinguishes and identifies multiple users from one another, and can recognize users using various methods such as facial recognition or gait recognition.

[0194] At this time, the user recognition model may be configured to include an artificial neural network and may be supervised learning using a machine learning algorithm or a deep learning algorithm.

[0195] At this time, the processor (260) can receive terminal usage information through the communication unit (210) and can store user identification information included in the received terminal usage information in the memory (230). In addition, the user identification information can be used for training a user recognition model.

[0196] At this time, the processor (260) or the learning processor (240) can train a user recognition model using image data containing a specific user and user identification information about that user.

[0197] The processor (260) receives image data of a user from at least one of a user terminal, an artificial intelligence device (100) placed in a service area, or a camera placed in a service area through a communication unit (210), and recognizes the user from the received image data to generate user state information including at least one of the user's location, the user's movement path, the user's direction of movement, the user's direction of gaze, or the user's action.

[0198] For example, the processor (260) can recognize the user's face from image data about the user and determine the direction of the user's gaze by determining the direction of the user's face or eyes.

[0199] Additionally, the processor (260) may use multiple image data to generate user state information more accurately. For example, the processor (260) may generate user state information using image data generated from multiple devices, or it may generate user state information using multiple image data generated from a single device.

[0200] For example, the processor (260) can determine the user's location from multiple image data of the user and determine the user's movement path or direction of movement based on changes in the user's location. Additionally, the processor (260) can determine the user's posture from multiple image data of the user and determine the user's movement based on changes in the user's posture.

[0201] The processor (260) can determine the location of a user by using location information for each of the artificial intelligence devices (100) placed within the service area stored in the memory (230) and image data for the user received from the plurality of artificial intelligence devices (100).

[0202] For example, if image data about a user includes depth information, the processor (260) can recognize the user in the image data, calculate the distance between the recognized user and the artificial intelligence device (100) that generated the image data, and determine the location of the user using the calculated distance.

[0203] In one embodiment, the processor (260) receives a user recognition result or a user identification result from the artificial intelligence device (100) and may generate user status information based on the received result.

[0204] That is, each artificial intelligence device (100) recognizes a user using a user recognition model, and the processor (260) receives a user recognition result from the artificial intelligence device (100) instead of voice data or image data, and can generate user status information based on the received recognition result.

[0205] In this case, the processor (260) can transmit at least one of the user recognition model or user identification information used for user recognition to each artificial intelligence device (100) through the communication unit (210).

[0206] And, the processor (260) of the artificial intelligence server (200) is Determine the information providing device based on the user's status information (S507).

[0207] The processor (260) can determine at least one information providing device to provide information to the user among the artificial intelligence devices (100) placed within the service area.

[0208] The information providing device can be selected from among artificial intelligence devices (100) equipped with a speaker or a display.

[0209] The processor (260) can determine an information providing device based on at least one of the user's location, the user's direction of movement, the user's movement path, and the user's direction of gaze, which are included in the user's state information.

[0210] At this time, the processor (260) can calculate the distance from each artificial intelligence device (100) to the user based on the location information for each artificial intelligence device (100) placed within the service area and the location information of the user. Then, it can determine an information providing device among the artificial intelligence devices (100) whose distance to the user is within a certain level.

[0211] For example, the processor (260) can determine an artificial intelligence device (100) located within a first reference distance from the user as an information providing device.

[0212] At this time, the processor (260) can calculate the distance or proximity from each artificial intelligence device (100) to the user's movement path based on location information for each artificial intelligence device (100) placed within the service area and the user's movement path. Then, it can determine an information providing device among the artificial intelligence devices (100) whose distance to the user's movement path is within a certain level.

[0213] For example, the processor (260) can determine an artificial intelligence device (100) adjacent to the user's path as an information providing device.

[0214] At this time, the processor (260) can determine an information providing device among the artificial intelligence devices (100) included in the user's field of vision based on the user's gaze direction.

[0215] For example, the processor (260) may determine an artificial intelligence device (100) as an information providing device, which is located within a certain angle from the direction of the user's gaze and has a distance to the user that is smaller than a second reference distance.

[0216] Additionally, the processor (260) can determine the information providing device by considering not only the user's state information but also the user's intention information.

[0217] At this time, the processor (260) can obtain destination information that the user intends to move to or path information that the user must move to from the user's intention information, and determine an information providing device based on the obtained destination information or path information.

[0218] For example, the processor (260) can determine a path from the current user's location to the destination and from the user's intention information, and determine a device adjacent to the determined path among the artificial intelligence devices (100) as an information providing device.

[0219] In other words, the information providing device may be determined based on the actual user's location, such as the user's location or the path the user is moving, or based on the path the user is expected to travel, such as the path to the destination.

[0220] And, the processor (260) of the artificial intelligence server (200) is Generate output information to be output from the determined information providing device (S509).

[0221] The processor (260) can generate output information based on the user's intention information.

[0222] For example, if the user's intention information is "check the route to XX Cafe," the processor (260) can generate output information including "the route to XX Cafe."

[0223] The processor (260) can generate output information by distinguishing it for each determined information providing device.

[0224] For example, if the output information is "path to XX Cafe," the processor (260) can determine the first output information to be output by the first information providing device as "path from the first information providing device to XX Cafe" and the second output information to be output by the second information providing device as "path from the second information providing device to XX Cafe".

[0225] At this time, the processor (260) can generate output information including information related to the user's intention information.

[0226] For example, if the user's intent information is "check the route to XX Cafe," the processor (260) can generate output information including "the route to XX Cafe" and "the business hours of XX Cafe."

[0227] At this time, the processor (260) can generate output information including advertising information related to the user's intention information.

[0228] For example, if the user intent information is "check route to XX Cafe," the processor (260) can generate output information including advertisements for other cafes or advertisements for coffee-related products.

[0229] At this time, the processor (260) may generate output information including information designating the user to receive the information.

[0230] And, the processor (260) of the artificial intelligence server (200) is Transmit a control signal that outputs generated output information to the determined information providing device. (S511).

[0231] Output information can be generated separately for each information providing device, and the processor (260) can transmit a control signal that outputs output information corresponding to each information providing device.

[0232] The control signal may include at least one of a control signal that outputs output information as an image through the display unit of the information providing device or a control signal that outputs output information as sound through the speaker of the information providing device.

[0233] The above description is merely an explanation focusing on the relationship between the artificial intelligence server (200) and one user, and the artificial intelligence server (200) can individually generate intention information and state information for each of the multiple users and determine the information providing device and output information for each of the multiple users.

[0234] Accordingly, the artificial intelligence server (200) can provide personalized information to the user using multiple artificial intelligence devices (100).

[0235] The steps (S501 to S511) illustrated in FIG. 5 can be performed repeatedly, and thus the artificial intelligence server (200) can periodically or in real time update the user's intention information or the user's status information, and if the user's intention information changes or the user's status information changes, the output information or the information providing device can be changed accordingly.

[0236] For example, when the user is looking at the first external device, the processor (260) determines the first external device as an information providing device, and when the user changes their gaze from the first external device to the second external device, the processor (260) can change the information providing device from the first external device to the second external device.

[0237] For example, when a user is adjacent to a first external device, the processor (260) determines the first external device as an information providing device, and when the user moves near a second external device, the processor (260) can change the information providing device from the first external device to the second external device.

[0239] FIG. 6 is a diagram showing an example of generating user intention information according to one embodiment of the present invention.

[0240] Referring to FIG. 6, the artificial intelligence server (601) can communicate with an artificial intelligence device (611) or a user terminal (612) placed within the service area.

[0241] The artificial intelligence device (611) may refer to a kiosk, digital signage, home appliance, etc.

[0242] The user terminal (612) may mean a smartphone.

[0243] The first user (621) can speak (622) "Way to X1 cafe?" to the artificial intelligence device (611), the second user (631) can speak (632) "Where is X2 cafe?" to the artificial intelligence device (611), and the third user (641) can search for information about "X3 restaurant" (642) using the user terminal (612).

[0244] The artificial intelligence server (601) can receive voice data corresponding to the voice (622) spoken by the first user (621) and voice data corresponding to the voice (632) spoken by the second user (631) from the artificial intelligence device (611), and can receive terminal usage information corresponding to the information (642) searched by the third user (641) from the user terminal (612).

[0245] And, based on the received voice data and terminal usage information, the artificial intelligence server (601) can determine the intention information of the first user (621) as "checking the route to X1 Cafe," the intention information of the second user (631) as "checking the route to X2 Cafe," and the intention information of the third user (641) as "checking the route to X3 Restaurant."

[0246] And, as described above, the artificial intelligence server (601) can determine an information providing device among a plurality of artificial intelligence devices or user terminals placed in a service area, generate output information to be output from the information providing device, and transmit a control signal to output the generated output information to the information providing device.

[0247] In FIG. 6, the user's artificial intelligence device (611) generates the user's voice data and the user terminal (612) generates the user's terminal usage information, but the present invention is not limited thereto.

[0248] That is, when a user uses the artificial intelligence device (611), the artificial intelligence device (611) can generate terminal usage information and transmit it to the artificial intelligence server (601). Additionally, when a user makes a voice query using the user terminal (612), the user terminal (612) can generate voice data corresponding to the user's spoken voice and transmit it to the artificial intelligence server (601). Furthermore, the artificial intelligence server (601) can generate intention information for each user using the received terminal usage information or voice data.

[0250] FIG. 7 is an operation flowchart illustrating a method of providing information to a user according to one embodiment of the present invention.

[0251] Descriptions that overlap with Fig. 5 are omitted.

[0252] Referring to FIG. 7, the processor (260) of the artificial intelligence server (200) is Receiving at least one of the user's voice data or the user's terminal usage information (S701). This corresponds to step (S501) of Fig. 5.

[0253] And, the processor (260) of the artificial intelligence server (200) is Generate user intent information based on at least one of the user's voice data or the user's terminal usage information. (S703). This corresponds to step (S503) of Fig. 5.

[0254] And, the processor (260) of the artificial intelligence server (200) is Generate user status information (S705). This corresponds to step (S505) of Fig. 5.

[0255] And, the processor (260) of the artificial intelligence server (200) is Determine the information providing device based on the user's status information (S707). This corresponds to step (S507) of Fig. 5.

[0256] And, the processor (260) of the artificial intelligence server (200) is Generate output information to be output from the determined information providing device (S709). This corresponds to step (S509) of Fig. 5.

[0257] And, the processor (260) of the artificial intelligence server (200) is Transmit a control signal that outputs generated output information to the determined information providing device.(S711). This corresponds to step (S511) of Fig. 5.

[0258] And, the processor (260) of the artificial intelligence server (200) is Update user status information (S713).

[0259] The processor (260) receives at least one of voice data, image data, or terminal usage information from at least one of the artificial intelligence devices (100) or user terminals placed within the service area, and can update the user's status information using the received data.

[0260] For example, the processor (260) can identify a user in the image data and determine the location of the identified user accordingly.

[0261] And, the processor (260) of the artificial intelligence server (200) is Determine whether to terminate the provision of information (S715).

[0262] The processor (260) can determine whether to terminate the provision of information based on the updated user status information.

[0263] At this time, the processor (260) can determine whether to terminate the provision of information by determining whether the user has completed an action corresponding to the user's intention information, whether the user's intention information has changed, or whether the user has left the service area.

[0264] For example, if the user's intention information is "to move to location AA" or "to check the route to location AA," the processor (260) may decide to terminate the provision of information when it determines that the user has arrived at "location AA" as a destination.

[0265] For example, if the user's intention information is "move to AA location," the processor (260) may decide to terminate the provision of information regarding the intention information if it determines that the user's intention information has been changed to something else. In this case, the processor (260) may proceed again from step (S701) to generate new user intention information and provide information to the user accordingly.

[0266] For example, if the user leaves the service area, the processor (260) may decide to stop providing information.

[0267] If it is determined that the provision of information is terminated as a result of the judgment in step (S715), the processor (260) terminates the provision of information to the user.

[0268] If it is determined that the provision of information is not terminated as a result of the judgment in step (S715), the processor (260) can return to the step (S707) of determining the information providing device and continue to provide information to the user.

[0270] FIGS. 8 to 11 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention.

[0271] FIG. 8 illustrates a situation in which a first user (801) asks a first artificial intelligence device (821) for directions, and a second user (802) asks a second artificial intelligence device (822) for directions. Here, the artificial intelligence devices (821, 822, 823, 824) may be digital signage, kiosks, etc.

[0272] Referring to FIG. 8, the first user (801) can ask the first artificial intelligence device (821) for the way to X2 cafe (842) by speaking (811) such as "Way to X2 cafe?". Similarly, the second user (802) can ask the second artificial intelligence device (822) for the way to X1 cafe (841) by speaking (812) such as "Where is X1 cafe?".

[0273] Although not illustrated in FIG. 8, the artificial intelligence server (200) can receive voice data for the first user (801) from the first artificial intelligence device (821) and voice data for the second user (802) from the second artificial intelligence device (822), thereby determining the first intention information of the first user (801) as "checking the route to X1 Cafe (841)" and the second intention information of the second user (802) as "checking the route to X2 Cafe (842)".

[0274] And, the artificial intelligence server (200) can determine the first artificial intelligence device (821) adjacent to the first user (801) as an information providing device for the first user (801), and the second artificial intelligence device (822) adjacent to the second user (802) as an information providing device for the second user (802).

[0275] And, the artificial intelligence server (200) can transmit a control signal that outputs output information corresponding to each information providing device.

[0276] That is, the first artificial intelligence device (821) can output a message including "X2 cafe ↑" meaning to proceed forward to move to X2 cafe (842) as output information corresponding to the first intention information of the first user (801) (831).

[0277] Additionally, the second artificial intelligence device (822) can output a message including "X1 cafe ↑" meaning to proceed forward to move to X1 cafe (841) as output information corresponding to the second intention information of the second user (802) (832).

[0278] However, the output message may be displayed in text format through the display, but it may also be displayed in image format, or output as voice through the speaker.

[0279] At this time, the artificial intelligence server (200) may not output output information for the first user (801) and the second user (802) to the third artificial intelligence device (823) and the fourth artificial intelligence device (824) that are not adjacent to the first user (801) and the second user (802) (833, 834).

[0280] FIG. 9 shows a situation after the situation of FIG. 8 in which the first user (801) and the second user (802) move forward and are adjacent to the third artificial intelligence device (823).

[0281] Referring to FIG. 9, the artificial intelligence server (200) can determine that the first user (801) and the second user (802) are far from the first artificial intelligence device (821) and the second artificial intelligence device (822) through image data or voice data generated by the first artificial intelligence device (821) and the second artificial intelligence device (822). Similarly, the artificial intelligence server (200) can determine that the first user (801) and the second user (802) are adjacent to the third artificial intelligence device (823) through image data or voice data generated by the third artificial intelligence device (823).

[0282] And, the artificial intelligence server (200) can determine the third artificial intelligence device (823) as an information providing device for the first user (801) and the second user (802).

[0283] And, the artificial intelligence server (200) can transmit a control signal to the third artificial intelligence device (823) to output information for the first user (801) and output information for the second user (802).

[0284] That is, the third artificial intelligence device (823) can output a message including "X1 cafe ↑" and "X2 cafe ↑" as output information corresponding to the first intention information of the first user (801) and output information corresponding to the intention information of the second user (802) (933).

[0285] At this time, the artificial intelligence server (200) can control the first artificial intelligence device (821), the second artificial intelligence device (822), and the fourth artificial intelligence device (824) that are not adjacent to the first user (801) and the second user (802) so as not to output information about the first user (801) and the second user (802) (931, 932, 934).

[0286] FIG. 10 shows a situation after the situation of FIG. 9 in which the first user (801) moves forward and is adjacent to the fourth artificial intelligence device (824), and the second user (802) moves forward but is still adjacent to the third artificial intelligence device (823).

[0287] Referring to FIG. 10, the artificial intelligence server (200) can determine that the first user (801) has moved away from the third artificial intelligence device (823) through image data or voice data generated by the third artificial intelligence device (823). Likewise, the artificial intelligence server (200) can determine that the first user (801) is adjacent to the fourth artificial intelligence device (824) through image data or voice data generated by the fourth artificial intelligence device (824).

[0288] And, the artificial intelligence server (200) can maintain the third artificial intelligence device (823) as an information providing device for the second user (802) and determine the fourth artificial intelligence device (824) as an information providing device for the first user (801).

[0289] And, the artificial intelligence server (200) can transmit a control signal to output information for the first user (801) to the fourth artificial intelligence device (824) and transmit a control signal to output information for the second user (802) to the third artificial intelligence device (823).

[0290] That is, the fourth artificial intelligence device (824) can output a message including "X2 cafe →" as output information corresponding to the first intention information of the first user (801) (1034).

[0291] Additionally, the third artificial intelligence device (823) can output a message including "X1 cafe ↑" as output information corresponding to the intention information of the second user (802) (1033).

[0292] At this time, the artificial intelligence server (200) can control the first artificial intelligence device (821) and the second artificial intelligence device (822) that are not adjacent to the first user (801) and the second user (802) so that output information about the first user (801) and the second user (802) is not output (1031, 1032).

[0293] FIG. 11 shows a situation after the situation of FIG. 10 in which the first user (801) moves to the right but is still adjacent to the fourth artificial intelligence device (824), and the second user (802) moves forward but is adjacent to the fourth artificial intelligence device (824).

[0294] Referring to FIG. 11, the artificial intelligence server (200) can determine that the second user (802) has moved away from the third artificial intelligence device (823) through image data or voice data generated by the third artificial intelligence device (823). Similarly, the artificial intelligence server (200) can determine that the first user (801) and the second user (802) are adjacent to the fourth artificial intelligence device (824) through image data or voice data generated by the fourth artificial intelligence device (824).

[0295] And, the artificial intelligence server (200) can determine the fourth artificial intelligence device (824) as an information providing device for the first user (801) and the second user (802).

[0296] And, the artificial intelligence server (200) can transmit a control signal to the fourth artificial intelligence device (824) to output information for the first user (801) and output information for the second user (802).

[0297] That is, the fourth artificial intelligence device (824) can output a message including "X1 cafe ←" and "X2 cafe →" as output information corresponding to the first intention information of the first user (801) and output information corresponding to the intention information of the second user (802) (1134).

[0298] At this time, the artificial intelligence server (200) can control the first artificial intelligence device (821), the second artificial intelligence device (822), and the third artificial intelligence device (823) that are not adjacent to the first user (801) and the second user (802) so that output information regarding the first user (801) and the second user (802) is not output (1131, 1132, 1133).

[0299] As illustrated in FIGS. 8 to 11, the artificial intelligence server (200) can generate intention information for each user, determine an information providing device for each user, and transmit a control signal to output information to the determined information providing device to provide information suitable for each user.

[0300] In FIGS. 8 to 11, the information providing device was determined based only on the distance between the user and the artificial intelligence device, but as described above, the present invention is not limited thereto.

[0301] In addition, in FIGS. 8 to 11, the information providing device is determined from among artificial intelligence devices, but a user terminal can also be determined as the information providing device. That is, the artificial intelligence server (200) can provide information to the user using a user terminal.

[0303] FIGS. 12 and 13 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention.

[0304] FIG. 12 illustrates a situation in which a user (1201) asks a first artificial intelligence device (1221) for directions. Here, the first artificial intelligence device (1221) may be a TV. And, the second artificial intelligence device (1222) may be a vehicle.

[0305] Referring to FIG. 12, the user (1201) can ask for the way to X3 restaurant (1241) by uttering (1211) such as "Way to X3 restaurant?".

[0306] Although not illustrated in FIG. 12, the artificial intelligence server (200) can determine the user's (1201) intention information as "confirming the route to X3 Restaurant (1231)" by receiving voice data about the user (1201) from the first artificial intelligence device (1221).

[0307] In addition, the artificial intelligence server (200) may obtain user schedule information from a user terminal, etc., even if the user (1201) does not speak directly, and determine intention information based on the obtained user schedule information.

[0308] And, the artificial intelligence server (200) determines the first artificial intelligence device (1221) adjacent to the user (1201) as an information providing device for the user (1201), and the artificial intelligence server (200) can transmit a control signal that outputs output information corresponding to the information providing device.

[0309] That is, the first artificial intelligence device (1221) can output a map showing a route to X3 Restaurant (1241) as output information corresponding to the user's (1201) intention information (1231).

[0310] At this time, the artificial intelligence server (200) can control the second artificial intelligence device (1222) that is not adjacent to the user (1201) so that output information about the user (1201) is not output (1232).

[0311] FIG. 13 shows a situation in which the user (1201) boards the second artificial intelligence device (1222) after the situation of FIG. 12.

[0312] Referring to FIG. 13, the artificial intelligence server (200) can determine that the user (1201) has moved away from the first artificial intelligence device (1221) through image data or voice data generated by the first artificial intelligence device (1221).

[0313] Likewise, the artificial intelligence server (200) can determine that the user (1201) has boarded the second artificial intelligence device (1222) through image data or voice data generated by the second artificial intelligence device (1222). Alternatively, the artificial intelligence server (200) can determine that the user (1201) has boarded the second artificial intelligence device (1222) through terminal usage information or status information (e.g., ignition information, door opening / closing information) regarding the second artificial intelligence device (1222).

[0314] And, the artificial intelligence server (200) determines the second artificial intelligence device (1222) as an information providing device for the user (1201), and the artificial intelligence server (200) can transmit a control signal to the second artificial intelligence device (1222) to output information about the user (1201).

[0315] That is, the second artificial intelligence device (1222) can output at least one of a map showing a route to X3 Restaurant (1241) or navigation guidance information as output information corresponding to the user's (1201) intention information (1332).

[0316] At this time, the artificial intelligence server (200) can control the first artificial intelligence device (1221) that is not adjacent to the user (1201) so that output information about the user (1201) is not output (1331).

[0318] FIGS. 14 to 16 are drawings illustrating examples of providing information to a user according to an embodiment of the present invention.

[0319] FIG. 14 illustrates a situation in which a first artificial intelligence device (1421) receives a voice in which a user (1401) speaks about what to cook. This includes not only a situation in which the user (1401) speaks with the intention of inputting it into the first artificial intelligence device (1421), but also a situation in which the user (1401) speaks to himself. Here, the first artificial intelligence device (1421) may be a TV. And, the second artificial intelligence device (1422) may be a refrigerator.

[0320] Referring to FIG. 14, the user (1201) can utter (1411) "Let's have Bulgogi."

[0321] Although not illustrated in FIG. 14, the artificial intelligence server (200) can determine the user's (1401) intention information as "eat bulgogi" by receiving voice data about the user (1401) from the first artificial intelligence device (1421).

[0322] In addition, unlike Fig. 14, the artificial intelligence server (200) receives user terminal usage information even if the user (1401) does not speak directly, and when the user searches for a bulgogi recipe using the user terminal, the server may determine intention information using the received terminal usage information.

[0323] And, the artificial intelligence server (200) determines the first artificial intelligence device (1421) adjacent to the user (1401) as an information providing device for the user (1401), and the artificial intelligence server (200) can transmit a control signal that outputs output information corresponding to the information providing device.

[0324] That is, the first artificial intelligence device (1421) can output at least one of the following as output information corresponding to the user's (1401) intention information: a bulgogi recipe or information about bulgogi ingredients stored in a refrigerator (1431).

[0325] In this case, the output information may include information that there is no beef in the refrigerator to use as an ingredient for bulgogi.

[0326] In particular, the first artificial intelligence device (1421) can output output information to only a part of the screen so that the user (1401) can continue to view the content that was previously being output. Additionally, the first artificial intelligence device (1421) can output the output information with the sound included in it muted so as not to interfere with the viewing of the content that was previously being output.

[0327] At this time, the artificial intelligence server (200) can control the second artificial intelligence device (1422) that is not adjacent to the user (1401) so that output information about the user (1401) is not output (1432).

[0328] FIG. 15 illustrates a situation where the user (1401) moves to the second artificial intelligence device (1422) after the situation of FIG. 14. For example, the user (1401) can check the inside of the refrigerator, which is the second artificial intelligence device (1422).

[0329] Referring to FIG. 15, the artificial intelligence server (200) can determine that the user (1401) has moved away from the first artificial intelligence device (1421) through image data or voice data generated by the first artificial intelligence device (1421).

[0330] Likewise, the artificial intelligence server (200) can determine that the user (1401) is adjacent to the second artificial intelligence device (1422) through image data or voice data generated by the second artificial intelligence device (1422). Alternatively, the artificial intelligence server (200) can determine that the user (1401) is adjacent to the second artificial intelligence device (1422) through terminal usage information or status information (e.g., door opening / closing information) regarding the second artificial intelligence device (1422).

[0331] And, the artificial intelligence server (200) determines the second artificial intelligence device (1422) as an information providing device for the user (1401), and the artificial intelligence server (200) can transmit a control signal to the second artificial intelligence device (1422) to output information about the user (1401).

[0332] That is, the second artificial intelligence device (1422) can output at least one of the following as output information corresponding to the user's (1401) intention information: a bulgogi recipe or information about bulgogi ingredients stored in a refrigerator (1532).

[0333] In this case, the output information may include information that there is no beef in the refrigerator to use as an ingredient for bulgogi.

[0334] At this time, the artificial intelligence server (200) can control the first artificial intelligence device (1421) that is not adjacent to the user (1401) so that output information about the user (1401) is not output (1531).

[0335] FIG. 16 illustrates a situation where, after the situation of FIG. 15, the user (1401) moves again near the first artificial intelligence device (1121). For example, the user (1401) can move near the TV, which is the first artificial intelligence device (1421), and watch TV.

[0336] Referring to FIG. 16, the user (1401) can utter (1611) such as "Well, what should I eat?".

[0337] Although not illustrated in FIG. 16, the artificial intelligence server (200) can determine the user's (1401) intention information as "deciding on food" by receiving voice data about the user (1401) from the first artificial intelligence device (1421). That is, the artificial intelligence server (200) can update the user's (1401) intention information.

[0338] In addition, unlike FIG. 16, the artificial intelligence server (200) receives user terminal usage information even if the user (1401) does not speak directly, and when the user searches for recommended food or uses a food delivery application using the user terminal, the server may update intention information using the received terminal usage information.

[0339] At this time, the artificial intelligence server (200) can determine that the user (1401) has moved away from the second artificial intelligence device (1422) through image data or voice data generated by the second artificial intelligence device (1422).

[0340] Likewise, the artificial intelligence server (200) can determine that the user (1401) is adjacent to the first artificial intelligence device (1421) through image data or voice data generated by the first artificial intelligence device (1421).

[0341] And, the artificial intelligence server (200) determines the first artificial intelligence device (1421) as an information providing device for the user (1401) and can transmit a control signal to the first artificial intelligence device (1421) to output information about the user (1401).

[0342] That is, the first artificial intelligence device (1422) can output at least one of the recommended foods that can be made using ingredients stored in the refrigerator or foods that can be delivered as output information corresponding to the user's (1401) intention information (1631).

[0343] At this time, the artificial intelligence server (200) can control the second artificial intelligence device (1422) that is not adjacent to the user (1401) so that output information about the user (1401) is not output (1632).

[0344] In one embodiment, when the user (1401) shifts their gaze to the user terminal, the artificial intelligence server (200) may control the first artificial intelligence device (1421) to stop outputting output information and control the output information to be output to the user terminal.

[0346] FIG. 17 is a drawing showing an example of providing information to a user according to one embodiment of the present invention.

[0347] Referring to FIG. 17, the first artificial intelligence device (1703_1) can acquire voice data corresponding to the voice spoken by the user (S1711) and transmit the acquired voice data to the artificial intelligence server (1701) (S1713).

[0348] And, the artificial intelligence server (1701) generates user intent information using the received voice data (S1715).

[0349] And, each artificial intelligence device (1703_1, 1703_2, ..., 1703_n) can acquire image data (S1717) and transmit the acquired image data to the artificial intelligence server (1701) (S1719).

[0350] And, the artificial intelligence server (1701) can generate user state information using the received image data (S1721), determine the second artificial intelligence device (1703_2) as the information providing device based on the generated user state information (S1723), generate output information based on the generated user intention information and the information providing device (S1725), and transmit an output signal that outputs the generated output information to the second artificial intelligence device (1703_2) as the information providing device (S1727).

[0351] And, the second artificial intelligence device (1703_2) as an information providing device can output output information based on the received output signal (S1729).

[0352] In the example of FIG. 17, each artificial intelligence device (1703_1, 1703_2, ..., 1703_n) simply acquires image data about a user, and the artificial intelligence server (1701) recognizes the user from the image data received from each artificial intelligence device (1703_1, 1703_2, ..., 1703_n) and generates user state information.

[0354] FIG. 18 is a drawing showing an example of providing information to a user according to one embodiment of the present invention.

[0355] Referring to FIG. 18, the first artificial intelligence device (1803_1) can acquire voice data corresponding to the voice spoken by the user (S1811) and transmit the acquired voice data to the artificial intelligence server (1801) (S1813).

[0356] And, the artificial intelligence server (1801) generates user intent information using the received voice data (S1815).

[0357] And, each artificial intelligence device (1803_1, 1803_2, ..., 1803_n) can generate user recognition information (S1817) and transmit the generated user recognition information to the artificial intelligence server (1801) (S1819).

[0358] At this time, each artificial intelligence device (1803_1, 1803_2, ..., 1803_n) can acquire image data and recognize a user from the acquired image data to generate user recognition information.

[0359] And, the artificial intelligence server (1801) can generate user state information using user recognition information (S1821), determine the second artificial intelligence device (1803_2) as an information providing device based on the generated user state information (S1823), generate output information based on the generated user intention information and the information providing device (S1825), and transmit an output signal that outputs the generated output information to the second artificial intelligence device (1803_2) as an information providing device (S1827).

[0360] And, the second artificial intelligence device (1803_2) as an information providing device can output output information based on the received output signal (S1829).

[0361] In the example of FIG. 18, each artificial intelligence device (1803_1, 1803_2, ..., 1803_n) acquires image data of a user and recognizes the user from the acquired image data to generate user recognition information. Then, the artificial intelligence server (1801) generates user state information using the user recognition information generated by each artificial intelligence device (1803_1, 1803_2, ..., 1803_n).

[0363] The present invention described above can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

Claim 1 In an artificial intelligence server that provides information to a user, a communication unit that communicates with a plurality of artificial intelligence devices placed within a service area; An artificial intelligence server comprising a processor that receives at least one of the user's voice data or the user's terminal usage information from at least one of the plurality of artificial intelligence devices, generates the user's first intention information using a first intention information generation model based on the received voice data, generates the user's second intention information using a second intention information generation model based on the received terminal usage information, determines the user's intention information based on at least one of the first intention information and the second intention information, generates the user's state information including a change in the user's location using the plurality of artificial intelligence devices or the terminal, obtains destination information that the user intends to move to or path information that the user must move to from the user's intention information, determines at least one information providing device among the terminal and the plurality of artificial intelligence devices based on the obtained destination information or path information, the generated user's state information and the intention information, generates output information to be output from the determined at least one information providing device based on the user's intention information, and transmits a control signal to output the generated output information to the determined at least one information providing device. Claim 2 The artificial intelligence server according to claim 1, wherein the terminal usage information includes a user's usage record for the artificial intelligence device and user identification information identifying the user. Claim 3 The artificial intelligence server of claim 1, wherein the processor generates intent information from the received voice data using natural language processing (NLP) techniques. Claim 4 The artificial intelligence server according to claim 1, wherein the processor generates the intention information from the received terminal usage record using an intention information generation model, and the intention information generation model includes an artificial neural network and is learned using a machine learning algorithm or a deep learning algorithm. Claim 5 An artificial intelligence server according to claim 1, wherein the state information comprises at least one of the user's location, the user's movement path based on a change in location, the user's direction of movement, the user's direction of gaze, or an action based on a change in the user's posture. Claim 6 An artificial intelligence server according to claim 5, wherein the processor receives voice data for the user, image data for the user, or terminal status information of the user from the plurality of artificial intelligence devices, and generates the status information based on at least one of the received voice data, the received image data, or the received terminal status information. Claim 7 ◈Claim 7 was abandoned upon payment of the registration fee.◈ In Claim 6, the processor recognizes the user in the received image data using a user recognition model and generates state information using the recognition result, and the user recognition model includes an artificial neural network and is trained using a machine learning algorithm or a deep learning algorithm, an artificial intelligence server. Claim 8 ◈Claim 8 was abandoned upon payment of the registration fee.◈ In Claim 5, the processor receives user recognition information generated for the user from the plurality of artificial intelligence devices and generates state information using the received user recognition information, an artificial intelligence server. Claim 9 ◈Claim 9 was abandoned upon payment of the registration fee.◈ In Claim 5, the processor determines an artificial intelligence server, wherein, among the plurality of artificial intelligence devices, a device located within a first reference distance from the user is the information providing device. Claim 10 ◈Claim 10 was abandoned upon payment of the registration fee.◈ In Claim 9, the processor determines as the information providing device a device located within a certain angle from the direction of the user's gaze among the plurality of artificial intelligence devices and located within a second reference distance from the user. Claim 11 An artificial intelligence server according to claim 1, wherein the processor periodically or in real time updates at least one of the intention information or the state information, and when at least one of the intention information or the state information changes, updates at least one of the information providing device or the output information. Claim 12 A method for providing information to a user from an artificial intelligence server comprises: receiving at least one of the user's voice data or the user's terminal usage information from at least one of a plurality of artificial intelligence devices disposed within a service area by a processor of the artificial intelligence server; generating the user's first intention information using a first intention information generation model based on the received voice data by the processor; generating the user's second intention information using a second intention information generation model based on the received terminal usage information by the processor by the processor; determining the user's intention information based on at least one of the first intention information and the second intention information by the processor; generating the user's state information using the plurality of artificial intelligence devices by the processor by the processor; obtaining destination information of the user's intended movement or path information of the user's movement from the user's intention information by the processor, and determining at least one information providing device among the user's terminal and the plurality of artificial intelligence devices based on the obtained destination information or path information, the generated user's state information, and the intention information by the processor; and generating output information to be output from the determined at least one information providing device based on the user's intention information by the processor. A method comprising the step of transmitting a control signal to output the generated output information to at least one information providing device determined by the processor. Claim 13 A recording medium having a program recorded thereon for performing a method of providing information to a user from an artificial intelligence server, wherein the method of providing information to the user comprises: a step of receiving at least one of the user’s voice data or the user’s terminal usage information from at least one of a plurality of artificial intelligence devices disposed within a service area by a processor of the artificial intelligence server; a step of generating the user’s first intention information using a first intention information generation model based on the received voice data by the processor; a step of generating the user’s second intention information using a second intention information generation model based on the received terminal usage information by the processor by the processor; a step of determining the user’s intention information based on at least one of the first intention information and the second intention information by the processor by the processor; a step of generating the user’s state information using the plurality of artificial intelligence devices by the processor; and a step of obtaining destination information that the user intends to move to or path information that the user must move to from the user’s intention information by the processor, and determining the user’s terminal and at least one information providing device among the plurality of artificial intelligence devices based on the obtained destination information or path information, the generated user’s state information, and the intention information. A recording medium comprising: a step of generating output information to be output from at least one information providing device determined by the processor based on the user's intention information; and a step of transmitting a control signal to output the generated output information to at least one information providing device determined by the processor. Claim 14 delete Claim 15 The artificial intelligence server of claim 1, wherein the processor generates output information separately for each information providing device and transmits a control signal for outputting output information corresponding to each information providing device.

Images