Artificial intelligence device and operation method thereof

Abstract

An artificial intelligence device according to one embodiment of the present invention comprises: a memory for storing a first encoder outputting an embedding vector from data of a first type representing lifestyle patterns of a user, the first encoder being an encoder trained through contrastive learning with a second encoder outputting an embedding vector from data of a second type representing the lifestyle patterns of the user; and one or more processors for acquiring a log data set of the first type representing the lifestyle patterns of the user during a specific period of time, acquiring embedding vectors from the acquired log data set through the first encoder, generating embedding vector distribution information on the basis of the acquired embedding vectors, predicting embedding vectors for a specific point in time or a specific time slot on the basis of the embedding vector distribution information, acquiring context data matching the predicted embedding vectors, and providing a service based on the acquired context data.

Description

Artificial intelligence device and its operating method

[0001] The present invention relates to an artificial intelligence server, and more particularly, to an artificial intelligence device that predicts situations based on past life patterns.

[0002] A multimodal model represents a model that processes multiple types of input data and produces multiple types of output.

[0003] Multimodal models can primarily handle multiple data types simultaneously, including text, images, and audio. For example, they can simultaneously understand images and captions to generate descriptions, or process text and speech together to improve natural language understanding and speech recognition.

[0004] However, a methodology for comprehensively understanding a specific situation by utilizing multi-modal data with various forms and characteristics, such as images, sounds, home appliance usage logs, sensors, and the Internet of Things (IoT), and converting the understanding of a specific situation into continuous data to predict patterns has not yet been established.

[0005] In addition, although it may have been possible to predict situations in a text domain that language models understand, there was no method to understand changes in situations identified from multi-modal data without text information and to predict patterns of situation changes.

[0006] An object of the present disclosure may be to provide an artificial intelligence device that can predict future situations by analyzing a user's past behavioral patterns.

[0007] The purpose of the present disclosure may be to accurately predict situation changes from various types of multimodal data.

[0008] An artificial intelligence device according to one embodiment of the present disclosure may include a memory storing a first encoder that outputs an embedding vector from a first type of data representing a user's lifestyle pattern, the first encoder being an encoder learned through contrastive learning with a second encoder that outputs an embedding vector from a second type of data representing the user's lifestyle pattern; and one or more processors that obtain a log data set of the first type representing the user's lifestyle pattern for a specific period of time, obtain embedding vectors from the obtained log data set through the first encoder, generate embedding vector distribution information based on the obtained embedding vectors, predict an embedding vector at a specific point in time or a specific time period based on the embedding vector distribution information, obtain context data matching the predicted embedding vector, and provide a service based on the obtained context data.

[0009] An operating method of an artificial intelligence device according to an embodiment of the present disclosure may include: obtaining a first type of log data set representing a user's lifestyle pattern for a specific period of time; obtaining embedding vectors from the obtained log data set through a first encoder, the first encoder being an encoder learned through contrastive learning that outputs an embedding vector from the first type of data representing the user's lifestyle pattern and outputs an embedding vector from a second type of data representing the user's lifestyle pattern; generating embedding vector distribution information based on the obtained embedding vectors; predicting an embedding vector at a specific point in time or a specific time period based on the embedding vector distribution information; obtaining context data matching the predicted embedding vector; and providing a service based on the obtained context data.

[0010] A computer-readable program for performing an operating method of an artificial intelligence device according to an embodiment of the present disclosure is recorded on a recording medium, wherein the operating method of the artificial intelligence device may include the steps of: obtaining a first type of log data set representing a user's lifestyle pattern for a specific period of time; obtaining embedding vectors from the obtained log data set through a first encoder; the first encoder being an encoder learned through contrastive learning that outputs an embedding vector from the first type of data representing the user's lifestyle pattern and outputs an embedding vector from a second type of data representing the user's lifestyle pattern; generating embedding vector distribution information based on the obtained embedding vectors; predicting an embedding vector at a specific point in time or a specific time period based on the embedding vector distribution information; obtaining context data matching the predicted embedding vector; and providing a service based on the obtained context data.

[0011] According to embodiments of the present disclosure, multimodal data with various formats and characteristics can be utilized to comprehensively understand a situation, and by processing this understood situation as time-series data, user behavior patterns can be predicted. Accordingly, customized services tailored to the user's personalized lifestyle can be provided, significantly enhancing user convenience.

[0012] According to an embodiment of the present disclosure, a user's behavioral pattern can be predicted more accurately by utilizing multimodal data having various forms and characteristics.

[0013] Figure 1 illustrates an AI device according to one embodiment of the present disclosure.

[0014] FIG. 2 illustrates an AI server according to one embodiment of the present disclosure.

[0015] FIG. 3 is a sequence diagram for explaining an operation method of an artificial intelligence system according to an embodiment of the present disclosure.

[0016] FIG. 4 is a diagram illustrating a process of learning a text encoder according to an embodiment of the present disclosure.

[0017] FIG. 5 is a diagram illustrating a process of learning a log encoder according to an embodiment of the present disclosure.

[0018] Figures 6a and 6b are diagrams illustrating an embedding vector sequence having time-series characteristics output through a log encoder.

[0019] FIG. 7 is a diagram illustrating a process of learning a log encoder through contrastive learning according to one embodiment of the present disclosure.

[0020] FIG. 8 is a flowchart illustrating a process for obtaining embedding vector distribution information according to one embodiment of the present disclosure.

[0021] FIG. 9a is a diagram illustrating embedding vector accumulation information output through a log encoder according to an embodiment of the present disclosure, and FIG. 9b is a diagram illustrating embedding vector distribution information based on the embedding vector accumulation information.

[0022] FIG. 10 is a diagram illustrating a situation inference result representing a user's lifestyle pattern for a week corresponding to embedding vectors determined according to one embodiment of the present disclosure.

[0023] Figure 11 is a diagram illustrating the process of learning a log encoder through contrastive learning using an image encoder.

[0024] Figure 12 is a diagram showing embedding vector distribution information based on the embedding vectors output by the image encoder and the bound-learned log encoder.

[0025] Figure 13 is a diagram illustrating the process of learning a log encoder through contrastive learning using an image encoder and a text encoder.

[0026] Figure 14 is a diagram showing embedding vector distribution information based on embedding vectors output by an image encoder, a text encoder, and a bound-learned log encoder.

[0027] FIG. 15 is a block diagram illustrating the configuration of an artificial intelligence device according to another embodiment of the present disclosure.

[0028] Artificial intelligence (AI) is the study of artificial intelligence or the methodologies for creating it, while machine learning (ML) defines various problems in the field of AI and studies the methodologies for solving them. Machine learning is also defined as an algorithm that improves performance on a task through consistent experience.

[0029] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0030] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0031] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0032] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0033] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0034] Supervised learning refers to a method for training an artificial neural network when given labels for the training data. The labels can refer to the correct answer (or output value) that the artificial neural network must infer when the training data is input to the artificial neural network. Unsupervised learning can refer to a method for training an artificial neural network when the training data is not given labels. Reinforcement learning can refer to a learning method in which an agent defined within a given environment is trained to select actions or action sequences that maximize the cumulative reward in each state.

[0035] Machine learning implemented with a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, the term "machine learning" is used to encompass deep learning.

[0036] FIG. 1 illustrates an artificial intelligence (AI) device according to one embodiment of the present disclosure.

[0037] The AI ​​device (100) can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a desktop computer, a laptop, a digital broadcasting terminal, a PDA (personal digital assistant), a PMP (portable multimedia player), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, digital signage, a robot, a vehicle, etc.

[0038] Referring to FIG. 1, the artificial intelligence device (100) may include a communication circuit (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0039] The communication circuit (110) can transmit and receive data with external devices such as other AI devices (100a to 100e) or AI servers (200) using wired or wireless communication technology.

[0040] The communication circuit (110) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

[0041] Communication technologies used by the communication circuit (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0042] The input interface (120) can obtain various types of data.

[0043] The input interface (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input interface for receiving information from a user, etc.

[0044] Cameras and microphones can be treated as sensors, and signals obtained from the camera or microphone can be called sensing data or sensor information.

[0045] The input interface (120) can obtain input data to be used when obtaining output using learning data and learning models for model learning. The input interface (120) can also obtain unprocessed input data, in which case the processor (180) or learning processor (130) can extract input features as preprocessing for the input data.

[0046] The learning processor (130) can train a model composed of an artificial neural network using learning data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer result values ​​for new input data other than the learning data, and the inferred values ​​can be used as a basis for making decisions regarding certain actions.

[0047] The running processor (130) can perform AI processing together with the running processor (240) of the AI ​​server (200).

[0048] The running processor (130) may include memory integrated or implemented in the AI ​​device (100). Alternatively, the running processor (130) may be implemented using memory (170), external memory directly coupled to the AI ​​device (100), or memory maintained in an external device.

[0049] The sensor (140) can obtain at least one of internal information of the AI ​​device (100), information about the surrounding environment of the AI ​​device (100), and user information using a plurality of sensors.

[0050] The sensor (140) may include one or more of a proximity sensor, a light sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, and a radar.

[0051] The output interface (150) can generate output related to visual, auditory, or tactile sensations.

[0052] The output interface (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0053] The memory (170) can store data that supports various functions of the AI ​​device (100). The memory (170) can store input data, learning data, learning models, learning history, etc. obtained from the input interface (120).

[0054] The processor (180) can determine at least one executable operation of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0055] The processor (180) can control components of the AI ​​device (100) to perform determined operations.

[0056] To this end, the processor (180) can request, retrieve, receive or utilize data from the running processor (130) or memory (170), and control components of the AI ​​device (100) to execute at least one of the executable operations, a predicted operation or an operation determined to be desirable.

[0057] When the processor (180) requires connection to an external device to perform a determined operation, it can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

[0058] The processor (180) can obtain intent information for user input and determine the user's requirements based on the obtained intent information.

[0059] The processor (180) can obtain intent information corresponding to the user input by using at least one of a STT (Speech To Text) engine for converting voice input into a string or a natural language processing (NLP) engine for obtaining intent information of natural language.

[0060] At least one of the STT engine or the NLP engine may be configured with an artificial neural network, at least in part, trained according to a machine learning algorithm. Furthermore, at least one of the STT engine or the NLP engine may be trained by the learning processor (130), the learning processor (240) of the AI ​​server (200), or through distributed processing thereof.

[0061] The processor (180) can collect history information including the operation details of the AI ​​device (100) or the user's feedback on the operation, and store the information in the memory (170) or the learning processor (130), or transmit the information to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0062] The processor (180) can control at least some of the components of the AI ​​device (100) to drive an application program stored in the memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination to drive the application program.

[0063] FIG. 2 illustrates an AI server (200) according to one embodiment of the present disclosure.

[0064] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or utilizes a trained artificial neural network. Here, the AI ​​server (200) may be composed of multiple servers to perform distributed processing, and may be defined as a 5G network.

[0065] The AI ​​server (200) may be included as part of the configuration of the AI ​​device (100) and may perform at least part of the AI ​​processing together.

[0066] The AI ​​server (200) may include a communication circuit (210), a memory (230), a learning processor (240), and a processor (260).

[0067] The communication circuit (210) can transmit and receive data with an external device such as an AI device (100).

[0068] The memory (230) may include a model storage unit (231). The model storage unit (231) may store a model (or artificial neural network, 231a) being learned or learned through the learning processor (240).

[0069] A learning processor (240) can train an artificial neural network (231a) using learning data. The learning model can be used while mounted on the AI ​​server (200) of the artificial neural network, or can be mounted on an external device such as an AI device (100).

[0070] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0071] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0072] FIG. 3 is a sequence diagram for explaining an operation method of an artificial intelligence system according to an embodiment of the present disclosure.

[0073] Hereinafter, the artificial intelligence device (100) may be referred to as an edge device.

[0074] The artificial intelligence device (100) may be a home appliance such as a refrigerator, washing machine, TV, or air conditioner. The artificial intelligence device (100) may also be a hub device that acts as a hub for multiple home appliances.

[0075] Hereinafter, modality data may be any one of text data, log data, image data, audio data, and sensing data.

[0076] Here, the embedding vector can represent a specific situation at a specific time (e.g., watching TV at a specific time, cooking at a specific time). In other words, the embedding vector can reflect the context or situation at a specific time.

[0077] Hereinafter, data indicating a user's lifestyle patterns may include at least one of the following: the user's bedtime, wake-up time, mealtime, and home appliance usage logs. The user's lifestyle patterns may be referred to as the user's behavioral patterns.

[0078] The learning processor (240) or processor (260) of the AI ​​server (200) can learn the log encoder through contrastive learning (S301).

[0079] The learning processor (240) or processor (260) can learn multiple encoders. The multiple encoders may include a text encoder and a log encoder. The multiple encoders may further include an image encoder that outputs an embedding vector from image data and a sensor encoder that outputs an embedding vector from sensing data.

[0080] The learning processor (240) or processor (260) can learn a log encoder using at least one of a text encoder, an image encoder, and a sensor encoder.

[0081] A text encoder may be an encoder that converts training text data representing a user's lifestyle pattern into a first embedding vector.

[0082] Text data representing a user's lifestyle patterns can be data that represents a series of repetitive activities in the user's daily life. For example, text data representing a user's lifestyle patterns might include wake-up times, meal times, bedtimes, and home appliance usage habits.

[0083] A text encoder can be a pre-trained model with a large amount of text data.

[0084] FIG. 4 is a diagram illustrating a process of learning a text encoder according to an embodiment of the present disclosure.

[0085] Referring to FIG. 4, a text encoder (400) is illustrated. The text encoder (400) may be an encoder that outputs an embedding vector from first to third training text data (410, 420, 430) representing a user's lifestyle pattern.

[0086] The learning processor (240) or processor (260) of the AI ​​server (200) can learn a text encoder using the BERT (Bidirectional Encoder Representations from Transformers) algorithm. The embedding vector may be a vector with 512 dimensions. The 512-dimensional embedding vector may be a numeric array that compressively expresses the patterns, relationships, and meaning of the original data.

[0087] Each of the text data for training (410, 420, 430) for the first to third sessions may be data collected continuously in 10-minute intervals. However, the 10-minute interval is merely an example.

[0088] The first to third training text data (410, 420, 430) can be input to the text encoder (400) in a time-series order.

[0089] Each training text data can be input into the text encoder (400) in 10-minute units. Accordingly, the text encoder (400) can also output an embedding vector in 10-minute units.

[0090] The first training text data (410) may represent text representing a user's actions of cooking using a cooktop in the morning.

[0091] The second training text data (420) may represent text representing a user's action of replenishing water through a water purifier in the afternoon.

[0092] The third training text data (430) may represent text representing an act of preparing a dish using ingredients in the refrigerator in the evening.

[0093] The running processor (240) or processor (260) can pre-train the text encoder (400) through sufficient text data and store the trained text encoder (400) in the memory (230).

[0094] Again, Figure 3 is explained.

[0095] A log encoder may be a model that outputs a second embedding vector based on training log data representing a user's lifestyle patterns. The log data representing a user's lifestyle patterns may be data indicating the user's behavior at a specific time and space. For example, the log data may indicate a user opening the refrigerator at 9:00 AM on Sunday. The log data may be an array of encrypted characters or numbers.

[0096] The running processor (240) or processor (260) can collect training log data from multiple artificial intelligence devices.

[0097] The learning processor (240) or processor (260) can convert training log data into knowledge data. The knowledge data may primarily be structured data on a specific domain or topic.

[0098] Knowledge data may be data converted from log data into a graph or table format. A log parser (not shown) or log converter (not shown) included in the processor (260) may generate knowledge data using log data.

[0099] The log encoder can convert the transformed knowledge data into a second embedding vector.

[0100] FIG. 5 is a diagram illustrating a process of learning a log encoder according to an embodiment of the present disclosure, and FIGS. 6a and 6b are diagrams illustrating an embedding vector sequence having time-series characteristics output through the log encoder.

[0101] Referring to FIG. 5, the learning processor (240) or processor (260) of the AI ​​server (200) can convert the first to third training log data (511, 512, 513) into the first to third training knowledge data (520, 530, 540). In FIG. 5, the first to third training log data (511, 512, 513) are written in a text-understandable format, but are arbitrarily interpreted data and can be recorded in an encrypted form. The first to third training log data (511, 512, 513) can have the form of a string or a number array.

[0102] For example, the first to third training log data (511, 512, 513) may be data representing the usage patterns of home appliances included in the user's lifestyle. The data representing the usage patterns of home appliances may include one or more of the usage frequency, usage time, power consumption patterns, or used functions of the home appliances.

[0103] Each of the first to third training log data (511, 512, 513) may be data collected continuously over a specific period of time. The specific period may be 10 minutes, but this is merely an example.

[0104] The first to third training knowledge data (520, 530, 540) may be data structured in the form of a graph or table, such as the first to third training log data (511, 512, 513).

[0105] The first to third training knowledge data (520, 530, 540) can be input to the text encoder (400) in chronological order. Each training knowledge data can be input to the log encoder (400). Accordingly, the log encoder (400) can also output an embedding vector in 10-minute units.

[0106] In another embodiment, the data input to the log encoder (500) may be output data obtained by inputting log data into a large language model (LLM).

[0107] The log encoder (500) can generate an embedding vector from training knowledge data (520, 530, 540) using the trans E method.

[0108] Translating Embeddings (TEMs) can be a technique for embedding entities and relationships in knowledge data into a vector space. TEMs can be a learning method that ensures that, if entities h and t are connected by a relationship r, the relationship h+t=r holds.

[0109] Referring to Fig. 6a, embedding vector accumulation information (610) representing the distribution of 512-dimensional values ​​for embedding vectors accumulated over a three-month period is illustrated. Embedding vectors can be extracted in 10-minute units, and embedding vectors based on log data accumulated over a three-month period can be plotted over a one-week window. In other words, Fig. 6a can be a diagram visualizing changes in each of the 512 dimensions of the embedding vector over time.

[0110] Embedding vector accumulation information (610) may include a plurality of vector dimension sequences (610-1 to 610-512). The plurality of vector dimension sequences (610-1 to 610-512) may represent a sequence of accumulated values ​​for each dimension of 512 embedding vectors accumulated over a period of three months. The horizontal axis of each vector dimension sequence may be a time axis, and the vertical axis may be a value of the dimension.

[0111] Fig. 6b illustrates embedding vector distribution information (620). The embedding vector distribution information (620) may be information showing the average and standard deviation for each vector dimension sequence included in the embedding vector accumulation information (610). In particular, the embedding vector distribution information (620) may include information on the average and standard deviation for each dimension at a specific time period or point in time.

[0112] That is, the mean and standard deviation of each dimension can be calculated through the values ​​for the dimension in the same time period.

[0113] Embedding vector distribution information (620) may include a plurality of vector dimension distributions (620-1 to 620-512). Each of the plurality of vector dimension distributions (620-1 to 620-512) may represent a one-week distribution of the mean and standard deviation of values ​​for each dimension of embedding vectors accumulated over a three-month period.

[0114] Referring to Figures 6a and 6b, when an embedding vector is extracted from a single modality data (here, log data) using a domain-specific encoder (here, a log encoder), the data exhibits discontinuous characteristics along the time axis. In other words, the data values ​​are too scattered, making it difficult to utilize them for analyzing and predicting users' lifestyle patterns.

[0115] Again, Figure 3 is explained.

[0116] The learning processor (240) or processor (260) may train the log encoder using contrastive learning so that the first embedding vector obtained from the text encoder and the second embedding vector obtained from the log encoder are located at the same point in the vector space. Each of the first embedding vector and the second embedding vector may correspond to different types of modality data obtained during the same period.

[0117] Contrastive learning can be referred to as binding learning.

[0118] Contrastive learning can be a method of learning to make the distance between similar vector pairs in a vector space closer to each other and the distance between dissimilar vector pairs farther away.

[0119] The running processor (240) or processor (260) can train a log encoder using two or more types of modality data collected during the same period so that pairs of embedding vectors corresponding to two or more types of modality data are located at the same point in the embedding space.

[0120] The running processor (240) or processor (260) can train a log encoder using multiple modality data of different types so that the embedding vector of each of the multiple modality data has the same value.

[0121] The log classifier can infer situations that match the second embedding vector transformed through the log encoder.

[0122] FIG. 7 is a diagram illustrating a process of learning a log encoder through contrastive learning according to one embodiment of the present disclosure.

[0123] Referring to FIG. 7, the learning processor (240) or processor (260) can train the log encoder (500-1) so that the embedding vector output from text data through the text encoder (400) and the embedding vector output from knowledge data through the log encoder (500-1) are located at the same point in the vector space.

[0124] For example, the learning processor (240) or processor (260) can train the log encoder (500-1) so that the embedding vector output from the first training knowledge data (520) through the log encoder (500-1) is located at the same point in the vector space as the embedding vector output from the first training text data (410) through the text encoder (400). The first training knowledge data (520) that serves as the basis for the first training text data (410) and the first training log data (511) may be data based on user lifestyle pattern data collected during the same period.

[0125] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-1) so that the embedding vector output from the second training knowledge data (530) through the log encoder (500-1) is located at the same point in the vector space as the embedding vector output from the second training text data (420) through the text encoder (400). The second training knowledge data (530) that serves as the basis for the second training text data (420) and the second training log data (512) may be data based on user lifestyle pattern data collected during the same period.

[0126] The learning processor (240) or processor (260) can train the log encoder (500-1) so that the embedding vector output from the third training knowledge data (540) through the log encoder (500) is located at the same point in the vector space as the embedding vector output from the third training text data (430) through the text encoder (400). The third training knowledge data (540), which is the basis of the third training text data (430) and the third training log data (513), may be data based on user lifestyle pattern data collected during the same period.

[0127] The learning processor (240) or processor (260) can transmit the log encoder (500) for which learning has been completed to the AI ​​device (100) through the communication circuit (210).

[0128] Again, Figure 3 is explained.

[0129] The learning processor (240) or processor (260) of the AI ​​server (200) can transmit a log encoder among the plurality of learned encoders to the AI ​​device (100) through the communication circuit (210) (S303).

[0130] The AI ​​server (200) can transmit the learned log encoder to the AI ​​device (100) as shown in FIG. 7. The AI ​​server (200) can transmit information about parameters such as weights of the log encoder to the AI ​​device (100). The AI ​​server (200) can transmit information necessary for installing the log encoder on the AI ​​device (100) to the AI ​​device (100).

[0131] The processor (180) of the AI ​​device (100) can acquire a log data set for a specific period of time (S305).

[0132] A specific period of time may be three months, but this is only an example.

[0133] In one embodiment, a log data set acquired over a specific period of time may be a data set used to train a log encoder in the AI ​​server (200).

[0134] In another embodiment, the log data set acquired over a specific period of time may be a data set collected by the AI ​​device (100) over a specific period of time.

[0135] A log data set can be a collection of log data collected over a period of time. The period of time can be 10 minutes, but this is merely an example.

[0136] The processor (180) of the AI ​​device (100) can generate embedding vector distribution information based on the acquired log data set (S307).

[0137] The processor (180) can convert each log data into knowledge data and generate an embedding vector from the converted knowledge data through a log encoder.

[0138] The processor (180) can obtain embedding vector distribution information using a plurality of embedding vectors obtained over a specific period of time.

[0139] Step S307 is described with reference to FIG. 8.

[0140] FIG. 8 is a flowchart illustrating a process for obtaining embedding vector distribution information according to one embodiment of the present disclosure.

[0141] The processor (180) of the AI ​​device (100) can convert log data into knowledge data (S801).

[0142] In one embodiment, the processor (180) may receive log data from one or more home appliances. The log data may be collected over a period of time or at a specific point in time. The log data may be a period of time. The period of time may be 10 minutes, but this is merely an example.

[0143] In another embodiment, the processor (180) may receive log data from a management server that manages one or more home appliances. The processor (180) may convert the log data into knowledge data through either a log parser (not shown) or a log converter (not shown).

[0144] Log data can be obtained from at least one of a home appliance, a sensor, or a camera.

[0145] The processor (180) can acquire log data at regular intervals. The regular interval may be 10 minutes, but this is merely an example. The 10-minute log data can be later converted into a single embedding vector.

[0146] The processor (180) of the AI ​​device (100) can obtain an embedding vector from knowledge data through a log encoder received from the AI ​​server (200) (S803).

[0147] The processor (180) can obtain an embedding vector from knowledge data through a log encoder received from the AI ​​server (200).

[0148] In another embodiment, the processor (180) may use the output data of an LLM model for log data as input data of a log encoder. The processor (180) may also generate an embedding vector from the output data of the LLM model through the log encoder.

[0149] The processor (180) can match a plurality of embedding vectors and context data corresponding to each of the plurality of embedding vectors and store them in the memory (170). The context data corresponding to each embedding vector may be data indicating a specific situation. The context data may be any one of log data, knowledge data, or text data indicating a situation matched to the log data.

[0150] A plurality of embedding vectors stored in memory (170) may be referred to as an embedding vector matrix.

[0151] The processor (180) of the AI ​​device (100) can generate embedding vector distribution information based on embedding vectors acquired over a specific period of time (S805).

[0152] The processor (180) can generate embedding vector distribution information based on the values ​​of 512 dimensions included in each of the embedding vectors acquired during a specific period.

[0153] The processor (180) can store the generated embedding vector distribution information in the memory (170).

[0154] FIG. 9a is a diagram illustrating embedding vector accumulation information output through a log encoder according to an embodiment of the present disclosure, and FIG. 9b is a diagram illustrating embedding vector distribution information based on the embedding vector accumulation information.

[0155] Referring to Fig. 9a, cumulative information (910) of an embedding vector having time-series characteristics is illustrated. The log encoder (500-1) may be an encoder learned through contrastive learning with a text encoder.

[0156] Embedding vector accumulation information (910) can represent the accumulated results of embedding vectors output from the log encoder (500-1) over a specific period of time, plotted over a window of one week.

[0157] Embedding vector accumulation information (910) may include a plurality of vector dimension sequences (910-1 to 910-512). The plurality of vector dimension sequences (910-1 to 910-512) may represent a sequence of values ​​of each dimension of the embedding vector accumulated over a period of three months. The horizontal axis of each vector dimension sequence may be a time axis, and the vertical axis may be a value of the dimension.

[0158] Comparing the embedding vector accumulation information (910) of FIG. 9a with the embedding vector accumulation information (610) obtained using only the log data of FIG. 6a, it can be confirmed that the flow of embedding vectors over time is more continuously aligned.

[0159] Referring to FIG. 9b, embedding vector distribution information (920) obtained based on the embedding vector accumulation information (910) of FIG. 9a is illustrated. The embedding vector distribution information (920) may be information showing the average and standard deviation for each vector dimension sequence included in the embedding vector distribution information (910).

[0160] Embedding vector distribution information (920) may include a plurality of vector dimension distributions (920-1 to 920-512). Each of the plurality of vector dimension distributions (920-1 to 920-512) may represent a one-week distribution of the mean and standard deviation of values ​​for each dimension of embedding vectors accumulated over a three-month period.

[0161] Again, Figure 3 is explained.

[0162] The processor (180) of the AI ​​device (100) can predict the embedding vector of a specific time period based on the embedding vector distribution information (S309).

[0163] The processor (180) can determine an embedding vector based on the distribution of values ​​of each dimension corresponding to a specific time period included in the embedding vector distribution information (920).

[0164] In one embodiment, the processor (180) may determine the average value of the values ​​of each dimension corresponding to a specific time period included in the embedding vector distribution information (920) as the value of the corresponding dimension. However, this is merely an example, and other methods for determining the values ​​of the dimensions of the embedding vector may be used.

[0165] For example, referring to FIG. 9B, the processor (180) may determine the average value of the accumulated values ​​of the first dimension from the first vector dimension distribution (920-1) at 8:00 PM on Friday as the value of the first dimension of the embedding vector. In this manner, the processor (180) may determine 512 values ​​(930-1 to 930-512) of 512 dimensions corresponding to the time of 8:00 PM on Friday as the embedding vector.

[0166] The processor (180) of the AI ​​device (100) can obtain context data corresponding to the predicted embedding vector (S311).

[0167] The processor (180) can extract an embedding vector having the greatest similarity to the determined embedding vector from the memory (170). The processor (180) can compare a plurality of embedding vectors stored in the memory (170) with the determined embedding vector, and can extract an embedding vector having the greatest similarity among the plurality of embedding vectors.

[0168] The processor (180) can extract context data corresponding to the embedding vector with the greatest similarity from the memory (170).

[0169] The processor (180) of the AI ​​device (100) can provide a service based on the acquired context data (S313).

[0170] The processor (180) can provide a proactive service to the user based on context data. The proactive service may be either a service that sends an alarm to the user's terminal or an automatic setting service for a home appliance, but these are merely examples.

[0171] Contextual data may include information about a user's anticipated lifestyle patterns at a specific point in time or over a specific period of time.

[0172] For example, if the acquired context data is <8 PM on Friday, start cooking>, the processor (180) can turn on the power of the oven or induction when it is 8 PM on Friday.

[0173] In this way, according to an embodiment of the present disclosure, a situation can be comprehensively understood by utilizing multimodal data having various forms and characteristics, and the understood situation can be processed as sequential data to predict a user's pattern.

[0174] Accordingly, customized services tailored to the user's personalized lifestyle patterns can be provided, greatly improving user convenience.

[0175] FIG. 10 is a diagram illustrating a situation inference result representing a user's lifestyle pattern for a week corresponding to embedding vectors determined according to one embodiment of the present disclosure.

[0176] Figure 10 shows a situation inference result (1000) corresponding to the predicted embedding vectors based on the embedding vector distribution information (920) of Figure 9b. The situation inference result (1000) may be a result of interpreting the predicted embedding vectors.

[0177] The situation inference result (1000) may be context data representing the user's lifestyle pattern for a week.

[0178] The situation inference result (1000) may include information about the user's lifestyle pattern for the morning time on each day of the week, the user's lifestyle pattern for the evening time on each day of the week, and the user's bedtime on each day of the week.

[0179] The text generator can generate a text-based situation inference result (1000) based on log data matching the embedding vectors. The text generator can be included in the processor (180) or configured separately.

[0180] The situation inference result (1000) can be input into a large language model and used to obtain the information desired by the user.

[0181] Fig. 11 is a diagram explaining the process of learning a log encoder through contrastive learning using an image encoder, and Fig. 12 is a diagram showing embedding vector distribution information based on embedding vectors output by an image encoder and a bound-learned log encoder.

[0182] Referring to FIG. 11, the learning processor (240) or processor (260) of the AI ​​server (200) can train the log encoder (500-2) so that the embedding vector output from the image data through the image encoder (1100) and the embedding vector output from the knowledge data through the log encoder (500-2) are located at the same point in the vector space.

[0183] For example, the learning processor (240) or processor (260) can train the log encoder (500-2) so that the embedding vector output from the first training knowledge data (520) through the log encoder (500-2) is located at the same point in the vector space as the embedding vector output from the first training image data (1110) through the image encoder (1100). The first training image data (1110) and the first training knowledge data (520) may be data based on user lifestyle pattern data collected during the same period.

[0184] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-2) so that the embedding vector output from the second training knowledge data (530) through the log encoder (500-2) is located at the same point in the vector space as the embedding vector output from the second training image data (1120) through the image encoder (1100). The second training image data (1120) and the second training knowledge data (530) may be data based on user lifestyle pattern data collected during the same period.

[0185] The learning processor (240) or processor (260) can train the log encoder (500-2) so that the embedding vector output from the third training knowledge data (540) through the log encoder (500-2) is located at the same point in the vector space as the embedding vector output from the third training image data (1130) through the image encoder (1100). The third training image data (1130) and the third training knowledge data (540) may be data based on user lifestyle pattern data collected during the same period.

[0186] The learning processor (240) or processor (260) can transmit the log encoder (500-2) for which learning has been completed to the AI ​​device (100) through the communication circuit (210).

[0187] That is, the log encoder (500-2) may be trained through contrastive learning with an image encoder (1100) that uses image data rather than a text encoder (400).

[0188] Referring to FIG. 12, embedding vector distribution information (1200) based on embedding vectors output by an image encoder (1100) and a log encoder (500-2) learned through contrastive learning is illustrated.

[0189] Comparing the embedding vector distribution information (1200) of FIG. 12 with the embedding vector distribution information (900) of FIG. 9b, the embedding vector distribution information (1200) may have a smaller variance in each dimension overall than the embedding vector distribution information (900) of FIG. 9b. This may be a result of the greater diversity of image data than that of text data.

[0190] The processor (180) can determine an embedding vector for a specific time period based on the embedding vector distribution information (1200).

[0191] Fig. 13 is a diagram illustrating a process of learning a log encoder through contrastive learning using an image encoder and a text encoder, and Fig. 14 is a diagram showing embedding vector distribution information based on embedding vectors output by a log encoder that has been bound-learned with an image encoder and a text encoder.

[0192] Referring to FIG. 13, the learning processor (240) or processor (260) of the AI ​​server (200) can train the log encoder (500-3) so that the embedding vector output from image data through the image encoder (1100), the embedding vector output from text data through the text encoder (400), and the embedding vector output from knowledge data through the log encoder (500-3) are located at the same point in the vector space.

[0193] For example, the learning processor (240) or the processor (260) can train the log encoder (500-3) so that the embedding vector output from the first training knowledge data (520) through the log encoder (500-3) is located at the same point in the vector space as the embedding vector output from the first training image data (1110) through the image encoder (1100) and the embedding vector output from the first training text data (410) through the text encoder (400). The first training image data (1110), the first training text data (410), and the first training knowledge data (520) may be data based on user lifestyle pattern data collected during the same period.

[0194] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-3) so that the embedding vector output from the second training knowledge data (530) through the log encoder (500-3) is located at the same point in the vector space as the embedding vector output from the second training image data (1120) through the image encoder (1100) and the embedding vector output from the second training text data (420) through the text encoder (400). The second training image data (1120), the second training text data (420), and the second training knowledge data (530) may be data based on user lifestyle pattern data collected during the same period.

[0195] The learning processor (240) or processor (260) can train the log encoder (500-3) so that the embedding vector output from the third training knowledge data (540) through the log encoder (500-3) is located at the same point in the vector space as the embedding vector output from the third training image data (1130) through the image encoder (1100) and the embedding vector output from the third training text data (430) through the text encoder (400). The third training image data (1130), the third training text data (430), and the third training knowledge data (540) may be data based on user lifestyle pattern data collected during the same period.

[0196] The learning processor (240) or processor (260) can transmit the log encoder (500-3) for which learning has been completed to the AI ​​device (100) through the communication circuit (210).

[0197] That is, the log encoder (500-3) may be trained through contrastive learning with the text encoder (400) and the image encoder (1100).

[0198] Referring to FIG. 14, embedding vector distribution information (1400) based on embedding vectors output by a log encoder (500-3) learned through contrastive learning with an image encoder (1100) and a text encoder (400) is illustrated.

[0199] Comparing the embedding vector distribution information (1400) of Fig. 14 with the embedding vector distribution information (900) of Fig. 9b and the embedding vector distribution information (1200) of Fig. 12, the embedding vector distribution information (1400) of Fig. 14 may have a smaller variance value for each dimension overall than the embedding vector distribution information (900) of Fig. 9b and the embedding vector distribution information (1200) of Fig. 12. This may indicate that as the number of encoders using different types of modality data bound to the log encoder (500-3) increases, a more stable situation can be predicted.

[0200] The processor (180) can determine an embedding vector for a specific time period based on the embedding vector distribution information (1400).

[0201] FIG. 15 is a block diagram illustrating the configuration of an artificial intelligence device according to another embodiment of the present disclosure.

[0202] Referring to FIG. 15, the edge device (1500) may include a log converter (1510), a log encoder (1520), a log embedding matrix DB (1530), an action predictor (1540), a log classifier (1550), a log search DB (1560), a DB manager (1570), and a text generator (1580).

[0203] The log converter (1510) can convert log data into knowledge data.

[0204] The log encoder (1520) can convert knowledge data into an embedding vector.

[0205] The log encoder (1520) may be an encoder trained through contrastive learning using at least one of a text encoder, an image encoder, and a sensor encoder.

[0206] The log embedding matrix DB (1530) can store embedding vectors output from the log encoder (1520). Each embedding vector can be stored with a matching log data collection period.

[0207] The action predictor (1540) can generate embedding vector distribution information based on embedding vectors, and can predict an embedding vector for a specific time period through the embedding vector distribution information.

[0208] The log classifier (1550) can classify the embedding vector output from the log encoder (1520). The log classifier (1550) can match the embedding vector to log data or knowledge data based on log data and store it in the log search DB (1560).

[0209] The log search DB (1560) can store multiple embedding vectors and log data or knowledge data matching each embedding vector.

[0210] The DB manager (1570) can extract an embedding vector most similar to the embedding vector of a specific time period predicted by the action predictor (1540) among multiple embedding vectors stored in the log search DB (1560).

[0211] The DB manager (1570) can obtain log data matching the embedding vector extracted from the log search DB (1560).

[0212] The DB manager (1570) can transmit log data of the acquired embedding vector and period data for a specific period to the text generator (1580).

[0213] The text generator (1580) can output a situation inference result (1000) as in FIG. 10 based on the input data.

[0214] The edge device (1500) may be an example of the artificial intelligence device (100) of FIG. 1.

[0215] The log converter (1510), log encoder (1520), behavior predictor (1540), log classifier (1550), DB manager (1570), and text generator (1580) may be included in the processor (180) of FIG. 1 or may be separate components from the processor (180).

[0216] The log embedding matrix DB (1530) and the log search DB (1560) may be included in the memory (170) of FIG. 1 or may be separate components from the memory (170).

[0217] The above-described present disclosure can be implemented as computer-readable code on a program-recorded medium. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0218] Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In artificial intelligence devices, A memory storing a first encoder that outputs an embedding vector from a first type of data representing a user's lifestyle pattern, wherein the first encoder is an encoder learned through contrastive learning with a second encoder that outputs an embedding vector from a second type of data representing the user's lifestyle pattern; and A device comprising: a first type of log data set representing a user's lifestyle pattern for a specific period of time; one or more processors for obtaining embedding vectors from the obtained log data set through the first encoder; generating embedding vector distribution information based on the obtained embedding vectors; predicting an embedding vector at a specific point in time or a specific time period based on the embedding vector distribution information; obtaining context data matching the predicted embedding vector; and providing a service based on the obtained context data. Artificial intelligence device.

2. In paragraph 1, The above first type of data is It is log data, The second type of data is Any one of text data, image data, or sensing data Artificial intelligence device.

3. In paragraph 1, One or more of the above processors Converting log data included in the first type of log data set into knowledge data, and obtaining an embedding vector from the converted knowledge data through the first encoder. Artificial intelligence device.

4. In paragraph 3, One or more of the above processors Generate the embedding vector distribution information based on the embedding vectors accumulated during the specific period, The above embedding vector distribution information is Representing the continuous distribution of values ​​of the dimensions of the accumulated embedding vectors Artificial intelligence device.

5. In paragraph 4, One or more of the above processors Predicting the embedding vector based on determining the average value of the values ​​of each dimension corresponding to a specific time period included in the embedding vector distribution information as the value of the corresponding dimension. Artificial intelligence device.

6. In paragraph 1, The above memory is Storing the above embedding vectors and log data corresponding to each embedding vector in the memory. Artificial intelligence device.

7. In paragraph 6, One or more of the above processors Extracting an embedding vector most similar to the predicted embedding vector among the stored embedding vectors, and obtaining log data corresponding to the extracted embedding vector as the context data. Artificial intelligence device.

8. In paragraph 1, The first type of data and the second type of data are data collected at the same time. Artificial intelligence device.

9. In paragraph 1, The above first encoder An encoder trained so that the embedding vector output from the first encoder is located at the same point in the vector space as the embedding vector output from the second encoder. Artificial intelligence device.

10. In paragraph 1, The above service is Either an alarm sent to the user's terminal or an automatic setting of the home appliance Artificial intelligence device.

11. In the method of operating an artificial intelligence device, A step of obtaining a first type of log data set representing a user's lifestyle pattern for a specific period of time; A step of obtaining embedding vectors from the obtained log data set through a first encoder, wherein the first encoder outputs an embedding vector from the first type of data representing the user's lifestyle pattern, and the second encoder outputs an embedding vector from the second type of data representing the user's lifestyle pattern, and is an encoder learned through contrastive learning; A step of generating embedding vector distribution information based on the obtained embedding vectors; A step of predicting an embedding vector at a specific point in time or a specific time period based on the embedding vector distribution information; A step of obtaining context data matching the predicted embedding vector; and Including a step of providing a service based on the acquired context data. How an artificial intelligence device operates.

12. In paragraph 11, The above first type of data is It is log data, The second type of data is Any one of text data, image data, or sensing data How an artificial intelligence device operates.

13. In paragraph 11, The step of obtaining the above embedding vectors is A step of converting log data included in the first type of log data set into knowledge data and obtaining an embedding vector from the converted knowledge data through the first encoder. How an artificial intelligence device operates.

14. In paragraph 13, The step of generating the above embedding vector distribution information is A step of generating the embedding vector distribution information based on the embedding vectors accumulated during the specific period of time, The above embedding vector distribution information is Representing the continuous distribution of values ​​of the dimensions of the accumulated embedding vectors How an artificial intelligence device operates.

15. In paragraph 14, The above predicting steps are A step of predicting the embedding vector based on determining the average value of values ​​of each dimension corresponding to a specific time period included in the embedding vector distribution information as the value of the corresponding dimension. How an artificial intelligence device operates.

16. In paragraph 11, Further comprising a step of storing the above embedding vectors and log data corresponding to each embedding vector. How an artificial intelligence device operates.

17. In paragraph 16, The step of obtaining the above context data is A step of extracting an embedding vector most similar to the predicted embedding vector among the stored embedding vectors, and A step of obtaining log data corresponding to the extracted embedding vector as the context data is included. How an artificial intelligence device operates.

18. In paragraph 11, The first type of data and the second type of data are data collected at the same time. How an artificial intelligence device operates.

19. In paragraph 11, The above first encoder An encoder trained so that the embedding vector output from the first encoder is located at the same point in the vector space as the embedding vector output from the second encoder. How an artificial intelligence device operates.

20. A recording medium having recorded thereon a computer-readable program for performing an operation method of an artificial intelligence device, The method of operation of the above artificial intelligence device is A step of obtaining a first type of log data set representing a user's lifestyle pattern for a specific period of time; A step of obtaining embedding vectors from the obtained log data set through a first encoder, wherein the first encoder outputs an embedding vector from the first type of data representing the user's lifestyle pattern, and the second encoder outputs an embedding vector from the second type of data representing the user's lifestyle pattern, and is an encoder learned through contrastive learning. A step of generating embedding vector distribution information based on the acquired embedding vectors, A step of predicting an embedding vector at a specific point in time or a specific time period based on the above embedding vector distribution information, A step of obtaining context data matching the predicted embedding vector, and Including a step of providing a service based on the acquired context data. Recording medium.

Images