Artificial intelligence device and operation method thereof

Abstract

An artificial intelligence device according to one embodiment of the present disclosure comprises: a memory storing a first encoder for outputting embedding vectors from first-type data representing a life pattern of a user, wherein the first encoder is an encoder trained through contrastive training with a second encoder for outputting embedding vectors from second-type data representing the life pattern of the user; and one or more processors for periodically acquiring log data, acquiring, through the log encoder, embedding vectors in an embedding space from the log data that has been periodically acquired, converting the embedding vectors into first time-series continuous data and smoothing data, respectively, and acquiring second time-series continuous data satisfying a condition in which the first time-series continuous data converges to the smoothing data.

Description

Artificial intelligence device and its method of operation

[0001] The present invention relates to an artificial intelligence device, and more specifically, to an artificial intelligence device that infers a situation based on log data.

[0002] A multimodal model represents a model that processes various types of input data and generates various types of outputs.

[0003] Multimodal models can primarily handle various forms of data simultaneously, such as text, images, and audio. For example, they can generate descriptions by understanding images and captions at the same time, or improve natural language understanding and speech recognition by processing text and voice together.

[0004] Conventionally, encoders utilizing different types of modality data can be bound and trained to predict future situations. Time series models predict embedding vectors for future situations based on embedding vectors in the embedding space output by the encoders.

[0005] However, according to conventional technology, although the deviation of the embedding vector distribution was reduced through an encoder learned by binding between modality data, discontinuous characteristics remain when observing the trend along the time axis.

[0006] If discontinuous characteristics remain due to noise, discontinuities, or missing data in the embedding vector, there is a problem that it is unsuitable for time series forecasting, which requires the prediction of accurate values ​​at every time point.

[0007] The purpose of the present disclosure may be to serialize the embedding vectors input to a time series model to improve the performance of the time series model.

[0008] The purpose of the present disclosure may be to provide an artificial intelligence device capable of serializing irregular time series data so that it can be restored.

[0009] The purpose of the present disclosure may be to improve the prediction performance of a time series model by removing noise from the embedding vectors input to the time series model.

[0010] An artificial intelligence device according to one embodiment of the present disclosure may include: a memory storing a first encoder that outputs an embedding vector from first type data representing a user's lifestyle pattern, wherein the first encoder is an encoder learned through comparative learning with a second encoder that outputs an embedding vector from second type data representing the user's lifestyle pattern; and one or more processors that periodically acquire log data, acquire embedding vectors in an embedding space from the periodically acquired log data through the log encoder, convert the embedding vectors into first time series continuous data and smoothing data, respectively, and acquire second time series continuous data satisfying the condition that the first time series continuous data converges to the smoothing data.

[0011] A method of operation of an artificial intelligence device according to one embodiment of the present disclosure may include: a step of periodically acquiring log data; a step of acquiring embedding vectors in an embedding space from the periodically acquired log data; a step of converting the embedding vectors into a first time series continuous data and smoothing data, respectively; and a step of acquiring a second time series continuous data satisfying a condition in which the first time series continuous data converges to the smoothing data.

[0012] In a recording medium storing a computer-readable program for performing a method of operation of an artificial intelligence device according to one embodiment of the present disclosure, the method of operation of the artificial intelligence device may include: a step of periodically acquiring log data; a step of acquiring embedding vectors in an embedding space from the periodically acquired log data; a step of converting the embedding vectors into a first time series continuous data and smoothing data, respectively; and a step of acquiring a second time series continuous data satisfying a condition in which the first time series continuous data converges to the smoothing data.

[0013] According to an embodiment of the present disclosure, vectors input to a time series model can be prepared in an optimal state through a process of preprocessing embedding vectors and a process of smoothing embedding vectors. Accordingly, the problem of performance degradation of the time series model can be resolved.

[0014] In addition, as the performance of the time series model improves, the ability to infer future situations can also be enhanced.

[0015] According to an embodiment of the present disclosure, smoothed time series continuous data can be fully restored to the original data through the inverse operation of an MPM.

[0016] FIG. 1 shows an AI device according to one embodiment of the present disclosure.

[0017] FIG. 2 shows an AI server according to one embodiment of the present disclosure.

[0018] FIG. 3 is a sequence diagram for explaining the operation method of an artificial intelligence system according to one embodiment of the present disclosure.

[0019] FIG. 4 is a diagram illustrating the process of learning a text encoder according to one embodiment of the present disclosure.

[0020] FIG. 5 is a diagram illustrating the process of learning a log encoder according to one embodiment of the present disclosure.

[0021] Figures 6A and 6B illustrate an embedding vector sequence having time-series characteristics output through a log encoder.

[0022] FIG. 7 is a diagram illustrating the process of learning a log encoder through comparative learning according to one embodiment of the present disclosure.

[0023] FIG. 8 is a flowchart illustrating the process of obtaining embedding vector distribution information according to one embodiment of the present disclosure.

[0024] FIG. 9A is a diagram illustrating embedding vector accumulation information output through a log encoder according to an embodiment of the present disclosure, and FIG. 9B is a diagram illustrating embedding vector distribution information based on embedding vector accumulation information.

[0025] FIG. 10 is a diagram illustrating the results of situation inference representing a user's lifestyle pattern over a week corresponding to embedding vectors determined according to one embodiment of the present disclosure.

[0026] Figure 11 is a diagram illustrating the process of learning a log encoder through contrast learning using an image encoder.

[0027] Figure 12 is a diagram showing embedding vector distribution information based on embedding vectors output by an image encoder and a bound learned log encoder.

[0028] Figure 13 is a diagram illustrating the process of learning a log encoder through contrast learning using an image encoder and a text encoder.

[0029] Figure 14 is a diagram showing embedding vector distribution information based on embedding vectors output by an image encoder and a text encoder and a bounding-learned log encoder.

[0030] FIG. 15 is a block diagram illustrating the configuration of an artificial intelligence device according to another embodiment of the present disclosure.

[0031] FIG. 16 is a diagram illustrating the process of predicting future embedding vectors from embedding vectors through a time series model according to one embodiment of the present disclosure.

[0032] Figure 17 is a diagram illustrating embedding vector distribution information based on embedding vector accumulation information obtained based on the embedding vectors of Figure 16.

[0033] Figure 18 is a diagram illustrating the prediction results output through a time series model from the embedding vector distribution information of Figure 17.

[0034] FIG. 19 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0035] FIG. 20 is a diagram illustrating the process of preprocessing embedding vectors through a sequence split method according to one embodiment of the present disclosure.

[0036] FIG. 21 is a diagram illustrating the process of learning the difference between the first time series continuous data and the smoothing data and the process of restoring the embedding vectors according to an embodiment of the present disclosure.

[0037] FIG. 22 is a diagram illustrating the restoration results for the output of the time series prediction data and the time series model generated with the weights of the final MPM, after undergoing a sequence splitting method for embedding vectors according to an embodiment of the present disclosure.

[0038] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define and solve various problems addressed within the field of artificial intelligence. Machine learning is also defined as an algorithm that improves performance on a task through continuous experience.

[0039] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0040] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0041] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0042] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0043] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0044] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0045] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.

[0046] FIG. 1 shows an artificial intelligence (AI) device according to one embodiment of the present disclosure.

[0047] The AI ​​device (100) can be implemented as a stationary device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0048] Referring to FIG. 1, the artificial intelligence device (100) may include a communication circuit (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180), etc.

[0049] The communication circuit (110) can transmit and receive data with external devices such as other AI devices (100a to 100e) or an AI server (200) using wired and wireless communication technology.

[0050] The communication circuit (110) can transmit and receive sensor information, user input, learning model, control signal, etc., with external devices.

[0051] The communication technologies used by the communication circuit (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0052] The input interface (120) can acquire various types of data.

[0053] The input interface (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input interface for receiving information from a user, etc.

[0054] Cameras or microphones can be treated as sensors, and signals acquired from cameras or microphones can be referred to as sensing data or sensor information.

[0055] The input interface (120) can obtain training data for model training and input data to be used when obtaining an output using the training model. The input interface (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input features as a preprocessing step for the input data.

[0056] The learning processor (130) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform an action.

[0057] The learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0058] The learning processor (130) may include memory integrated into or implemented in the AI ​​device (100). Alternatively, the learning processor (130) may be implemented using memory (170), external memory directly coupled to the AI ​​device (100), or memory maintained in an external device.

[0059] The sensor (140) can acquire at least one of internal information of the AI ​​device (100), surrounding environment information of the AI ​​device (100), and user information using a plurality of sensors.

[0060] The sensor (140) may include one or more of a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, and a radar.

[0061] The output interface (150) can generate output related to visual, auditory, or tactile senses.

[0062] The output interface (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0063] The memory (170) can store data that supports various functions of the AI ​​device (100). The memory (170) can store input data, training data, training models, training history, etc. obtained from the input interface (120).

[0064] The processor (180) can determine at least one executable action of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0065] The processor (180) can control the components of the AI ​​device (100) to perform a determined operation.

[0066] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the AI ​​device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0067] If the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0068] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0069] The processor (180) can obtain intent information corresponding to user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0070] At least one of the STT engine or NLP engine may be composed of an artificial neural network, at least a portion of which is trained according to a machine learning algorithm. Additionally, at least one of the STT engine or NLP engine may be trained by a learning processor (130), trained by a learning processor (240) of an AI server (200), or trained by distributed processing thereof.

[0071] The processor (180) may collect history information, including the operation details of the AI ​​device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information may be used to update a learning model.

[0072] The processor (180) can control at least some of the components of the AI ​​device (100) to run an application stored in memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination with each other to run the application.

[0073] FIG. 2 shows an AI server (200) according to one embodiment of the present disclosure.

[0074] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0075] The AI ​​server (200) may be included as part of the configuration of the AI ​​device (100) and may perform at least some of the AI ​​processing together.

[0076] The AI ​​server (200) may include a communication circuit (210), memory (230), a learning processor (240), and a processor (260), etc.

[0077] The communication circuit (210) can transmit and receive data with an external device such as an AI device (100).

[0078] The memory (230) may include a model storage unit (231). The model storage unit (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through a learning processor (240).

[0079] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an AI device (100).

[0080] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0081] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0082] FIG. 3 is a sequence diagram for explaining the operation method of an artificial intelligence system according to one embodiment of the present disclosure.

[0083] In the following, the artificial intelligence device (100) may be referred to as an edge device.

[0084] The artificial intelligence device (100) may be a home appliance such as a refrigerator, washing machine, TV, or air conditioner. The artificial intelligence device (100) may also be a hub device that serves as a hub for multiple home appliances.

[0085] In the following, modality data may be any one of text data, log data, image data, audio data, and sensing data.

[0086] In the following, embedding vectors can represent specific situations at specific times (e.g., watching TV or cooking at a specific time). In other words, embedding vectors can reflect the context or situation of a specific time.

[0087] In the following, data representing the user's lifestyle patterns may include at least one of the user's bedtime, wake-up time, mealtime, and usage logs of home appliances. The user's lifestyle patterns may be referred to as the user's behavioral patterns.

[0088] The learning processor (240) or processor (260) of the AI ​​server (200) can learn the log encoder through contrast learning (S301).

[0089] The learning processor (240) or processor (260) can learn multiple encoders. The multiple encoders may include a text encoder and a log encoder. The multiple encoders may further include an image encoder that outputs an embedding vector from image data and a sensor encoder that outputs an embedding vector from sensing data.

[0090] The learning processor (240) or processor (260) can learn the log encoder using at least one of a text encoder, an image encoder, and a sensor encoder.

[0091] The text encoder may be an encoder that converts training text data representing a user's lifestyle patterns into a first embedding vector.

[0092] Text data representing a user's lifestyle patterns may be data that describes the flow of a series of activities the user repeats in their daily life in text form. For example, text data representing a user's lifestyle patterns may include wake-up times, meal times, bedtimes, and appliance usage habits.

[0093] A text encoder can be a model pre-trained with a large amount of text data.

[0094] FIG. 4 is a diagram illustrating the process of learning a text encoder according to one embodiment of the present disclosure.

[0095] Referring to FIG. 4, a text encoder (400) is illustrated. The text encoder (400) may be an encoder that outputs an embedding vector from first to third training text data (410, 420, 430) representing a user's lifestyle pattern.

[0096] The learning processor (240) or processor (260) of the AI ​​server (200) can learn a text encoder through the BERT (Bidirectional Encoder Representations from Transformers) algorithm. The embedding vector may be a vector having 512 dimensions. The 512-dimensional embedding vector may be a numeric array that compresses and represents the patterns, relationships, and meanings of the original data.

[0097] Each of the first to third training text data (410, 420, 430) may be data collected continuously at 10-minute intervals. However, 10 minutes is merely an example value.

[0098] The first to third training text data (410, 420, 430) can be input into the text encoder (400) in a chronological order.

[0099] Each training text data can be input into the text encoder (400) in 10-minute intervals. Accordingly, the text encoder (400) can also output embedding vectors in 10-minute intervals.

[0100] The first training text data (410) can represent text indicating the action of a user cooking with a stovetop in the morning.

[0101] The second training text data (420) can represent text indicating the action of a user replenishing fluids through a water purifier in the afternoon.

[0102] The third training text data (430) can represent text indicating the act of preparing a meal with ingredients in the refrigerator in the evening.

[0103] The learning processor (240) or processor (260) can pre-train the text encoder (400) through sufficient text data and store the trained text encoder (400) in memory (230).

[0104] Again, Figure 3 is explained.

[0105] A log encoder may be a model that outputs a second embedding vector based on training log data representing a user's lifestyle patterns. The log data representing a user's lifestyle patterns may be data representing a user's behavior at a specific time and in a specific space. For example, the log data may be data representing the act of a user opening a refrigerator at 9:00 on Sunday. The log data may be an array of characters in an encrypted form or an array of numbers.

[0106] The learning processor (240) or processor (260) can collect training log data from multiple artificial intelligence devices.

[0107] The learning processor (240) or processor (260) can convert training log data into knowledge data. The knowledge data may be structured data primarily about a specific domain or topic.

[0108] Knowledge data may be data in which log data has been converted into a graph or table form. A log parser (not shown) or a log converter (not shown) included in the processor (260) can generate knowledge data using the log data.

[0109] The log encoder can convert the transformed knowledge data into a second embedding vector.

[0110] FIG. 5 is a diagram illustrating the process of learning a log encoder according to one embodiment of the present disclosure, and FIG. 6A and FIG. 6B are diagrams illustrating an embedding vector sequence having time-series characteristics output through a log encoder.

[0111] Referring to FIG. 5, the learning processor (240) or processor (260) of the AI ​​server (200) can convert the first to third training log data (511, 512, 513) into the first to third training knowledge data (520, 530, 540). In FIG. 5, the first to third training log data (511, 512, 513) is written at a level that is understandable as text, but is arbitrarily interpreted data and may be recorded in an encrypted form. The first to third training log data (511, 512, 513) may have the form of a string or a number array.

[0112] For example, the first to third training log data (511, 512, 513) may be data representing the usage patterns of home appliances included in the user's lifestyle patterns. The data representing the usage patterns of home appliances may include one or more of the frequency of use of home appliances, time of use, power consumption patterns, or functions used by home appliances.

[0113] Each of the first to third training log data (511, 512, 513) may be data collected continuously in units of a specific period. The specific period may be 10 minutes, but this is merely an example.

[0114] The first to third training knowledge data (520, 530, 540) may be data in which the first to third training log data (511, 512, 513) is structured in the form of a graph or a table.

[0115] The first to third training knowledge data (520, 530, 540) can be input into the text encoder (400) in a chronological order. Each training knowledge data can be input into the log encoder (400). Accordingly, the log encoder (400) can also output an embedding vector in 10-minute intervals.

[0116] In another embodiment, the data input to the log encoder (500) may be output data obtained by inputting log data into a Large Language Model (LLM).

[0117] The log encoder (500) can generate embedding vectors from training knowledge data (520, 530, 540) using the Trans E method.

[0118] Translating Embeddings (Trans E) can be a technique for embedding entities and relationships in knowledge data into a vector space. Translating E can be a method of learning such that when entity h and entity t are connected by relationship r, the relationship h+t=r holds.

[0119] Referring to FIG. 6A, embedding vector accumulation information (610) is shown, representing the distribution of 512-dimensional values ​​for embedding vectors accumulated over a period of three months. The embedding vectors can be extracted at 10-minute intervals, and the embedding vectors based on log data accumulated over three months can be plotted in a one-week window. That is, FIG. 6A may be a diagram visualizing the changes over time of each of the 512 dimensions of the embedding vectors.

[0120] The embedding vector accumulation information (610) may include a plurality of vector dimension sequences (610-1 to 610-512). The plurality of vector dimension sequences (610-1 to 610-512) may represent a sequence of accumulated values ​​for each of the dimensions of 512 embedding vectors accumulated over a period of three months. The horizontal axis of each vector dimension sequence may be the time axis, and the vertical axis may be the value of the dimension.

[0121] FIG. 6B illustrates embedding vector distribution information (620). The embedding vector distribution information (620) may be information showing the mean and standard deviation for each vector dimension sequence included in the embedding vector accumulation information (610). In particular, the embedding vector distribution information (620) may include information on the mean and standard deviation of each dimension at a specific time or at a specific point in time.

[0122] In other words, the mean and standard deviation of each dimension can be calculated using the values ​​for the dimension at the same time interval.

[0123] The embedding vector distribution information (620) may include a plurality of vector dimension distributions (620-1 to 620-512). Each of the plurality of vector dimension distributions (620-1 to 620-512) may represent a one-week distribution of the mean and standard deviation of values ​​for each dimension of the embedding vectors accumulated over a period of three months.

[0124] Referring to Figures 6A and 6B, when an embedding vector is extracted from a single modality data (here, log data) through a domain-specific encoder (here, a log encoder), it exhibits discontinuous characteristics with respect to the time axis. In other words, the data values ​​are too scattered, making it difficult to use for analyzing and predicting users' lifestyle patterns.

[0125] Again, Figure 3 is explained.

[0126] A learning processor (240) or a processor (260) can train a log encoder using contrastive learning so that a first embedding vector obtained from a text encoder and a second embedding vector obtained from a log encoder are located at the same point in vector space. Each of the first embedding vector and the second embedding vector may correspond to different types of modality data obtained during the same period.

[0127] Contrastive learning can be referred to as binding learning.

[0128] Contrastive learning can be a method of learning to make similar pairs of vectors closer to each other and dissimilar pairs of vectors farther apart in a vector space.

[0129] A learning processor (240) or a processor (260) can learn a log encoder using two or more types of modality data collected during the same period so that pairs of embedding vectors corresponding to two or more types of modality data are located at the same point in the embedding space.

[0130] A learning processor (240) or a processor (260) can learn a log encoder using multiple modality data of different types so that the embedding vector of each of the multiple modality data has the same value.

[0131] The log classifier can infer situations that match the second embedding vector transformed through the log encoder.

[0132] FIG. 7 is a diagram illustrating the process of learning a log encoder through comparative learning according to one embodiment of the present disclosure.

[0133] Referring to FIG. 7, the learning processor (240) or processor (260) can train the log encoder (500-1) such that the embedding vector output from text data through the text encoder (400) and the embedding vector output from knowledge data through the log encoder (500-1) are located at the same point in the vector space.

[0134] For example, a learning processor (240) or a processor (260) can train the log encoder (500-1) such that the embedding vector output through the log encoder (500-1) from the first training knowledge data (520) is located at the same point in vector space as the embedding vector output through the text encoder (400) from the first training text data (410). The first training knowledge data (520), which is the basis of the first training text data (410) and the first training log data (511), may be data based on user lifestyle pattern data collected during the same period.

[0135] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-1) such that the embedding vector output through the log encoder (500-1) from the second training knowledge data (530) is located at the same point in vector space as the embedding vector output through the text encoder (400) from the second training text data (420). The second training knowledge data (530), which forms the basis of the second training text data (420) and the second training log data (512), may be data based on user lifestyle pattern data collected during the same period.

[0136] The learning processor (240) or processor (260) can train the log encoder (500-1) such that the embedding vector output through the log encoder (500) from the third training knowledge data (540) is located at the same point in vector space as the embedding vector output through the text encoder (400) from the third training text data (430). The third training knowledge data (540), which is the basis of the third training text data (430) and the third training log data (513), may be data based on user lifestyle pattern data collected during the same period.

[0137] The learning processor (240) or processor (260) can transmit the learned log encoder (500) to the AI ​​device (100) through the communication circuit (210).

[0138] Again, Figure 3 is explained.

[0139] The learning processor (240) or processor (260) of the AI ​​server (200) can transmit a log encoder among a plurality of encoders learned through the communication circuit (210) to the AI ​​device (100) (S303).

[0140] The AI ​​server (200) can transmit a learned log encoder to the AI ​​device (100) as shown in FIG. 7. The AI ​​server (200) can transmit information about parameters, such as the weights of the log encoder, to the AI ​​device (100). The AI ​​server (200) can transmit information necessary to mount the log encoder on the AI ​​device (100) to the AI ​​device (100).

[0141] The processor (180) of the AI ​​device (100) can acquire a log data set for a specific period (S305).

[0142] The specific period may be 3 months, but this is merely an example.

[0143] In one embodiment, the log data set acquired during a specific period may be the data set used to train the log encoder in the AI ​​server (200).

[0144] In another embodiment, the log data set obtained during a specific period may be a data set collected by the AI ​​device (100) during a specific period.

[0145] A log data set may be a collection of log data collected over a unit period. The unit period may be 10 minutes, but this is merely an example.

[0146] The processor (180) of the AI ​​device (100) can generate embedding vector distribution information based on the acquired log data set (S307).

[0147] The processor (180) can convert each log data into knowledge data and can generate an embedding vector from the knowledge data converted through a log encoder.

[0148] The processor (180) can obtain embedding vector distribution information using multiple embedding vectors obtained during a specific period.

[0149] Step S307 is explained with reference to Fig. 8.

[0150] FIG. 8 is a flowchart illustrating the process of obtaining embedding vector distribution information according to one embodiment of the present disclosure.

[0151] The processor (180) of the AI ​​device (100) can convert log data into knowledge data (S801).

[0152] In one embodiment, the processor (180) may receive log data from one or more home appliances. The log data may be data collected over a certain period or collected at a specific point in time. It may be a unit period of a certain period. The unit period may be 10 minutes, but this is merely an example.

[0153] In another embodiment, the processor (180) may receive log data from a management server that manages one or more home appliances. The processor (180) may convert the log data into knowledge data through either a log parser (not shown) or a log converter (not shown).

[0154] Log data can be obtained from at least one of a home appliance, a sensor, or a camera.

[0155] The processor (180) can acquire log data in units of a set period. The set period may be 10 minutes, but this is merely an example. Log data for 10 minutes can later be converted into a single embedding vector.

[0156] The processor (180) of the AI ​​device (100) can obtain an embedding vector from knowledge data through a log encoder received from the AI ​​server (200) (S803).

[0157] The processor (180) can obtain an embedding vector from knowledge data through a log encoder received from the AI ​​server (200).

[0158] In another embodiment, the processor (180) may use the output data of an LLM model for log data as input data to a log encoder. The processor (180) may also generate an embedding vector from the output data of an LLM model through a log encoder.

[0159] The processor (180) can match a plurality of embedding vectors and context data corresponding to each of the plurality of embedding vectors and store them in memory (170). The context data corresponding to each embedding vector may be data representing a specific situation. The context data may be any one of log data, knowledge data, or text data representing a situation matched to the log data.

[0160] Multiple embedding vectors stored in memory (170) may be referred to as an embedding vector matrix.

[0161] The processor (180) of the AI ​​device (100) can generate embedding vector distribution information based on embedding vectors acquired during a specific period (S805).

[0162] The processor (180) can generate embedding vector distribution information based on the values ​​of 512 dimensions included in each of the embedding vectors acquired during a specific period.

[0163] The processor (180) can store the generated embedding vector distribution information in memory (170).

[0164] FIG. 9A is a diagram illustrating embedding vector accumulation information output through a log encoder according to an embodiment of the present disclosure, and FIG. 9B is a diagram illustrating embedding vector distribution information based on embedding vector accumulation information.

[0165] Referring to FIG. 9A, embedding vector accumulation information (910) having time-series characteristics is illustrated. The log encoder (500-1) may be an encoder learned through contrast learning with a text encoder.

[0166] The embedding vector accumulation information (910) can represent the accumulated results of embedding vectors output from the log encoder (500-1) over a specific period plotted in a window of one week.

[0167] The embedding vector accumulation information (910) may include a plurality of vector dimension sequences (910-1 to 910-512). The plurality of vector dimension sequences (910-1 to 910-512) may represent a sequence of values ​​for each dimension of the embedding vector accumulated over a period of three months. The horizontal axis of each vector dimension sequence may be the time axis, and the vertical axis may be the value of the dimension.

[0168] When comparing the embedding vector accumulation information (910) of FIG. 9A with the embedding vector accumulation information (610) obtained using only the log data of FIG. 6A, it can be seen that the flow of embedding vectors over time is more continuously aligned.

[0169] Referring to FIG. 9B, embedding vector distribution information (920) obtained based on the embedding vector accumulation information (910) of FIG. 9A is shown. The embedding vector distribution information (920) may be information showing the mean and standard deviation for each vector dimension sequence included in the embedding vector distribution information (910).

[0170] The embedding vector distribution information (920) may include a plurality of vector dimension distributions (920-1 to 920-512). Each of the plurality of vector dimension distributions (920-1 to 920-512) may represent a one-week distribution of the mean and standard deviation of values ​​for each dimension of the embedding vectors accumulated over a period of three months.

[0171] Again, Figure 3 is explained.

[0172] The processor (180) of the AI ​​device (100) can predict an embedding vector of a specific time period based on embedding vector distribution information (S309).

[0173] The processor (180) can determine the embedding vector based on the distribution of values ​​of each dimension corresponding to a specific time range included in the embedding vector distribution information (920).

[0174] In one embodiment, the processor (180) may determine the average value of the values ​​of each dimension corresponding to a specific time period included in the embedding vector distribution information (920) as the value of the corresponding dimension. However, this is merely an example, and other methods for determining the value of the dimension of the embedding vector may be used.

[0175] For example, referring to FIG. 9B, the processor (180) can determine the average value of the accumulated values ​​of the first dimension from the first vector dimension distribution (920-1) at the time of 8:00 PM on Friday as the value of the first dimension of the embedding vector. In this way, the processor (180) can determine 512 values ​​(930-1 to 930-512) of 512 dimensions corresponding to the time of 8:00 PM on Friday as the embedding vector.

[0176] The processor (180) of the AI ​​device (100) can obtain context data corresponding to the predicted embedding vector (S311).

[0177] The processor (180) can extract the embedding vector with the greatest similarity to the determined embedding vector from the memory (170). The processor (180) can compare a plurality of embedding vectors stored in the memory (170) with the determined embedding vector, and can extract the embedding vector with the greatest similarity among the plurality of embedding vectors.

[0178] The processor (180) can extract context data corresponding to the embedding vector with the greatest similarity from memory (170).

[0179] The processor (180) of the AI ​​device (100) can provide a service based on acquired context data (S313).

[0180] The processor (180) can provide a preemptive service to the user based on context data. The preemptive service may be either a service that sends an alarm to the user's terminal or an automatic setting service for home appliances, but this is merely an example.

[0181] Context data may include information about the user's predicted lifestyle patterns at a specific point in time or during a specific period.

[0182] For example, if the acquired context data is <Friday 8 PM, start cooking>, the processor (180) can turn on the power of the oven or induction when it becomes Friday 8 PM.

[0183] As such, according to an embodiment of the present disclosure, a situation can be comprehensively understood by utilizing multimodal data having various forms and characteristics, and the understood situation can be treated as sequential data to predict user patterns.

[0184] Accordingly, customized services tailored to the user's personalized lifestyle patterns can be provided, which can significantly improve user convenience.

[0185] FIG. 10 is a diagram illustrating the results of situation inference representing a user's lifestyle pattern over a week corresponding to embedding vectors determined according to one embodiment of the present disclosure.

[0186] FIG. 10 shows a situation inference result (1000) corresponding to the embedding vectors predicted based on the embedding vector distribution information (920) of FIG. 9B. The situation inference result (1000) may be the result of interpreting the predicted embedding vectors.

[0187] The situation inference result (1000) may be context data representing the user's lifestyle pattern over a week.

[0188] The situation inference result (1000) may include information on the user's lifestyle pattern for the morning time of each day of the week, the user's lifestyle pattern for the evening time of each day of the week, and the bedtime of each day of the week.

[0189] The text generator can generate a text-based situation inference result (1000) based on log data that matches the embedding vectors. The text generator may be included in the processor (180) or configured separately.

[0190] The situation inference result (1000) can be input into a large language model and used to obtain information desired by the user.

[0191] Figure 11 is a diagram illustrating the process of learning a log encoder through contrast learning using an image encoder, and Figure 12 is a diagram showing embedding vector distribution information based on embedding vectors output by the image encoder and the bound learned log encoder.

[0192] Referring to FIG. 11, the learning processor (240) or processor (260) of the AI ​​server (200) can train the log encoder (500-2) so that the embedding vector output from image data through the image encoder (1100) and the embedding vector output from knowledge data through the log encoder (500-2) are located at the same point in the vector space.

[0193] For example, a learning processor (240) or a processor (260) can train the log encoder (500-2) such that the embedding vector output through the log encoder (500-2) from the first training knowledge data (520) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the first training image data (1110). The first training image data (1110) and the first training knowledge data (520) may be data based on user lifestyle pattern data collected during the same period.

[0194] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-2) such that the embedding vector output through the log encoder (500-2) from the second training knowledge data (530) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the second training image data (1120). The second training image data (1120) and the second training knowledge data (530) may be data based on user lifestyle pattern data collected during the same period.

[0195] The learning processor (240) or processor (260) can train the log encoder (500-2) such that the embedding vector output through the log encoder (500-2) from the third training knowledge data (540) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the third training image data (1130). The third training image data (1130) and the third training knowledge data (540) may be data based on user lifestyle pattern data collected during the same period.

[0196] The learning processor (240) or processor (260) can transmit the learned log encoder (500-2) to the AI ​​device (100) through the communication circuit (210).

[0197] That is, the log encoder (500-2) may be trained through contrast learning with an image encoder (1100) that uses image data rather than a text encoder (400).

[0198] Referring to FIG. 12, embedding vector distribution information (1200) based on embedding vectors output by a log encoder (500-2) learned through contrast learning with an image encoder (1100) is shown.

[0199] When comparing the embedding vector distribution information (1200) of FIG. 12 with the embedding vector distribution information (900) of FIG. 9B, the variance value of each dimension in the case of the embedding vector distribution information (1200) may be smaller overall compared to the case of the embedding vector distribution information (900) of FIG. 9B. This may be a result of the diversity of image data being greater than the diversity of text data.

[0200] The processor (180) can determine the embedding vector for a specific time period based on the embedding vector distribution information (1200).

[0201] FIG. 13 is a diagram illustrating the process of learning a log encoder through contrast learning using an image encoder and a text encoder, and FIG. 14 is a diagram showing embedding vector distribution information based on embedding vectors output by the image encoder and the text encoder and the bound learned log encoder.

[0202] Referring to FIG. 13, the learning processor (240) or processor (260) of the AI ​​server (200) can train the log encoder (500-3) so that the embedding vector output from image data through the image encoder (1100), the embedding vector output from text data through the text encoder (400), and the embedding vector output from knowledge data through the log encoder (500-3) are located at the same point in the vector space.

[0203] For example, a learning processor (240) or a processor (260) can train the log encoder (500-3) such that the embedding vector output through the log encoder (500-3) from the first training knowledge data (520) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the first training image data (1110) and the embedding vector output through the text encoder (400) from the first training text data (410). The first training image data (1110), the first training text data (410), and the first training knowledge data (520) may be data based on user lifestyle pattern data collected during the same period.

[0204] Likewise, the learning processor (240) or processor (260) can train the log encoder (500-3) such that the embedding vector output through the log encoder (500-3) from the second training knowledge data (530) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the second training image data (1120) and the embedding vector output through the text encoder (400) from the second training text data (420). The second training image data (1120), the second training text data (420), and the second training knowledge data (530) may be data based on user lifestyle pattern data collected during the same period.

[0205] The learning processor (240) or processor (260) can train the log encoder (500-3) such that the embedding vector output through the log encoder (500-3) from the third training knowledge data (540) is located at the same point in vector space as the embedding vector output through the image encoder (1100) from the third training image data (1130) and the embedding vector output through the text encoder (400) from the third training text data (430). The third training image data (1130), the third training text data (430), and the third training knowledge data (540) may be data based on user lifestyle pattern data collected during the same period.

[0206] The learning processor (240) or processor (260) can transmit the learned log encoder (500-3) to the AI ​​device (100) through the communication circuit (210).

[0207] That is, the log encoder (500-3) may be learned through contrast learning with the text encoder (400) and image encoder (1100).

[0208] Referring to FIG. 14, embedding vector distribution information (1400) based on embedding vectors output by a log encoder (500-3) learned through contrast learning with an image encoder (1100) and a text encoder (400) is shown.

[0209] When comparing the embedding vector distribution information (1400) of FIG. 14 with the embedding vector distribution information (900) of FIG. 9B and the embedding vector distribution information (1200) of FIG. 12, the variance value of each dimension may be smaller overall in the case of the embedding vector distribution information (1400) of FIG. 14 compared to the case of the embedding vector distribution information (900) of FIG. 9B and the embedding vector distribution information (1200) of FIG. 12. This indicates that as the number of encoders using different types of modality data bound to the log encoder (500-3) increases, it may be possible to predict a stable situation.

[0210] The processor (180) can determine the embedding vector for a specific time period based on the embedding vector distribution information (1400).

[0211] FIG. 15 is a block diagram illustrating the configuration of an artificial intelligence device according to another embodiment of the present disclosure.

[0212] Referring to FIG. 15, the edge device (1500) may include a log converter (1510), a log encoder (1520), a log embedding matrix DB (1530), a behavior predictor (1540), a log classifier (1550), a log search DB (1560), a DB manager (1570), and a text generator (1580).

[0213] The log converter (1510) can convert log data into knowledge data.

[0214] The log encoder (1520) can convert knowledge data into embedding vectors.

[0215] The log encoder (1520) may be an encoder learned through contrast learning using at least one of a text encoder, an image encoder, and a sensor encoder.

[0216] The log embedding matrix DB (1530) can store embedding vectors output from the log encoder (1520). Each embedding vector can be stored with a matching log data collection period.

[0217] The behavior predictor (1540) can generate embedding vector distribution information based on embedding vectors and can predict embedding vectors of a specific time period through the embedding vector distribution information.

[0218] The log classifier (1550) can classify the embedding vector output from the log encoder (1520). The log classifier (1550) can match the embedding vector to log data or knowledge data based on log data and store it in the log search DB (1560).

[0219] The log search DB (1560) can store multiple embedding vectors and log data or knowledge data that matches each embedding vector.

[0220] The DB manager (1570) can extract the embedding vector most similar to the embedding vector of a specific time period predicted by the behavior predictor (1540) among the multiple embedding vectors stored in the log search DB (1560).

[0221] The DB manager (1570) can obtain log data that matches the embedding vector extracted from the log search DB (1560).

[0222] The DB manager (1570) can transmit log data of the acquired embedding vector and period data for a specific period to the text generator (1580).

[0223] The text generator (1580) can output a situation inference result (1000) such as Fig. 10 based on input data.

[0224] The edge device (1500) may be an example of the artificial intelligence device (100) of FIG. 1.

[0225] The log converter (1510), log encoder (1520), behavior predictor (1540), log classifier (1550), DB manager (1570), and text generator (1580) may be included in the processor (180) of FIG. 1 or may be separate components from the processor (180).

[0226] The log embedding matrix DB (1530) and the log search DB (1560) may be included in the memory (170) of FIG. 1, or may be separate components from the memory (170).

[0227] FIG. 16 is a diagram illustrating the process of predicting future embedding vectors from embedding vectors through a time series model according to one embodiment of the present disclosure.

[0228] Referring to FIG. 16, the log encoder (500-3) may be an encoder learned through contrast learning with the text encoder (400) and the image encoder (1100).

[0229] The log encoder (500-3) can periodically embed log data into an embedding space (1610). The log encoder (500-3) can periodically convert collected log data into an embedding vector.

[0230] The time series model (1620) can infer future embedding vectors from a series of embedding vectors. The inferred future embedding vectors are restored (1630) to the embedding space (1610) and can be used to infer future situations.

[0231] However, the embedding vector input to the time series model (1620) has a problem in that the prediction performance of the time series model (1620) is degraded due to the presence of noise, discontinuities, and missing data.

[0232] Figure 17 is a diagram illustrating embedding vector distribution information based on embedding vector accumulation information obtained based on the embedding vectors of Figure 16.

[0233] FIG. 17 shows embedding vector distribution information (1700) obtained based on embedding vectors located in the embedding space (1610).

[0234] The embedding vector distribution information (1700) may include a plurality of vector dimension distributions (1700-1 to 1700-512). Each of the plurality of vector dimension distributions (1700-1 to 1700-512) may represent a one-week distribution of the mean and standard deviation of the values ​​of each dimension of the embedding vectors accumulated over a period of 13 weeks.

[0235] The dark dot (1700-1) of the embedding vector distribution information (1700) may represent the average value of values ​​for which the standard deviation of the distribution of values ​​of a specific dimension at a specific time is 0.01 or less.

[0236] Referring to FIG. 17, a repetitive pattern of dark dots over time is clearly visible for each dimension. However, when this pattern is input into the time series model (1620), the performance of the time series model (1620) may be degraded because it is actually input as discontinuous data.

[0237] Figure 18 is a diagram illustrating the prediction results output through a time series model from the embedding vector distribution information of Figure 17.

[0238] The time series model (1620) can output a prediction result (1810) for the situation after the current time point (t1).

[0239] The prediction result (1810) may be the result for the value of the third dimension of the embedding vector. By referring to the prediction result (1810), it can be confirmed that discontinuous characteristics remain.

[0240] In this disclosure, a method for serializing embedding vectors along the time axis is proposed to improve the performance of a time series model (1620).

[0241] FIG. 19 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0242] The processor (180) may include one or more hardware processors.

[0243] Referring to FIG. 19, the processor (180) of the artificial intelligence device (100) can periodically acquire log data representing the user's lifestyle pattern (S1901).

[0244] The processor (180) can periodically acquire log data over a specific period. The specific period may be 3 months or 13 weeks, but this is merely an example. The processor (180) can acquire log data at unit time intervals. The unit time interval may be 10 minutes, but this is merely an example.

[0245] The processor (180) can obtain embedding vectors in the embedding space from log data periodically obtained through a log encoder (S1903).

[0246] In one embodiment, each of the embedding vectors may be a vector obtained in a time series.

[0247] The processor (180) can generate an embedding vector from log data corresponding to a unit time through a log encoder. The log encoder (500-1, 500-2, 500-3, 1520) may be an encoder learned through contrast learning using at least one of a text encoder (400), an image encoder (1100), and a sensor encoder.

[0248] Log encoders (500-1, 500-2, 500-3, 1520) can be received from the AI ​​server (200).

[0249] The processor (180) can place the acquired embedding vectors in an embedding space (a first embedding space). The embedding vectors placed in the embedding space can represent a specific situation.

[0250] The number of dimensions (or channels) of each embedding vector can be 512, but this is merely an example.

[0251] The processor (180) can convert the acquired embedding vectors into first time-series continuous data based on a multi-dimensional projection matrix (MPM) and convert the acquired embedding vectors into smoothing data based on a smoothing algorithm (S1905).

[0252] The processor (180) can obtain first time series continuous data from embedding vectors using an MPM. The MPM may be a matrix used to project high-dimensional data into a low-dimensional space. The MPM may be a matrix that reduces the dimensionality of the data to facilitate the analysis and visualization of the data.

[0253] An MPM can be a matrix for generating continuous data by readjusting the distances between values ​​of each dimension.

[0254] MPM can be a txt matrix. t can be 3, but this is just an example and can be an adjustable value.

[0255] The processor (180) can generate time series continuous unit data corresponding to values ​​of specific dimensions of the embedding vectors. If the number of dimensions of the embedding vectors is 512, 512 time series unit data can be generated for each dimension. That is, the first time series continuous data can include 512 time series unit data.

[0256] The first time series continuous data can be represented in the form of a vector. The processor (180) can embed the first time series continuous data in an embedding space different from the embedding space of the log data. That is, the log data can be embedded in the first embedding space, and the first time series continuous data can be embedded in the second embedding space.

[0257] The processor (180) can generate smoothing data from embedding vectors using a smoothing algorithm. Any one of the following algorithms may be used: Wavelet Transform Smoothing algorithm, Fourier Transform Smoothing algorithm, or Gaussian Process (GP) Smoothing algorithm.

[0258] If the number of dimensions of the embedding vector is 512, 512 smoothing unit data can be generated. The smoothing data can include 512 smoothing unit data.

[0259] Smoothing data can be the correct answer data obtained through a smoothing algorithm.

[0260] Meanwhile, the processor (180) can preprocess the embedding vectors to remove noise before converting the acquired embedding vectors into time-series continuous data and smoothing data.

[0261] FIG. 20 is a diagram illustrating the process of preprocessing embedding vectors through a sequence split method according to one embodiment of the present disclosure.

[0262] Referring to FIG. 20, the first to third embedding vector sequences (2010, 2020, 2030) are illustrated.

[0263] Each embedding vector sequence may be a combination of embedding vectors of log data acquired periodically.

[0264] Each embedding vector sequence may include at least one normal vector (2001) that does not have a missing value and an abnormal vector (2002) that has a missing value.

[0265] The processor (180) may discard part or all of the embedding vector sequence according to the following rules.

[0266] There must be at least 5 normal vectors (2001) before 3 abnormal vectors (2002) appear in succession. This rule may be referred to as the split sequence method.

[0267] In the case of the first embedding vector sequence (2010), since only four normal vectors (2001) exist before three consecutive abnormal vectors (2002) appear, the processor (180) can discard (or remove) the entire first embedding vector sequence (2010).

[0268] In the case of the second embedding vector sequence (2020), since there are five normal vectors (2001) before three consecutive abnormal vectors (2002) appear, the processor (180) can discard the embedding vectors obtained after the second embedding vector sequence (2020).

[0269] In the case of the third embedding vector sequence (2030), since the abnormal vector (2002) does not appear three times in succession, the embedding vectors obtained after the third embedding vector sequence (2030) may not be discarded.

[0270] As such, according to an embodiment of the present disclosure, missing data can be removed by applying a split sequence method to an embedding vector sequence. Accordingly, predictive analysis using log data can be made accurate.

[0271] Again, Fig. 19 is explained.

[0272] The processor (180) can determine whether the convergence condition between the first time series continuous data and the smoothing data is satisfied (S1907).

[0273] The processor (180) can determine whether the convergence condition between the first time series continuous data and the smoothing data corresponding to each dimension of the embedding vector is satisfied.

[0274] The processor (180) can adjust the weights of the MPM through a loss function so that the difference between the first time series continuous data and the smoothing data satisfies a convergence condition. The weights of the MPM may be the values ​​of the elements of the MPM.

[0275] The processor (180) can adjust the weights of the MPM so that the first time series continuous data is similar to the smoothing data.

[0276] The loss function may be a Mean Squared Error (MSE) function. The processor (180) can adjust the weights of the MPM so that the loss function corresponding to the difference between the first time series continuous data and the smoothing data is minimized.

[0277] The processor (180) may determine that the convergence condition is satisfied if the value of the loss function representing the difference between the first time series continuous data and the smoothing data is less than a certain value. The processor (180) may determine the weight of the MPM that satisfies the convergence condition as the weight of the final MPM.

[0278] The processor (180) can adjust the weight of the MPM if the convergence condition between the first time series continuous data and the smoothing data is not satisfied (S1909).

[0279] The processor (180) can adjust the value of one or more elements constituting the MPM when the convergence condition between the time series continuous data and the smoothing data is not satisfied.

[0280] When the processor (180) determines that the convergence condition between the first time series continuous data and the smoothing data is satisfied, it can obtain the second time series continuous data through the weight of the adjusted MPM (S1911).

[0281] The processor (180) can obtain second time series continuous data similar to smoothing data by using the weights of the adjusted MPM. The second time series continuous data can also be embedded in a second embedding space.

[0282] The processor (180) can predict embedding vectors from the second time series continuous data through a time series model (S1913).

[0283] A time series model may be an artificial neural network-based model trained through deep learning algorithms or machine learning algorithms. A time series model may be a model that predicts future embedding vectors by analyzing past data over time.

[0284] The processor (180) can restore the predicted embedding vectors into the embedding space (S1915).

[0285] The processor (180) can use a multi-dimensional projection inverse matrix to restore the predicted embedding vectors from the second embedding space to the first embedding space.

[0286] The processor (180) can obtain a predicted situation based on the restored embedding vectors (S1917).

[0287] The processor (180) can extract the embedding vector with the greatest similarity to each of the restored embedding vectors in the first embedding space, obtain a situation matching the extracted embedding vector, and predict the obtained situation as a future situation.

[0288] FIG. 21 is a diagram illustrating the process of learning the difference between the first time series continuous data and the smoothing data and the process of restoring the embedding vectors according to an embodiment of the present disclosure.

[0289] Referring to FIG. 21, T embedding vectors (2100-1 to 2100-T) preprocessed according to the split sequence method are shown. The number of dimensions (or channels) of each embedding vector may be 512.

[0290] The processor (180) can divide T embedding values ​​(2110) of the same dimension for embedding vectors (2100-1 to 2100-T) into 1xt matrices (2110-1 to 2110-T) and sequentially perform operations on each of the divided 1xt matrices (2110-1 to 2110-T) with the MPM (2130). Here, t may be a value corresponding to the dimension of the MPM.

[0291] T embedding values ​​(2110) of the same dimension can be represented as a 1xT matrix. The T embedding values ​​(2110) can be 512-dimensional values ​​corresponding to each of the embedding vectors (2100-1 to 2100-T).

[0292] T embedding values ​​(2110) of the same dimension can have an irregular first pattern (2120).

[0293] MPM(2130) can be a txt matrix. t can be 3, but this is just an example.

[0294] The processor (180) can obtain 1xt projection matrices (2140-1 to 2140-T) by sequentially performing operations with the MPM (2130) on each of the divided 1xt matrices (2110-1 to 2110-T).

[0295] The processor (180) can obtain T projection values ​​(2150) for the same dimension from 1xt projection matrices (2140-1 to 2140-T). The T projection values ​​(2150) for the same dimension may be referred to as time series unit data.

[0296] The processor (180) can obtain T time series continuous unit data corresponding to each of the multiple dimensions in this manner, and can obtain T projection vectors (2170-1 to 2170-T) through the T time series continuous unit data.

[0297] T projection vectors (2170-1 to 2170-T) can be referred to as time series continuous data.

[0298] Meanwhile, the processor (180) can generate smoothing data by applying a smoothing algorithm to the same T embedding values ​​(2110). The smoothing data may be referred to as Ground Truth (GT) data. The smoothing data may be the correct answer data that the time series continuous data must follow.

[0299] The processor (180) can adjust the weights of the MPM so that the loss function representing the difference between the time series continuous data and the smoothing data is minimized. The weights of the MPM may be the values ​​of the elements constituting the MPM.

[0300] The processor (180) can adjust the value of one or more elements of the MPM so that the loss function representing the difference between the time series continuous data and the smoothing data is minimized.

[0301] The processor (180) can determine the weight of the MPM as the final weight when the value of the loss function is less than a certain value.

[0302] The processor (180) can generate second time series continuous data from T embedding vectors (2100-1 to 2100-T) through an MPM having final weights. T projection values ​​of the same dimension for the second time series continuous data may have a second pattern (2160) that is continuous compared to the first pattern (2120).

[0303] The processor (180) can predict future embedding vectors from the second time series continuous data through the time series model (1620). The processor (180) can restore the future embedding vectors to the first embedding space using a multidimensional projection inverse matrix. The embedding vectors restored to the first embedding space may have a third pattern (2190) that is nearly similar or identical to the first pattern (2120).

[0304] The processor (180) can infer future situations based on embedding vectors restored to the first embedding space. The processor (180) can provide a service corresponding to the inferred situation (context data). This is replaced by the description of step S313 of FIG. 3.

[0305] Accordingly, according to an embodiment of the present disclosure, vectors input to a time series model can be prepared in an optimal state through a process of preprocessing embedding vectors and a process of smoothing embedding vectors. As a result, the problem of performance degradation of the time series model can be resolved.

[0306] In addition, as the performance of the time series model improves, the ability to infer future situations can also be enhanced.

[0307] FIG. 22 is a diagram illustrating the restoration results for the output of the time series prediction data and the time series model generated with the weights of the final MPM, after undergoing a sequence splitting method for the embedding vectors according to an embodiment of the present disclosure.

[0308] In the graph of Fig. 22, the horizontal axis is the time axis, and the vertical axis can represent the values ​​of the data.

[0309] Referring to Fig. 22, raw data can represent original log data.

[0310] The original line may be a line representing time series forecast data generated with the weights of the initial MPM.

[0311] The Smoothed line may be a line representing time series forecast data generated with the weights of the final MPM.

[0312] The inverse line may be a continuous line of embedding vectors representing future situations output by the time series model (1620).

[0313] Referring to Figure 22, it can be seen that the large fluctuations seen in the original raw data are significantly mitigated in the result value after passing through the MPM.

[0314] Furthermore, it can be confirmed that the curvature information of the original raw data is also preserved.

[0315] In addition, it can be confirmed that the original data can be fully restored through the inverse operation of the smoothed value.

[0316] An artificial intelligence device (100) according to one embodiment of the present disclosure may include: a memory (170) storing a first encoder that outputs an embedding vector from a first type of data representing a user's lifestyle pattern, wherein the first encoder is an encoder learned through comparative learning with a second encoder that outputs an embedding vector from a second type of data representing the user's lifestyle pattern; and one or more processors (180) that periodically acquire log data, acquire embedding vectors in an embedding space from the periodically acquired log data through the log encoder, convert the embedding vectors into a first time series continuous data and smoothing data, respectively, and acquire a second time series continuous data that satisfies the condition that the first time series continuous data converges to the smoothing data.

[0317] One or more processors (180) can convert the embedding vectors into the first time series continuous data based on a multi-dimensional projection matrix (MPM).

[0318] The above one or more processors (180) can adjust the weights of the MPM so as to satisfy the condition that the first time series continuous data converges to the smoothing data.

[0319] The above one or more processors (180) can obtain the second time series continuous data from the embedding vectors according to the weights of the adjusted MPM.

[0320] The memory (170) may further include a time series model, and the one or more processors (180) may input the second time series continuous data into the time series model and restore the future embedding vectors output through the time series model into the embedding space using a multidimensional projection inverse matrix.

[0321] The above one or more processors (180) can remove at least some of the embedding vectors based on a split sequence method.

[0322] One or more processors (180) may determine that the condition is satisfied if the value of the loss function representing the difference between the first time series continuous data and the smoothing data is less than a certain value.

[0323] The above-mentioned first type of data and second type of data may be data collected in the same time period.

[0324] The first encoder may be an encoder trained such that the embedding vector output from the first encoder is located at the same point in vector space as the embedding vector output from the second encoder.

[0325] The above first type of data is log data, and the second type of data is

[0326] It can be any one of text data, image data, or sensing data.

[0327] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. Computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0328] Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In an artificial intelligence device, A memory storing a first encoder that outputs an embedding vector from first type data representing a user's lifestyle pattern, wherein the first encoder is an encoder learned through contrast learning with a second encoder that outputs an embedding vector from second type data representing the user's lifestyle pattern; and One or more processors comprising periodically acquiring log data, acquiring embedding vectors in an embedding space from the periodically acquired log data through the log encoder, converting the embedding vectors into first time series continuous data and smoothing data, respectively, and acquiring second time series continuous data satisfying the condition that the first time series continuous data converges to the smoothing data. Artificial intelligence device.

2. In Paragraph 1, The above one or more processors Converting the above embedding vectors into the above first time-series continuous data based on a multi-dimensional projection matrix (MPM). Artificial intelligence device.

3. In Paragraph 2, The above one or more processors Adjusting the weights of the MPM so as to satisfy the condition that the first time series continuous data converges to the smoothing data. Artificial intelligence device.

4. In Paragraph 3, The above one or more processors Acquiring the second time series continuous data from the embedding vectors according to the weights of the adjusted MPM Artificial intelligence device.

5. In Paragraph 4, Memory further includes time series models, The above one or more processors Input the above second time series continuous data into a time series model, and restore the future time point embedding vectors output through the time series model into the embedding space using a multidimensional projection inverse matrix. Artificial intelligence device.

6. In Paragraph 1, The above one or more processors Removing at least some of the above embedding vectors based on the split sequence method Artificial intelligence device.

7. In Paragraph 1, The above one or more processors If the value of the loss function representing the difference between the first time series continuous data and the smoothing data is less than a certain value, it is determined that the above condition is satisfied. Artificial intelligence device.

8. In Paragraph 1, The above-mentioned first type of data and the above-mentioned second type of data are data collected in the same time period. Artificial intelligence device.

9. In Paragraph 1, The above-mentioned first encoder is An encoder trained such that the embedding vector output from the first encoder is located at the same point in vector space as the embedding vector output from the second encoder. Artificial intelligence device.

10. In Paragraph 1, The above first type of data It is log data, and Type 2 data text data, image data, or sensing data Artificial intelligence device.

11. In a method of operating an artificial intelligence device, Step of periodically acquiring log data; A step of obtaining embedding vectors in an embedding space from the log data periodically obtained above; A step of converting the above embedding vectors into first time-series continuous data and smoothing data, respectively; and The step of obtaining second time series continuous data satisfying the condition that the first time series continuous data converges to the smoothing data Method of operation of an artificial intelligence device.

12. In Paragraph 11, The step of converting the above embedding vectors into the first time series continuous data A step comprising converting the above embedding vectors into the first time-series continuous data based on a multi-dimensional projection matrix (MPM). Method of operation of an artificial intelligence device.

13. In Paragraph 12, The method further includes the step of adjusting the weights of the MPM so as to satisfy the condition that the first time series continuous data converges to the smoothing data. Method of operation of an artificial intelligence device.

14. In Paragraph 13, The step of acquiring the above-mentioned second time series continuous data The step of obtaining the second time series continuous data from the embedding vectors according to the weights of the adjusted MPM. Method of operation of an artificial intelligence device.

15. In Paragraph 14, The step of inputting the above second time series continuous data into a time series model; and The method further includes the step of restoring the embedding vectors of future points in time output through the time series model into the embedding space using a multidimensional projection inverse matrix. Method of operation of an artificial intelligence device.

16. In Paragraph 11, The method further includes the step of removing at least some of the embedding vectors based on a split sequence method. Method of operation of an artificial intelligence device.

17. In Paragraph 11, The method further includes a step of determining that the condition is satisfied when the value of the loss function representing the difference between the first time series continuous data and the smoothing data is less than a certain value. Method of operation of an artificial intelligence device.

18. In Paragraph 11, The above-mentioned first type of data and the above-mentioned second type of data are data collected in the same time period. Method of operation of an artificial intelligence device.

19. In Paragraph 11, The above-mentioned first encoder is An encoder trained such that the embedding vector output from the first encoder is located at the same point in vector space as the embedding vector output from the second encoder. Method of operation of an artificial intelligence device.

20. A recording medium having a computer-readable program for performing a method of operating an artificial intelligence device, The method of operation of the above artificial intelligence device Step of periodically acquiring log data, A step of obtaining embedding vectors in an embedding space from the log data periodically obtained above, A step of converting the above embedding vectors into first time-series continuous data and smoothing data, respectively, and The step of obtaining second time series continuous data satisfying the condition that the first time series continuous data converges to the smoothing data Recording media.

Images