Artificial Intelligence Server and Its Method of Operation

Abstract

An artificial intelligence server according to one embodiment of the present disclosure may include a memory for storing a multimodal model that infers a situation using a plurality of modalities having different types, a communication circuit for communicating with a plurality of edge devices or a cloud server, and a processor that receives parameter information for a fine-tuned layer of the multimodal model from the cloud server, receives a plurality of encrypted modalities from the plurality of edge devices, and updates the parameter information of the fine-tuned layer of the multimodal model using the received plurality of encrypted modalities.

Description

Technology Field

[0001] The present invention relates to an artificial intelligence server, and more specifically, to an artificial intelligence server that infers a situation using a multimodal model. Background Technology

[0002] A multimodal model represents a model that processes various types of input data and generates various types of outputs.

[0003] Multimodal models can primarily handle various forms of data simultaneously, such as text, images, and audio. For example, they can generate descriptions by understanding images and captions at the same time, or improve natural language understanding and speech recognition by processing text and voice together.

[0004] Conventionally, there was a technology that encrypted input information at edge devices and transmitted it to an AI server for processing to protect privacy.

[0005] However, conventionally, despite the fact that input information differs for each user, only non-personalized global AI models have been provided.

[0006] In addition, for each modality (each data type), there were no settings for the encoder parameters to encrypt the input information. The problem to be solved

[0007] The purpose of the present disclosure is to provide a personalized multimodal model by transmitting fine-tuned layers of a multimodal model trained by a cloud server to a home server and updating the fine-tuned layers through encrypted input information collected from the home server.

[0008] The purpose of the present disclosure is to train a multimodal model by encrypting modality data using an encryption module (encoder) according to the type of modality. means of solving the problem

[0009] An artificial intelligence server according to one embodiment of the present disclosure may include a memory for storing a multimodal model that infers a situation using a plurality of modalities having different types, a communication circuit for communicating with a plurality of edge devices or a cloud server, and a processor that receives parameter information for a fine-tuned layer of the multimodal model from the cloud server, receives a plurality of encrypted modalities from the plurality of edge devices, and updates the parameter information of the fine-tuned layer of the multimodal model using the received plurality of encrypted modalities.

[0010] An artificial intelligence server according to one embodiment of the present disclosure may have a method of operation of the artificial intelligence server that includes the steps of receiving parameter information for a fine-tuned layer of a multimodal model that infers a situation using a plurality of modalities having different types from a cloud server, receiving a plurality of encrypted modalities from the plurality of edge devices, and updating the parameter information of the fine-tuned layer of the multimodal model using the received plurality of encrypted modalities. Effects of the invention

[0011] According to an embodiment of the present disclosure, user-customized situational reasoning and measures for situational reasoning can be effectively performed according to a personalized multimodal model.

[0012] According to an embodiment of the present disclosure, an accurate situation can be inferred using various types of data, and measures suitable for the inferred situation can be taken. Brief explanation of the drawing

[0013] FIG. 1 shows an AI device according to one embodiment of the present disclosure. FIG. 2 shows an AI server according to one embodiment of the present disclosure. FIG. 3 is a drawing for explaining the configuration of an artificial intelligence system according to one embodiment of the present disclosure. FIG. 4 is a ladder diagram for explaining the operation method of an artificial intelligence system according to one embodiment of the present disclosure. FIG. 5a is a diagram illustrating the process of a cloud server collecting modalities according to one embodiment of the present disclosure, and FIG. 5b is a diagram illustrating the process of a cloud server learning a multimodal model using the collected modalities. FIG. 6 is a diagram illustrating the fine-tuning process of a multimodal model according to one embodiment of the present disclosure. FIG. 7 is a ladder diagram for explaining the operation method of an artificial intelligence system according to an embodiment of the present disclosure. FIGS. 8a to 8c are drawings illustrating a process of inferring a situation from a plurality of modalities using a multimodal model learned according to embodiments of the present disclosure. FIG. 9 is a drawing illustrating an example in which each of a plurality of home servers has a personalized multimodal model according to one embodiment of the present disclosure. Specific details for implementing the invention

[0014] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define and solve various problems addressed within the field of artificial intelligence. Machine learning is also defined as an algorithm that improves performance on a task through continuous experience.

[0015] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0016] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0017] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0018] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0019] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0020] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when the training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0021] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.

[0022] FIG. 1 shows an artificial intelligence (AI) device according to one embodiment of the present disclosure.

[0023] The AI ​​device (100) can be implemented as a stationary device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0024] Referring to FIG. 1, the terminal (100) may include a communication circuit (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180), etc.

[0025] The communication circuit (110) can transmit and receive data with external devices such as other AI devices (100a to 100e) or an AI server (200) using wired and wireless communication technology.

[0026] The communication circuit (110) can transmit and receive sensor information, user input, learning model, control signal, etc., with external devices.

[0027] The communication technologies used by the communication circuit (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0028] The input interface (120) can acquire various types of data.

[0029] The input interface (120) may include a camera for inputting a video signal, a microphone for receiving an audio signal, a user input interface for receiving information from a user, etc.

[0030] Cameras or microphones can be treated as sensors, and signals acquired from cameras or microphones can be referred to as sensing data or sensor information.

[0031] The input interface (120) can obtain training data for model training and input data to be used when obtaining an output using the training model. The input interface (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input features as a preprocessing step for the input data.

[0032] The learning processor (130) can train a model composed of an artificial neural network using training data. Here, the trained artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform an action.

[0033] The learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0034] The learning processor (130) may include memory integrated into or implemented in the AI ​​device (100). Alternatively, the learning processor (130) may be implemented using memory (170), external memory directly coupled to the AI ​​device (100), or memory maintained in an external device.

[0035] The sensor (140) can acquire at least one of internal information of the AI ​​device (100), surrounding environment information of the AI ​​device (100), and user information using a plurality of sensors.

[0036] The sensor (140) may include one or more of a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar, and a radar.

[0037] The output interface (150) can generate output related to visual, auditory, or tactile senses.

[0038] The output interface (150) may include a display unit that outputs visual information, a speaker that outputs auditory information, a haptic module that outputs tactile information, etc.

[0039] The memory (170) can store data that supports various functions of the AI ​​device (100). The memory (170) can store input data, training data, training models, training history, etc. obtained from the input interface (120).

[0040] The processor (180) can determine at least one executable action of the AI ​​device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0041] The processor (180) can control the components of the AI ​​device (100) to perform a determined operation.

[0042] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the AI ​​device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0043] If the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0044] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0045] The processor (180) can obtain intent information corresponding to user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0046] At least one of the STT engine or NLP engine may be composed of an artificial neural network, at least a portion of which is trained according to a machine learning algorithm. Additionally, at least one of the STT engine or NLP engine may be trained by a learning processor (130), trained by a learning processor (240) of an AI server (200), or trained by distributed processing thereof.

[0047] The processor (180) may collect history information, including the operation details of the AI ​​device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information may be used to update a learning model.

[0048] The processor (180) can control at least some of the components of the AI ​​device (100) to run an application stored in memory (170). Furthermore, the processor (180) can operate two or more of the components included in the AI ​​device (100) in combination with each other to run the application.

[0049] FIG. 2 shows an AI server (200) according to one embodiment of the present disclosure.

[0050] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network. Here, the AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network.

[0051] The AI ​​server (200) may be included as part of the configuration of the AI ​​device (100) and may perform at least some of the AI ​​processing together.

[0052] The AI ​​server (200) may include a communication circuit (210), memory (230), a learning processor (240), and a processor (260), etc.

[0053] The communication circuit (210) can transmit and receive data with an external device such as an AI device (100).

[0054] The memory (230) may include a model storage unit (231). The model storage unit (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through a learning processor (240).

[0055] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an AI device (100).

[0056] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0057] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0058] FIG. 3 is a drawing for explaining the configuration of an artificial intelligence system according to one embodiment of the present disclosure.

[0059] The artificial intelligence system (30) may include a cloud server (200-1), a home server (200-2), a first edge device (100-1), and a second edge device (100-2).

[0060] Figure 3 illustrates that the edge device consists of two devices, but this is merely an example, and more edge devices may exist.

[0061] Each of the cloud server (200-1) and the home server (200-2) may include the components of FIG. 2. That is, each of the cloud server (200-1) and the home server (200-2) may be an example of the AI ​​server (200) of FIG. 2. Each of the cloud server (200-1) and the home server (200-2) may be named an artificial intelligence server.

[0062] Each of the first edge device (100-1) and the second edge device (100-2) may include all the components shown in FIG. 1. That is, each of the first edge device (100-1) and the second edge device (100-2) may be an example of the AI ​​device (100) shown in FIG. 1.

[0063] As another example, the first edge device (100-1) and the second edge device (100-2) may each be a component of the AI ​​device (100).

[0064] Each of the first edge device (100-1) and the second edge device (100-2) may be an electronic device capable of collecting modality data, such as a camera, sensor, home appliance, TV, or microphone.

[0065] The first edge device (100-1) and the second edge device (100-2) may be electronic devices used by the same user. The first edge device (100-1) and the second edge device (100-2) may be provided in a specific space, for example, inside a home.

[0066] FIG. 4 is a ladder diagram for explaining the operation method of an artificial intelligence system according to one embodiment of the present disclosure.

[0067] In the following, input information is information collected by edge devices and may be a modality.

[0068] Referring to Fig. 4, The cloud server (200-1) can train a multi-model model (S401), and the cloud server (200-1) can set different encoders for each modality (S403).

[0069] The cloud server (200-1) may have multiple encoders corresponding to each of the multiple modalities. The multiple encoders may be included in the processor (260) or provided separately from the processor (260).

[0070] Each of the multiple encoders may be an encryption module that encrypts the modality received from the edge device (100-1).

[0071] The learning processor (240) or processor (260) of the cloud server (200-1) can set the encoder parameters differently for each modality.

[0072] The learning processor (240) or processor (260) of the cloud server (200-1) can learn multi-model models.

[0073] A multimodal model may be an artificial neural network-based model that infers a situation using multiple modalities as input.

[0074] A multimodal model can be a model trained using contrastive learning.

[0075] Contrastive learning can be a method of learning to make similar sample pairs closer to each other and dissimilar sample pairs farther apart in the embedding space.

[0076] The processor (260) of the cloud server (200-1) can train a multimodal model such that the distance between feature vectors of the same type of modalities is close, and the distance between feature vectors of different types of modalities is far apart.

[0077] Modality can be referred to as multimodal data as input data for a multimodal model.

[0078] Multimodal data may include one or more of image data, text data, sound data, sensor data, and log data (table data or knowledge data) of an edge device (100-1).

[0079] In one embodiment, the cloud server (200-1) can collect multiple modalities from multiple edge devices.

[0080] In another embodiment, multiple modalities may be data generated from a generative AI model of a generative AI server rather than actual data. This is because privacy issues may arise in the case of actual data.

[0081] FIG. 5a is a diagram illustrating the process of a cloud server collecting modalities according to one embodiment of the present disclosure, and FIG. 5b is a diagram illustrating the process of a cloud server learning a multimodal model using the collected modalities.

[0082] The cloud server (200-1) can collect image data, sensing data, sound data and text data.

[0083] The cloud server (200-1) can receive modalities from each of the multiple edge devices.

[0084] Multiple encoders may include an image encoder (501) that encrypts image data, a sensor encoder (503) that encrypts sensing data, a sound encoder (505) that encrypts sound data, and a text encoder (507) that encrypts text data.

[0085] The learning processor (240) or processor (260) of the cloud server (200-1) can train an image encoder (501) corresponding to image data through a Convolutional Neural Network (CNN).

[0086] The learning processor (240) or processor (260) of the cloud server (200-1) can train a sound encoder (505) corresponding to sound data through a recurrent neural network (RNN).

[0087] The learning processor (240) or processor (260) of the cloud server (200-1) can train a text encoder (507) corresponding to text data through the BERT (Bidirectional Encoder Representations from Transformers) algorithm.

[0088] The image encoder (501) can output a first set of vectors (511) that are input to a multimodal model using image data.

[0089] The sensor encoder (503) can output a second vector set (513) that is input to a multimodal model using the sensing data. The sensing data may be millimeter wave signals related to the detection of a person's presence or the person's posture.

[0090] The sound encoder (505) can output a third vector set (515) that is input to a multimodal model using sound data.

[0091] The text encoder (507) can output a set of fourth vectors (517) that are input to a multimodal model using text data.

[0092] The first to fourth sets of vectors (511 to 517) can later be used by a multimodal model to infer the situation. Each of the first to fourth feature vectors (511 to 517) may contain 512 vectors, but this is merely an example.

[0093] In addition, the cloud server (100-1) may collect additional log information of edge devices (100-1) or home appliances. The log information of home appliances may include information regarding the operating status and operating history of the home appliances.

[0094] Referring to FIG. 5b, the cloud server (200-1) can train a multimodal model using the first to fourth sets of vectors (511 to 517).

[0095] The cloud server (200-1) can train a multimodal model through contrast learning. The multimodal model can output an inferred situation using the first to fourth sets of vectors (511 to 517).

[0096] The multimodal model can be trained to gather the first to fourth sets of vectors (511 to 517) into specific data points in the embedding space.

[0097] For example, the multimodal model can infer a cooking situation (Context A) or a TV-watching situation (Context B) using the first to fourth sets of vectors (511 to 517) as input.

[0098] Again, Figure 4 is explained.

[0099] The processor (260) of the cloud server (200-1) can transmit first encoder parameter information according to the settings to the first edge device (100-1) through the communication circuit (210) (S405), and can transmit second encoder parameter information to the second edge device (100-2) through the communication circuit (210) (S407).

[0100] The first encoder parameter information may include information about the parameters of the first encoder.

[0101] The second encoder parameter information may include information about the parameters of the second encoder.

[0102] Parameter information may include parameters that can be used to encrypt input information input to an encoder according to the type of modality.

[0103] The processor (260) of the cloud server (200-1) can transmit first encoder parameter information corresponding to the modality type that the first edge device (100-1) can collect to the first edge device (100-1).

[0104] The processor (260) of the cloud server (200-1) can transmit second encoder parameter information corresponding to the modality type that the second edge device (100-2) can collect to the second edge device (100-2).

[0105] The first edge device (100-1) can encrypt the first input information based on the received first encoder parameter information (S409) and transmit the encrypted first input information to the cloud server (200-1) (S411).

[0106] The first edge device (100-1) can collect one or more modalities. The collected modalities may be data with privacy issues. That is, the modalities may be data with privacy issues, such as image data taken inside the home or sound data of a speaker conversing inside the home.

[0107] The first edge device (100-1) may have a first encoder corresponding to the type of modality it can collect.

[0108] For example, if the first edge device (100-1) is capable of collecting image data, the first edge device (100-1) may be equipped with an image encoder capable of encrypting the image data.

[0109] The first edge device (100-1) can update the parameters of the first encoder based on the first encoder parameter information.

[0110] The first edge device (100-1) can encrypt the first input information using the parameters of the updated first encoder.

[0111] The second edge device (100-2) can encrypt the second input information based on the received second encoder parameter information (S413) and transmit the encrypted second input information to the cloud server (200-1) (S415).

[0112] The second edge device (100-2) can collect one or more modalities. The collected modalities may be data with privacy issues. That is, the modalities may be data with privacy issues, such as image data taken inside the home or sound data of a speaker conversing inside the home.

[0113] The second edge device (100-2) may have a first encoder corresponding to the type of modality it can collect.

[0114] For example, if the second edge device (100-2) is capable of collecting sensing data, the second edge device (100-2) may be equipped with a sensing encoder capable of encrypting the sensing data.

[0115] The second edge device (100-2) can update the parameters of the second encoder based on the second encoder parameter information.

[0116] The second edge device (100-2) can encrypt the second input information using the parameters of the updated second encoder and transmit the encrypted second input information to the cloud server (200-1).

[0117] The cloud server (200-1) can fine-tune the layers of the multimodal model based on the encrypted first and second input information (S417).

[0118] The encrypted input information may be one or more of the multiple modalities.

[0119] A multimodal model may include multiple layers. The multiple layers may include an input layer that is input to the multimodal model, a hidden layer that extracts features of data input to the input layer, and an output layer that outputs a situation inferred using the extracted features.

[0120] Each of the input layer, hidden layer, and output layer may consist of one or more layers.

[0121] The learning processor (240) or processor (260) of the cloud server (200-1) can fine-tune the parameters of one or more layers among the multiple layers constituting the multimodal model based on encrypted input information.

[0122] The parameters of a layer can be weights.

[0123] Fine-tuning can be the process of adjusting some of the parameters of the layers that constitute a pre-trained multimodal model to suit a new purpose (or task).

[0124] Fine-tuning can be a process of improving the situation inference performance of a multimodal model by adjusting various parameters such as the amount of data used for training, the learning rate, and epochs.

[0125] The cloud server (200-1) can generate a personalized multimodal model through first input information received from the first edge device (100-1) and second input information received from the second edge device (100-2). The personalized multimodal model may be a model in which the parameters of some layers are adjusted based on the first and second input information, which are encrypted.

[0126] FIG. 6 is a diagram illustrating the fine-tuning process of a multimodal model according to one embodiment of the present disclosure.

[0127] Referring to FIG. 6, the first edge device (100-1) may be a camera that collects image data. The first edge device (100-1) may include an image encoder (610) that encrypts the image data.

[0128] The first edge device (100-1) can collect first input information and generate encrypted first input information through an image encoder (610). The encrypted first input information may be a first vector set containing 512 vectors. The first edge device (100-1) can transmit the encrypted first input information to a cloud server (200-1).

[0129] The second edge device (100-2) may be a sensor that collects sensing data. The second edge device (100-2) may include a sensor encoder (630) that encrypts the sensing data.

[0130] The second edge device (100-2) can collect second input information and generate encrypted second input information through a sensor encoder (630). The encrypted second input information may be a second vector set containing 512 vectors. The second edge device (100-2) can transmit the encrypted second input information to a cloud server (200-1).

[0131] The cloud server (200-1) can fine-tune some layers of the multimodal model based on the first input information received from the first edge device (100-1) and the second input information received from the second edge device (100-2).

[0132] The cloud server (200-1) can fine-tune some layers of the multimodal model so that the multimodal model, having received encrypted first and second input information, infers that it is a specific situation (Context A). The specific situation may be a situation where the user is cooking.

[0133] Again, Figure 4 is explained.

[0134] The cloud server (200-1) can transmit parameter information of the fine-tuned layer to the home server (200-2) through the communication circuit (210) (S419).

[0135] The processor (260) of the cloud server (200-1) can transmit information about the parameters updated according to the fine-tuning of the layers of the multimodal model to the home server (200-2) through the communication circuit (210).

[0136] The parameter information of a fine-tuned layer may include the weights of the parameters that constitute the layer. The weights may have values ​​between 0 and 1, and the number of weights may be 262144 (512 x 512).

[0137] The cloud server (200-1) can transmit parameter information for a layer that has undergone different fine-tuning for each home server to each home server. That is, the cloud server (200-1) can perform fine-tuning of a multimodal model through multiple modalities received from edge devices (100-1, 100-2) connected to the home server (200-2). Accordingly, a personalized multimodal model can be provided to each home server.

[0138] In another embodiment, the processor (260) of the cloud server (200-1) may transmit a multimodal model to the home server (200-2). The processor (260) of the cloud server (200-1) may transmit parameter information for multiple layers constituting the multimodal model to the home server (200-2).

[0139] The home server (200-2) can store parameter information of a finely tuned layer received from the cloud server (200-1) or parameter information of multiple layers constituting a multimodal model in memory (230).

[0140] The home server (200-2) can set parameters for layers of a stored multimodal model using parameter information of a finely tuned layer received from the cloud server (200-1).

[0141] After that, the first edge device (100-1) encrypts the third input information (S421) and transmits the encrypted third input information to the home server (200-2) (S423).

[0142] The second edge device (100-2) encrypts the fourth input information (S425) and transmits the encrypted fourth input information to the home server (200-2) (S427).

[0143] The home server (200-2) can update the layers of the multimodal model itself based on the encrypted third and fourth input information (S429).

[0144] The learning processor (240) or processor (260) of the home server (200-2) can fine-tune one or more layers among the multiple layers constituting the multimodal model based on encrypted third and fourth input information.

[0145] The learning processor (240) or processor (260) of the home server (200-2) can obtain a third vector set using a first encoder corresponding to the third input information, and can obtain a fourth vector set using a second encoder corresponding to the fourth input information.

[0146] The learning processor (240) or processor (260) of the home server (200-2) can adjust the weights of parameters of one or more layers among the multiple layers constituting the multimodal model using the third and fourth vector sets.

[0147] FIG. 7 is a ladder diagram for explaining the operation method of an artificial intelligence system according to an embodiment of the present disclosure.

[0148] Figure 4 is a diagram illustrating the learning and fine-tuning process of a multimodal model, and Figure 7 is a diagram illustrating the process of outputting situational inference using a fine-tuned multimodal model.

[0149] The embodiment of Fig. 7 may occur after the embodiment of Fig. 4.

[0150] Referring to Fig. 7, The first edge device (100-1) can transmit the encrypted first modality to the home server (200-2) (S701), and the second edge device (100-2) can transmit the encrypted second modality to the home server (200-2) (S703).

[0151] The first modality may be data collected by the first edge device (100-1), and the second modality may be data collected by the second edge device (100-2). Each of the first and second modalities may be image data, sensing data, sound data, or text data.

[0152] The processor (260) of the home server (200-2) can obtain the output value of the multimodal model using the encrypted first modality and the encrypted second modality (S705).

[0153] The multimodal model may be stored in the memory (230) of the home server (200-2).

[0154] The processor (260) of the home server (200-2) can extract a first set of feature vectors from an encrypted first modality and extract a second set of feature vectors from an encrypted second modality.

[0155] Each of the first feature vector set and the second feature vector set may contain 512 vectors.

[0156] The processor (260) of the home server (200-2) can input the first feature vector set and the second feature vector set into a multimodal model learned through contrast learning to obtain the output value of the multimodal model.

[0157] The output value of a multimodal model can be represented by a set of 512 output vectors.

[0158] The processor (260) of the home server (200-2) can obtain a situation inference result that matches the output value of the acquired multimodal model (S707).

[0159] The output value of a multimodal model may be matched to a specific situation.

[0160] Each of the multiple output values ​​may be matched with each of the multiple situation inference results. The situation inference result may be any one of the user's emotional state, occupancy status, intrusion status by an outsider into the home, or emergency status in a vehicle.

[0161] The processor (260) of the home server (200-2) can determine whether a response to the acquired situation inference result is possible (S709).

[0162] The processor (260) of the home server (200-2) can determine that a response is possible if action is possible regarding the result of situation inference, and can determine that a response is impossible if action is impossible regarding the result of situation inference.

[0163] For example, if the result of the situation inference is a situation where there is no presence in the home, control may be required to minimize the power consumption of the home appliances installed in the home. The processor (260) of the home server (200-2) may determine that a response to the result of the situation inference is impossible if the home server (200-2) does not have authority to control the home appliances, and may determine that a response is possible if the home server (200-2) has authority to control the home appliances.

[0164] If the processor (260) of the home server (200-2) determines that a response to the acquired situation inference result is possible, it can transmit the response result to the situation inference result to the first edge device (100-1) or the second edge device (100-2) through the communication circuit (210) (S711, 713).

[0165] If the result of the situation inference is that the user is absent from home, the response result may be a control command to minimize the power consumption of home appliances. The control command may be a command to turn off the power of home appliances or to control them to a standby state.

[0166] The processor (260) of the home server (200-2) may also transmit the response result to the situation inference result to a specific home appliance other than the first and second edge devices (100-1, 100-2).

[0167] If the processor (260) of the home server (200-2) determines that it is impossible to respond to the acquired situation inference result, it can transmit the situation inference result to the cloud server (200-1) through the communication circuit (210) (S713).

[0168] The processor (260) of the home server (200-2) can transmit the situation inference result and the output value of the multimodal model to the cloud server (200-1) through the communication circuit (210).

[0169] The processor (260) of the home server (200-2) may also transmit the situation inference result and the encrypted first and second modalities to the cloud server (200-1) through the communication circuit (210).

[0170] The processor (260) of the cloud server (200-1) can transmit the response result to the situation inference result to the first edge device (100-1) or the second edge device (100-2) through the communication circuit (210) (S715, 717).

[0171] The processor (260) of the cloud server (200-1) can obtain a response result for the situation inference result.

[0172] The processor (260) of the cloud server (200-1) may also transmit the acquired response result to the home server (200-2). In this case, the home server (200-2) may transmit the response result received from the cloud server (200-1) to the first edge device (100-1), the second edge device (100-2), or a specific home appliance.

[0173] Meanwhile, according to another embodiment of the present disclosure, the first situation inference result of the multimodal model of the home server (200-2) and the second situation inference result of the multimodal model of the cloud server (200-1) may be different from each other.

[0174] The cloud server (200-1) can receive the encrypted first and second modalities from the home server (200-2) and the first situation inference result obtained by the home server (200-2).

[0175] The cloud server (200-1) can obtain a second situation inference result from the encrypted first and second modalities using the multimodal model it stores.

[0176] If the cloud server (200-1) is different from the first situation inference result and the second situation inference result, it can send a notification indicating that the inference result is different to the home server (200-2) or the first and second edge devices (100-1, 100-2).

[0177] For example, the home server (200-2) may perceive the current situation as a normal situation, and the cloud server (200-2) may perceive the current situation as an emergency or critical situation. The cloud server (200-2) may send a notification indicating that it is an emergency or critical situation to the home server (200-2) or the first and second edge devices (100-1, 100-2). Accordingly, even if the situation determined through the multimodal model equipped in the home server (200-2) is incorrect, notification regarding the emergency / critical situation can be effectively provided.

[0178] FIGS. 8a to 8c are drawings illustrating a process of inferring a situation from a plurality of modalities using a multimodal model learned according to embodiments of the present disclosure.

[0179] In FIGS. 8a to 8c, the multimodal model (800) may be a global model stored in a cloud server (200-1) or a personalized model stored in a home server (200-2).

[0180] First, Figure 8a will be explained.

[0181] The first, second, and third edge devices (100-1, 100-2, 100-3) may be provided in a single device. For example, a home robot may include the first, second, and third edge devices (100-1, 100-2, 100-3).

[0182] The multimodal model (800) can receive encrypted image data from the first edge device (100-1), encrypted sound data from the second edge device (100-2), and text data from the third edge device (100-3).

[0183] The first edge device (100-1) can acquire image data including a user's facial expression and encrypt the image data through an image encoder.

[0184] The second edge device (100-2) can acquire sound data representing the voice spoken by the user and encrypt the sound data through a sound encoder.

[0185] The third edge device (100-3) can acquire text data representing the content of a conversation spoken by the user. The text data is data that does not require privacy, and the third edge device (100-3) may not be equipped with an encoder for encrypting the text data.

[0186] Text data can be converted into a vector set through a text encoder provided on a cloud server (200-1) or a home server (200-2).

[0187] The multimodal model (800) can output a situation inference result from a first vector set of encrypted image data, a second vector set of encrypted voice data, and a third vector set corresponding to text data.

[0188] For example, the result of situational inference can represent the user's emotional state. The emotional state may include anger and happiness.

[0189] The multimodal model (800) can place a first vector set of encrypted image data, a second vector set of encrypted voice data, and a third vector set in an embedding space containing multiple data points.

[0190] Each of the multiple data points can correspond to a specific emotional state.

[0191] The multimodal model (800) can obtain an emotional state corresponding to the nearest data point to the first, second, and third vector sets as a result of situation inference.

[0192] The multimodal model (800) can obtain an angry state as a result of situation inference, as shown in FIG. 8a.

[0193] The home server (200-2) or the cloud server (200-1) can send a control command to the home robot requesting a conversation to soothe the user or an empathetic conversation in response to the angry state. The home robot can output a voice to soothe the user or an empathetic voice according to the control command.

[0194] Next, Fig. 8b is explained.

[0195] The multimodal model (800) can receive encrypted sound data from the first edge device (100-1), encrypted sensing data from the second edge device (100-2), and log data from the third edge device (100-3).

[0196] The first edge device (100-1) can acquire sound data generated within the home and encrypt the sound data through a sound encoder.

[0197] The second edge device (100-2) can acquire sensing data representing a millimeter wave and encrypt the sensing data through a sensor encoder.

[0198] The third edge device (100-3) is a home appliance and can collect log data of the home appliance. The log data may include information regarding the time of operation, operating status, and power status of the home appliance. The log data may be data for which privacy is not required.

[0199] Log data can be converted into a vector set through an encoder provided on a cloud server (200-1) or a home server (200-2).

[0200] The multimodal model (800) can output a situation inference result from a first vector set of encrypted sound data, a second vector set of encrypted sensing data, and a third vector set corresponding to log data.

[0201] For example, the result of the situation inference may indicate the occupancy, absence, or intrusion of an outsider into the house.

[0202] The multimodal model (800) can place a first vector set of encrypted sound data, a second vector set of encrypted sensing data, and a third vector set in an embedding space containing multiple data points.

[0203] Each of the multiple data points can correspond to an occupancy status, an absence status, or an intrusion status.

[0204] The multimodal model (800) can obtain a state corresponding to the nearest data point to the first, second, and third vector sets as a result of situation inference.

[0205] The multimodal model (800) can obtain the absence state as a result of situation inference, as shown in FIG. 8b.

[0206] The home server (200-2) or cloud server (200-1) can send control commands to the home appliances to minimize the power consumption of the home appliances in response to the absence of the home server.

[0207] If the multimodal model (800) obtains an external intrusion status as a result of situation inference, the home server (200-2) or cloud server (200-1) can send a security notification to the user's terminal.

[0208] Next, Fig. 8c is explained.

[0209] The first, second, and third edge devices (100-1, 100-2, 100-3) may be provided in a single device. For example, a vehicle may include the first, second, and third edge devices (100-1, 100-2, 100-3).

[0210] The multimodal model (800) can receive encrypted image data from the first edge device (100-1), encrypted voice data from the second edge device (100-2), and log data from the third edge device (100-3).

[0211] The first edge device (100-1) can acquire image data captured inside the vehicle and encrypt the image data through an image encoder.

[0212] The second edge device (100-2) can acquire sound data within the vehicle and encrypt the sound data through a sound encoder.

[0213] The third edge device (100-3) can collect log data from the vehicle's processor. The log data may include the vehicle's door opening / closing status, vehicle speed, whether the vehicle is stopped, and the vehicle's operating status. The log data may be data for which privacy is not required.

[0214] Log data can be converted into a vector set through an encoder provided on a cloud server (200-1) or a home server (200-2).

[0215] The multimodal model (800) can output a situation inference result from a first vector set of encrypted image data, a second vector set of encrypted sound data, and a third vector set corresponding to log data.

[0216] For example, the result of situational inference may indicate an emergency situation or a normal situation inside a locked vehicle.

[0217] The multimodal model (800) can place a first vector set of encrypted image data, a second vector set of encrypted sound data, and a third vector set in an embedding space containing multiple data points.

[0218] Each of the multiple data points can represent an emergency state or a normal state inside a locked vehicle.

[0219] The multimodal model (800) can obtain a state corresponding to the nearest data point to the first, second, and third vector sets as a result of situation inference.

[0220] The multimodal model (800) can obtain an emergency situation inside a locked vehicle as a result of situation inference, as shown in FIG. 8c.

[0221] The home server (200-2) or cloud server (200-1) can respond to an emergency situation inside a locked vehicle by sending an emergency notification to the vehicle owner's terminal or unlocking the vehicle.

[0222] In this way, according to an embodiment of the present disclosure, an accurate situation can be inferred using various types of data, and measures suitable for the inferred situation can be taken.

[0223] FIG. 9 is a drawing illustrating an example in which each of a plurality of home servers has a personalized multimodal model according to one embodiment of the present disclosure.

[0224] Referring to FIG. 9, the cloud server (200-1) can store a global multimodal model (900).

[0225] The global multimodal model (900) can be transmitted to multiple home servers (200-2A, 200-2B).

[0226] The first home server (200-2A) can perform fine-tuning on some layers of the global multimodal model (900) and can have a fine-tuned first personalized model (900-1). This process may correspond to step S429 of FIG. 4.

[0227] The second home server (200-2B) can perform fine-tuning on some layers of the global multimodal model (900) and can have a fine-tuned second personalized model (900-2). This process may correspond to step S429 of FIG. 4.

[0228] That is, each home server can perform fine-tuning on a global multi-model (900) using multiple modalities received from edge devices, and can obtain a personalized multi-modal model based on the fine-tuning.

[0229] Based on a personalized multimodal model, user-customized situational reasoning and measures for situational reasoning can be effectively performed.

[0230] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. Computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0231] Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

Claim 1 An artificial intelligence server comprising: a memory for storing a multimodal model that infers a situation using multiple modalities having different types; a communication circuit that communicates with multiple edge devices or a cloud server; and a processor that receives parameter information for a fine-tuned layer of the multimodal model from the cloud server, receives multiple encrypted modalities from the multiple edge devices, and updates the parameter information of the fine-tuned layer of the multimodal model using the received multiple encrypted modalities. Claim 2 In claim 1, the processor further receives a plurality of encrypted modalities and obtains a first situation inference result from the further received plurality of encrypted modalities using the updated multimodal model, an artificial intelligence server. Claim 3 In paragraph 2, the processor is an artificial intelligence server that transmits a plurality of encrypted modalities and the first situation inference result to the cloud server through the communication circuit. Claim 4 In paragraph 3, the processor is an artificial intelligence server that receives a notification from the cloud server when the second situation inference result obtained by the cloud server and the first situation inference result are different. Claim 5 In paragraph 2, the processor determines whether a response to the first situation inference result is possible, transmits the response result to the plurality of edge devices through the communication circuit based on the determination that a response is possible, and transmits the situation inference result to the cloud server based on the determination that a response is impossible. Claim 6 In paragraph 5, the processor is an artificial intelligence server that receives a response result in response to transmitting the situation inference result from the cloud server to the cloud server. Claim 7 In claim 1, each of the plurality of modalities is an artificial intelligence server that is any one of image data, sound data, sensing data, text data, or log data. Claim 8 In claim 1, the multimodal model is an artificial intelligence server that is an artificial neural network-based model learned through contrastive learning. Claim 9 A method of operation of an artificial intelligence server comprising: receiving parameter information for a fine-tuned layer of a multimodal model that infers a situation using a plurality of modalities having different types from a cloud server; receiving a plurality of encrypted modalities from the plurality of edge devices; and updating the parameter information of the fine-tuned layer of the multimodal model using the received plurality of encrypted modalities. Claim 10 A method of operation of an artificial intelligence server according to claim 9, further comprising the step of additionally receiving a plurality of encrypted modalities and obtaining a first situation inference result from the additionally received plurality of encrypted modalities using the updated multimodal model. Claim 11 A method of operation of an artificial intelligence server according to claim 10, further comprising the step of transmitting a plurality of encrypted modalities and the first situation inference result to the cloud server through the communication circuit. Claim 12 A method of operation of an artificial intelligence server according to claim 11, further comprising the step of receiving a notification from the cloud server when the second situation inference result obtained by the cloud server and the first situation inference result are different. Claim 13 A method of operation of an artificial intelligence server according to claim 10, further comprising: a step of determining whether a response to the first situation inference result is possible; a step of transmitting the response result to the plurality of edge devices through the communication circuit based on the determination that a response is possible; and a step of transmitting the situation inference result to the cloud server based on the determination that a response is impossible. Claim 14 A method of operation of an artificial intelligence server according to claim 13, further comprising the step of receiving a response result in response to transmitting the situation inference result from the cloud server to the cloud server. Claim 15 A method of operation of an artificial intelligence server in which, in claim 9, each of the plurality of modalities is any one of image data, sound data, sensing data, text data, or log data.

Images