Electronic device for training integrated multimodal artificial intelligence model by using self-directed interactive learning, and operating method thereof

Abstract

According to the present disclosure, an electronic device and an operating method thereof are provided, the electronic device comprising: an encoder for encoding a behavior data set and outputting first and second encoded data; and a large-scale language model decoder which computes multi-head attention on the basis of an embedding vector generated on the basis of an instruction and the second encoded data, and of the first encoded data, and which, on the basis of attention scores of the multi-head attention, predicts, as an output, interpretation data representing text obtained by interpreting a companion animal.

Description

Electronic device for training an integrated multimodal artificial intelligence model using self-directed interactive learning and its operating method

[0001] Embodiments of the present disclosure relate to an electronic device and a method of operating the same, and more particularly, to an electronic device and a method of operating the same for training an integrated multimodal artificial intelligence model using self-directed interactive learning.

[0002] The number of people keeping pets has been growing exponentially due to recent factors such as rising income levels and the increasing number of single-person households resulting from an aging population. Companion animals are animals that people love, keep close, and cherish, and include animals such as dogs, cats, birds, and goldfish. In particular, some companion animals, such as dogs and cats, are expanding their role in our increasingly individualistic modern society, sharing life with their owners and sharing emotional bonds.

[0003] Pets can convey messages to their owners through specific behaviors, such as barking or moving. Because pets cannot speak human language, direct communication with humans can be challenging. Consequently, even when their pets are sick, it can be difficult to recognize their symptoms, and an increasing number of users are frustrated by the inability to gauge their pets' emotions or health.

[0004] Although various technologies have been proposed to understand the behavior of companion animals, the existing systems for communicating with companion animals have the problem that they only unilaterally provide the pet owner with information on the pet's physical condition, activity level, or food intake. In addition, unless the pet owner is constantly with the pet, it is difficult to easily understand the pet's appearance, possible future actions, or emotional state.

[0005] These conventional electronic devices and their operating methods have the problem of failing to provide adaptive AI models tailored to the temperament of companion animals. Embodiments of the present disclosure address these and other issues, and provide an electronic device and its operating method that provide an adaptive AI model tailored to the temperament of each companion animal for AI services. However, these tasks are exemplary and are not intended to limit the scope of the present disclosure.

[0006] According to one aspect of the present disclosure, an electronic device is provided, comprising: an encoder that receives a behavioral data set including first audio data representing audio of a companion animal, second audio data representing the companion animal's voice, video data representing a video of the companion animal, and IMU data representing an Inertial Measurement Unit (IMU) sensing result corresponding to a behavioral pattern of the companion animal, encodes the behavioral data set, and outputs first encoded data including features of each of the video data, the IMU data, and the first audio data, and second encoded data including features of each of the first audio data and the second audio data; and a large-scale language model decoder that receives an instruction, the first encoded data, and the second encoded data, calculates an embedding vector generated based on the instruction and the second encoded data, and multi-head attention based on the first encoded data, and predicts interpretation data representing a text interpreted about the companion animal based on attention scores of the multi-head attention as an output.

[0007] According to the present embodiment, the encoder may include a de-interlacer encoder that encodes video data, IMU data, and LFA (Low Frequency Audio) data of first audio data and provides the first encoded data to a large-scale language model decoder; and an audio encoder that encodes LFA data, second audio data, and HFA (High Frequency Audio) data of the first audio data and outputs second encoded data.

[0008] According to the present embodiment, the demagnetizer encoder may include: a video encoder that encodes video data to generate at least one video feature corresponding to a feature about the video; an IMU encoder that encodes IMU data to generate an IMU feature corresponding to a feature about the IMU; an LFA encoder that encodes LFA data to generate an LFA feature corresponding to a feature about the LFA; a synthesis unit that synthesizes IMU features and LFA features to generate synthesized IMU and LFA features; and a bind network that receives at least one video feature and the synthesized IMU and LFA features and provides a video feature closest to the synthesized IMU and LFA features to a large-scale language model decoder.

[0009] According to the present embodiment, the bind network may include a cache model that stores video features obtained during the learning process and, when synthesized IMU and LFA features are input, provides the video features closest to the synthesized IMU and LFA features among the stored video features to the large-scale language model decoder.

[0010] According to the present embodiment, the audio encoder may include a voice identifier that receives LFA data and second audio data and identifies whether a voice in the LFA data corresponds to a voice of a companion of the second audio data; a speech encoder that receives the LFA data and analyzes the voice in the LFA data by encoding the LFA data; and a sound encoder that receives the LFA data and HFA data, converts the HFA of the HFA data into the LFA, and analyzes environmental sounds and sounds of the companion animal from the LFA.

[0011] According to the present embodiment, the electronic device may further include a Q-Former that generates text tokens to be input to a large-scale language model decoder based on analysis results of a voice identifier, analysis results of a speech encoder, analysis results of a sound encoder, and learnable queries.

[0012] According to the present embodiment, the large-scale language model decoder can generate input torque based on the synthesis of second encoded data and instructions, perform embedding on the input torque to generate an embedding vector, perform a first RMS Norm (Root Mean Square Layer Normalization) on the embedding vector, perform RoPE (Rotary Positional Embeddings) on the vector on which the first RMS Norm was performed, compute masked self-attention on the embedding on which the RoPE was performed, and compute multi-head attention based on attention scores of the masked self-attention, the embedding vector, and a video feature closest to the IMU and LFA features synthesized from the first encoded data.

[0013] According to the present embodiment, a large-scale language model decoder can synthesize attention scores of masked self-attention, an embedding vector, and a video feature, perform a second RMS Norm on the synthesized result, compute multi-head attention on the embedding on which the second RMS Norm was performed and the key, query, and value of the embedding, and synthesize the attention scores of the multi-head attention and the embedding on which the second RMS Norm was performed.

[0014] According to the present embodiment, the large-scale language model decoder can perform a third RMS Norm on a result obtained by synthesizing attention scores of multi-head attention and an embedding on which a second RMS Norm is performed, perform a feedforward operation on the embedding on which the third RMS Norm is performed, synthesize the result of the feedforward operation and the embedding on which the third RMS Norm is performed, perform a fourth RMS Norm on a result obtained by synthesizing the result of the feedforward operation and the embedding on which the third RMS Norm is performed, linearize the embedding on which the fourth RMS Norm is performed, and output interpretation data using the linearized result and softmax.

[0015] According to one aspect of the present disclosure, there is provided an operating method of an electronic device, comprising: encoding a behavioral data set including first audio data representing audio of a companion animal, second audio data representing the companion animal's voice, video data representing a video in which the companion animal was filmed, and IMU data representing an Inertial Measurement Unit (IMU) sensing result corresponding to a behavioral pattern of the companion animal; providing first encoded data including features of each of the video data, the IMU data, and the first audio data, and second encoded data including features of each of the first audio data and the second audio data to a large-scale language model decoder; calculating multi-head attention based on instructions, an embedding vector generated based on the second encoded data, and the first encoded data; and predicting interpretation data representing a text interpreted about the companion animal based on attention scores of the multi-head attention.

[0016] Other aspects, features and advantages other than those described above will become apparent from the following detailed description, claims and drawings for carrying out the invention.

[0017] Additionally, these general and specific aspects may be implemented using any system, method, computer program, or combination of any system, method, or computer program.

[0018] According to an exemplary embodiment of the present disclosure, an electronic device and its operating method can be implemented that can provide an adaptive artificial intelligence model tailored to the temperament of each companion animal for AI services, and a multi-modal language model (MLLM) capable of predicting outputs as accurately as possible based on given inputs. Of course, the scope of the present disclosure is not limited by these effects.

[0019] FIG. 1 is a block diagram schematically illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0020] FIG. 2 is a diagram illustrating a behavioral data set according to an exemplary embodiment of the present disclosure.

[0021] FIG. 3 is a block diagram schematically illustrating the architecture of an artificial intelligence model according to an exemplary embodiment of the present disclosure.

[0022] FIG. 4 is a block diagram schematically illustrating a demagnetizer encoder according to an exemplary embodiment of the present disclosure.

[0023] FIG. 5 is a block diagram schematically illustrating an audio encoder according to an exemplary embodiment of the present disclosure.

[0024] Figures 6 to 9 are block diagrams for explaining transformers according to comparative examples.

[0025] FIG. 10 is a block diagram schematically illustrating a large-scale language model decoder according to an exemplary embodiment of the present disclosure.

[0026] FIG. 11 is a block diagram schematically illustrating one embodiment of the large-scale language model decoder of FIG. 10.

[0027] FIG. 12 is a diagram illustrating a neural network according to an exemplary embodiment of the present disclosure.

[0028] FIG. 13 is a diagram schematically illustrating an input feature map, a weight kernel, and an output feature map according to an exemplary embodiment of the present disclosure.

[0029] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to an exemplary embodiment of the present disclosure.

[0030] FIG. 15 and FIG. 16 are block diagrams showing the structure of an encoder according to an exemplary embodiment of the present disclosure.

[0031] The present disclosure is capable of various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present disclosure, as well as methods for achieving them, will become clearer with reference to the embodiments described in detail below, along with the drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various forms.

[0032] In the examples below, the terms first, second, etc. are not used in a limiting sense, but are used for the purpose of distinguishing one component from another.

[0033] In the examples below, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0034] In the following examples, terms such as “include” or “have” mean that a feature or component described in the specification is present, and do not preclude the possibility that one or more other features or components may be added.

[0035] In the following examples, when a part such as a layer, region, component, etc. is said to be on or above another part, it includes not only the case where it is directly above the other part, but also the case where another region, component, etc. is interposed in between.

[0036] For convenience of explanation, the sizes of components in the drawings may be exaggerated or reduced. For example, the sizes and thicknesses of each component shown in the drawings are arbitrarily indicated for convenience of explanation, and thus the present disclosure is not necessarily limited to the figures shown.

[0037] In some embodiments, where implementations are otherwise feasible, specific sequences of operations may be performed in a different order than described. For example, two steps described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0038] In this specification, “A and / or B” refers to the case where it is A, or B, or both A and B. And, “at least one of A and B refers to the case where it is A, or B, or both A and B.

[0039] In the following examples, when it is said that layers, regions, components, etc. are connected, it includes cases where the layers, regions, components, etc. are directly connected, and / or cases where other layers, regions, components, etc. are interposed between the layers, regions, components, etc. and are indirectly connected. For example, when it is said in this specification that layers, regions, components, etc. are electrically connected, it refers to cases where the layers, regions, components, etc. are directly electrically connected, and / or cases where other layers, regions, components, etc. are interposed between them and are indirectly electrically connected.

[0040] The x-axis, y-axis, and z-axis are not limited to the three axes in the Cartesian coordinate system, but can be interpreted in a broader sense that includes them. For example, the x-axis, y-axis, and z-axis may be orthogonal to each other, but they can also refer to different directions that are not orthogonal to each other.

[0041] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure is complete and to fully inform those skilled in the art of the scope of the present disclosure, and the present disclosure is defined solely by the scope of the claims.

[0042] The terminology used in this disclosure is for the purpose of describing embodiments only and is not intended to limit the present disclosure. In this disclosure, the singular may also include the plural unless specifically stated otherwise. The terms "comprises" and / or "comprising" as used herein do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the disclosure, and "and / or" may include each and any combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present disclosure.

[0043] The word "exemplary" is used herein to mean "serving as an example or illustration." Any embodiment described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments.

[0044] Embodiments of the present disclosure may be described in terms of a function or a block that performs a function. A block, which may be referred to as a "unit" or a "module" in the present disclosure, may be physically implemented by analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memories, passive electronic components, active electronic components, optical components, hardwired circuits, etc., and may optionally be driven by firmware and software. Furthermore, the term "unit" as used in the disclosure refers to software, hardware elements such as FPGAs or ASICs, and the "unit" may perform certain roles. However, the "unit" is not limited to software or hardware. The "unit" may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, a "part" may include elements such as software elements, object-oriented software elements, class elements, and task elements, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided within the elements and "parts" may be combined into a smaller number of elements and "parts" or further separated into additional elements and "parts."

[0045] Embodiments of the present disclosure can be implemented using at least one software program running on at least one hardware device and capable of performing network management functions to control elements.

[0046] Spatially relative terms such as "below," "beneath," "lower," "above," and "upper" may be used to readily describe the relationship between one component and other components as depicted in the drawings. Spatially relative terms may be understood to encompass different orientations of components during use or operation in addition to the orientations depicted in the drawings. For example, if a component depicted in the drawings were flipped over, a component described as "below" or "beneath" another component may end up "above" the other component. Thus, the exemplary term "below" may encompass both the above and below orientations. Components may also be oriented in other directions, and thus spatially relative terms may be interpreted accordingly.

[0047] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure may be used with the meaning commonly understood by those skilled in the art to which this disclosure pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0048] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals and redundant descriptions thereof will be omitted.

[0049] FIG. 1 is a block diagram schematically illustrating an electronic device (100) according to an exemplary embodiment of the present disclosure.

[0050] Referring to FIG. 1, an external device (11) can be worn on a companion animal such as a dog or puppy to sense various physical variables of the companion animal and output sensing data. The external device (11) can be worn, coupled, or attached to a companion animal (e.g., a puppy) and can be configured with a collar and an application. In an exemplary embodiment, the external device (11) can include an IMU unit (11_1), an audio unit (11_2), a video unit (11_3), a GPS unit (11_4), an all-factory unit (11_5), and a biometric information unit (11_6). The IMU unit (11_1) can output IMU data, the audio unit (11_2) can output audio data, the video unit (11_3) can generate video data, the GPS unit (11_4) can generate GPS data, the all-factory unit (11_5) can generate all-factory data, and the biometric information unit (11_6) can output biometric data. A set of data output from an external device (11) can be referred to as a behavioral data set.

[0051] In an exemplary embodiment, the biometric data may be data representing the biometric information of a companion animal. The biometric information unit (11_6) may include sensors that sense an electrocardiogram (ECG) (or EKG, an abbreviation for the German word "Elektrokardiogramm"), an Electro Encephalo Graphy (EEG), and a Photoplethysmogram (PPG). An ECG interprets the electrical activity of the heart at a given time. An electrocardiogram is recorded by electrodes attached to the skin and equipment outside the body. An EEG is an electrical signal that can indirectly measure the electrical activity of nerve cells that make up the brain through electrodes on the scalp. In other words, brain waves indirectly capture electrical activity information occurring inside the brain through an electric field. PPG is also referred to as photoplethysmography, photoplethysmography, or photoplethysmography. PPG is identified through the minute amount of blood flow that changes according to the pulse wave. When a PPG sensor shines light onto the skin, the amount of light absorbed varies depending on blood flow. By measuring how much light is absorbed, changes in blood flow can be identified. The instruction (12) may be information entered through a prompt.

[0052] The electronic device (100) may include an encoder (110) and a transformer model (130). The encoder (110) may receive various data provided from an external device (11), encode the data to extract characteristics of the data, and provide the encoded data to the transformer model (130). In an exemplary embodiment, the encoder (110) may include a demeanor encoder (111) and a behavioral encoder (112). The demeanor encoder (111) may encode IMU data, audio data, and video data, and provide the encoded data to the transformer model (130). In an exemplary embodiment, a demagnetizer encoder (111) may encode IMU data, LFA data, and video data, align video features, IMU features, and LFA features in the encoded data, and image-bind each feature to provide the bound data to a transformer model (130). A behavioral encoder (112) may encode audio data, video data, all-factory data, etc., and provide the encoded data to a transformer model (130). In an exemplary embodiment, the behavioral encoder (112) may encode first audio data and second audio data as an audio encoder, and output text tokens for the encoded data based on a Q-Former. Text tokens and instructions (12) may be synthesized (120), and the synthesized data may be provided to a transformer model (130). The transformer model (130) may be the aforementioned artificial intelligence model. In an exemplary embodiment, the Transformer model (130) may be a decoder-based artificial intelligence language model. The Transformer model (130) can take synthetic data and encoded data as input and predict interpretation data representing text interpreted by a companion animal as output. A detailed description of the Transformer model (130) is provided below with reference to FIGS. 3, 8, and 13.The transformer model (130) according to an exemplary embodiment of the present disclosure may be a multi-modal language model as an artificial intelligence model and may be referred to as PetAI. In the exemplary embodiment, when a user inputs text corresponding to a behavioral data set and an instruction (12) through a prompt, the behavioral data set is encoded, and the encoded data and the instruction (12) are converted into a plurality of tokens, which may then be input to the transformer model (130). The transformer model (130) may predict and output interpretation data.

[0053] In an exemplary embodiment, the electronic device (140) may be implemented as a device including a memory and a processor. Functions related to artificial intelligence according to the present disclosure are operated through the processor and memory.

[0054] The memory can store data for an algorithm for controlling the operation of components within the device or a program that reproduces the algorithm, and can be implemented with at least one processor that performs the aforementioned operation using the data stored in the memory. Here, the memory and the processor can be implemented as separate chips. Alternatively, the memory and the processor can be implemented as a single chip. The memory can store data supporting various functions of the device, a program for the operation of the processor, and can store input / output data. It can store a plurality of application programs (or applications) running on the device, data for the operation of the device, commands, and one or more instructions. At least some of these application programs can be downloaded from an external server via wireless communication.

[0055] A processor can execute one or more instructions stored in memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an AI-only processor such as an NPU. One or more processors control the processing of input data according to predefined operating rules or AI models stored in memory. Alternatively, if one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0056] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0057] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0058] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that imitates human neurons (biological neurons) to enable machines to learn. Artificial intelligence methodologies can be categorized into supervised learning, in which input data and output data are provided together as training data depending on the learning method, so that the solution (output data) to the problem (input data) is determined; unsupervised learning, in which only input data is provided without output data, so that the solution (output data) to the problem (input data) is not determined; and reinforcement learning, in which a reward (Reward) is provided from an external environment whenever an action (Action) is taken in the current state (State), and learning is performed in a direction to maximize this reward. In addition, artificial intelligence methodologies can be categorized according to the architecture of the learning model. The architectures of widely used deep learning technologies can be categorized into convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and generative adversarial networks (GANs).

[0059] The present device and system may include an artificial intelligence model. The artificial intelligence model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model in general that has problem-solving capabilities by changing the binding strength of synapses through learning, formed by artificial neurons (nodes) that form a network by combining synapses. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a desired result (output) from an arbitrary input (input) by changing the weights of neurons through learning.

[0060] The processor can create a neural network, train (or learn) a neural network, perform a calculation based on received input data, generate an information signal based on the calculation result, or retrain the neural network. The models of the neural network can include various types of models such as CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto. The processor can include one or more processors for performing calculations according to the models of the neural network. For example, the neural network can be a deep neural network. It may include a deep neural network.

[0061] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It will be understood by those skilled in the art that any neural network may be included, including but not limited to ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network).

[0062] According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a process for generating a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restrcted Boltzman Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Various artificial intelligence structures and algorithms can be used, including but not limited to Optimization, Recommendation, and Data Creation.

[0063] Although not shown, the electronic device (100) may further include a communication unit. The communication unit may perform communication with at least one user terminal, etc. In this case, in addition to a Wi-Fi module, the communication unit may include a wireless communication module that supports various wireless communication methods such as LTE (Long Term Evolution), 4G, 5G, and 6G.

[0064] According to the above-described embodiments, there is an effect of being able to provide an adaptive artificial intelligence model tailored to the temperament of each companion animal with respect to the AI ​​service.

[0065] Additionally, according to the embodiments described above, there is an effect that multiple adaptive artificial intelligence models can be trained and distributed using the system of the present disclosure.

[0066] In addition, according to the above-described embodiments, there is an effect of being able to provide a multi-modal language model (MLLM) that can predict output as accurately as possible based on a given input.

[0067] In addition, according to the above-described embodiments, there is an effect of providing a PetAI model with the desired performance in the present disclosure by matching predictable problems and given inputs as accurately as possible, and by providing answers that reflect the probability distribution of actual data for stochastic problems.

[0068] FIG. 2 is a diagram illustrating a behavioral data set according to an exemplary embodiment of the present disclosure.

[0069] Referring to FIGS. 1 and 2, in an exemplary embodiment, a behavioral data set input to an electronic device (100) may include IMU data (210), first audio data (220), second audio data (230), all-factory data (240), first text data (250), second text data (260), third text data (270), and video data (280).

[0070] The IMU data (210) may be data representing the IMU sensing results corresponding to the behavioral pattern of a companion animal. The IMU data (210) may be six-axis (or six-dimensional) time series data consisting of acceleration and angular velocity along tree axes. The category of the IMU data (210) is IMU, and the type of the IMU data (210) is time series. The IMU may include an acceleration sensor and an angular velocity sensor. The equipment used to acquire the IMU data (210) in a laboratory and field may be an IMU.

[0071] The first audio data (220) may be data representing audio of a companion animal. The first audio data (220) may include HFA (High Frequency Audio) data (221) and LFA (Low Frequency Audio) data (222). Each of the HFA data (221) and the LFA data (222) is of the audio type. The HFA data (221) may be data having a frequency range other than the frequency range of a human voice. For example, the HFA data (221) may have an audible frequency range (e.g., about 20 to 40 [kHz]) of a companion animal (e.g., a puppy or a dog). The category of the HFA data (221) is HFA. The equipment used to obtain the HFA data (221) in the laboratory may be a controlled speaker and microphone (Controlled Speaker & MIC). The equipment used to acquire HFA data (221) in the field may be a wide-range MIC. The LFA data (222) may be data having a frequency range other than the frequency range of the human voice but lower than the frequency range of the HFA data (221), or the frequency range of the voice of a human or a pet. For example, the LFA data (222) may have a frequency range of a human audible frequency (e.g., below about 20 [kHz]) or the voice of a human or a canine. The category of the LFA data (222) is LFA. The equipment used to acquire LFA data (222) in the laboratory may be a controlled speaker and camcorder. The equipment used to acquire HFA data (221) in the field may be a microphone (MIC).

[0072] In the audio domain, for example, a plant may produce a specific pattern of sound when it is dehydrated, or a tree may produce a specific pattern of sound when it is overly wet. Specifically, a stressed plant may produce an ultrasonic sound that can be recorded. The form of the ultrasonic sound may vary depending on the type of plant, the type of stress (drying, cutting), and the intensity of the stress (the number of days of drying). The form of the sound may be captured in a graph that can be defined by frequency, duration, and power (dB / Hz). The average maximum sound intensity recorded for dry plants may be approximately 61.6±0.1 [dBSPL] and 65.6±0.4 [dBSPL], and the average maximum frequency at 10 [cm] for tomatoes and tobacco, respectively, may be 49.6±0.4 [kHz] and 54.8±1.1 [kHz]. The average peak intensity of the sounds emitted by cut plants can be 65.6±0.2 [dBSPL] and 63.3±0.2 [dBSPL], and the average peak frequency at 10.0 [cm] for tomatoes and tobacco, respectively, can be 57.3±0.7 [kHz] and 57.8±0.7 [kHz]. For example, when certain organisms, especially species specialized in the ultrasonic range, such as dogs, walk in a park or mountain with many plants, dogs can hear the sounds of the aforementioned plants, so when a human does not hear any sounds, the dog may suddenly look into the distance and prick up its ears. In general, sharp ultrasonic emitting machines can be used for anti-barking purposes, but the soft or other high-frequency sounds emitted by nature, such as tomatoes, can be felt as everyday sounds for dogs, or can even be a factor that stimulates the enjoyment and curiosity of a walk.Based on the dynamic relationship between natural high-frequency sounds and the sensory organs and / or reactions of dogs, the present disclosure uses an artificial intelligence model to predict the actions, emotions, and future actions of companion animals by inputting physical characteristics such as hearing and smelling of the companion animal. In HFA, a spectrogram for frequency can be visualized and used for image learning. For HFA, two microphones, an all-band MIC and a stereo LFA MIC, can be installed on the companion animal's collar, and operations such as data cross-validation, data correction, missing value supplementation, and data augmentation are possible based on the data acquired by the two microphones.

[0073] The second audio data (230) may be data representing the voice of the companion animal. The second audio data (230) may be data representing a corpus containing the voices of the companion animal's parents. The category of the second audio data (230) is PPV, and the type of the second audio data (230) is audio. The equipment used to obtain the second audio data (230) in the laboratory may be a controlled speaker and microphone (Controlled Speaker & MIC). The equipment used to obtain the second audio data (230) in the field may be a microphone (MIC).

[0074] The olfactory data (240) may be data representing odors (olfactory) occurring around a companion animal. The olfactory data (240) may be a reference database of 32 odors at 3 concentration levels to address the noise issue of field gas sensor graphs. The category of the olfactory data (240) is olfactory (smell & gas), which includes odors and gases, and the type of the olfactory data (240) is time series. The equipment used to acquire the olfactory data (240) in the laboratory may be a smell simulator. The equipment used to acquire the olfactory data (240) in the field may be a gas sensor.

[0075] In the Allfactory domain, for example, 32 odors (house odors, fabric softener, bread, food, diffuser, etc.) can be classified based on multiple concentration levels. For example, pets are sensitive to body odor, so their body odors can be classified based on five or more concentration levels. In the laboratory, the body odor of the pet's owner may be primarily used, while in the field, natural odors, textile odors, etc. may be applied. Classification criteria in the Allfactory domain may include, for example, familiar and unfamiliar odors, odors based on racial classification, odors of people you've met, odors of spaces you've been to, unique household odors or incense odors, odors of commonly eaten foods, or odors based on the gender, race, cleanliness, food intake, and health of the primary caregiver.

[0076] The text data may be data representing information about a companion animal in text form. In an exemplary embodiment, the text data may include first to third text data.

[0077] The first text data (250) may be data representing the surrounding environment of the companion animal in text form. The first text data (250) may be data representing various environmental information around the companion animal, such as temperature, humidity, and location, in text form. The category of the first text data (250) is environmental information (Env (Environmental Info.)), and the type of the first text data (250) is text. The equipment used to obtain the first text data (250) in the laboratory is a measuring equipment for direct measurement, and in the field, the first text data (250) may be stored in the device as metadata.

[0078] The second text data (260) may be data representing the profile of a companion animal in text form. The second text data (260) may be data representing detailed information about the companion animal, such as the breed, age, sex, and neutering of the companion animal, in text form, or may be data representing dog temperament types. The category of the second text data (260) is profile, and the type of the second text data (260) is text. The second text data (260) may be obtained through text labels submitted by owners of companion animals in laboratories and fields (Owner-Submitted Text Labels) and / or through personal information labels from our custom survey (Personality Labels from Our Custom Survey).

[0079] The third text data (270) may be data that represents experimental conditions of a companion animal in text form. The third text data (270) may be data that represents experimental conditions in text form, such as who said what and where the dog goes for a walk. The category of the third text data (270) is "Cond" and the type of the third text data (270) is "text." The third text data (270) may be acquired through metadata in a laboratory.

[0080] Video data (280) may be data representing images and / or videos captured by a third-person perspective camera that captures the appearance of a target companion animal (e.g., a target puppy). The video data (280) is categorized as video, and the type of the video data (280) is video (images). The video data (280) may be acquired in a laboratory using a camcorder.

[0081] In an exemplary embodiment, the behavioral data set may include lab data obtained in a laboratory and field data obtained in the field. Lab data may include basic collected data such as video, LFA, HFA, IMU, olfactory, pet parent's voice (PPV), dog's profile, experimental conditions, GPS, etc., and health data such as ECG, EEG, PPG, and IMUs mounted at different locations. Field data may include LFA, HFA, IMU, olfactory, GPS, etc. The collected data may be stored on a server directly or over a network. Since various modalities are collected simultaneously, the data is organized and stored in sync with respect to time.

[0082] The collected data must be processed to be used in AI model training. Voice data (e.g., first audio data (220)) must be divided into LFA and HFA, and sound annotation must be performed for the LFA. Video data (280) is used to annotate the dog's appearance. Since the AI ​​model of the present disclosure does not use video data (280) as input, the annotation data obtained in this manner is paired with the IMU and used as training data during the model training process. All-factory data requires labeling according to odor type and concentration. If one training data clip is 10 seconds long, a spectrogram of the voice data is created and stored in 10-second units. Analysis of the IMU data provides direct information about the dog's movement, so the results of feature extraction from the IMU data (210) are stored as training data. After processing each data, the processed data are grouped and stored in a database so that they can be used as training data.

[0083] In model training, data augmentation techniques can be used to dramatically increase the amount of training data. Specifically, slightly perturbing the MU or voice information can increase the diversity of inputs while maintaining a nearly identical output. The correct answer (e.g., ground truth) for the model's output can be derived from analysis of audio data, such as the dog's appearance, emotions, and behavior. Based on this, the model can be trained so that its output resembles the video.

[0084] To deploy the model, the trained model is uploaded to a server, and when new animal behavior data arrives, predicted results are obtained based on the model trained on the server. The model of this disclosure is a model trained by observing the predicted appearance based on a video as ground truth. However, there are parts of the dog's surrounding information or health analysis that are not revealed in the image that the model of this disclosure has difficulty learning on its own. In this case, there may be differences from the actual correct answer, and these differences can be used to periodically update the already trained model. This update method is called SDILS. SDILS refers to a system that periodically updates the already trained model to narrow the gap between the model's output and the actual expected result. The ground truth for the output of this disclosure uses the results extracted from the video data (280). However, even if the ground truth is obtained from the video using a high-performance Vision-Language LLM, there may be errors in the analysis of the dog's emotions or behavior when viewed by an animal behavior expert. And it is even more difficult to trust the inference of a high-performance LLM as the correct answer when it comes to information related to the surrounding environment that is not revealed in the video, or areas that require veterinary knowledge such as ECG, EEG, PPG, or gait / respiratory analysis. Therefore, by analyzing the true correct answer for such data, the model of this disclosure can enable analysis that requires more advanced knowledge beyond the level of simple visual analysis. The results analyzed by SDILS are implemented through an additional RoLA fine-tuning technique. The reason for using the RoLA technique is that it learns new parameters without touching the parameters of the previously learned model, so the general inference ability is maintained while further improving the overall performance. In addition, since RoLA merges the parameters into the model parameters during inference, it can have the advantage of maintaining the size of the inference model and the inference time.

[0085] FIG. 3 is a block diagram schematically illustrating the architecture of an artificial intelligence model according to an exemplary embodiment of the present disclosure.

[0086] Referring to FIG. 3, condition data (30) may correspond to the third text data (270) of FIG. 2 and may be used in ground truth (350). In ground truth (350), condition data (30) may be a reference for calculating a mismatch for a predicted result of an appearance of a companion animal. Video data (31) may be provided to ground truth (350) and a deminer encoder (311) of the encoder (310). In ground truth (350), video data (31) may be a reference for calculating a mismatch for a predicted result of a situation of a companion animal. IMU data (32) may be provided to the deminer encoder (311) of the encoder (310). LFA data (33) may be provided to the encoder (310). PPV data (34) and HFA data (35) may be provided to an audio encoder (312) of an encoder (310). The PPV data (34) may correspond to the second audio data (230) of FIG. 2. The audio encoder (312) may be an embodiment of the behavioral encoder (112) of FIG. 1.

[0087] The demultiplexer encoder (311) can encode video data (31), IMU data (32), and LFA data (33) and provide the first encoded data to a large-scale language model decoder (330).

[0088] The audio encoder (312) can encode LFA data (33), PPV data (34), and HFA data (35) to output second encoded data. The second encoded data, environmental data (36), profile data (37), and instructions (38) can be input to a large-scale language model decoder (330) via a prompt (320).

[0089] A large-scale language model decoder (330) can predict interpretation data as output (340) based on the prompt (320) and the first encoded data.

[0090] In an exemplary embodiment, the interpretation data as output (340) may include text describing the appearance (or appearance) of the companion animal, including the companion animal's posture, behavior, and expression. The desired output (340) may include text that infers the companion animal's emotions and behavior (e.g., a dog) or predicts its next behavior based on the same. The correct answer to the output (340), i.e., the ground truth (350), may be obtained through analysis of the video data (31). In addition, the interpretation data as output (340) may further include text describing an expected circumstance for the companion animal. For example, if a situation occurs in which the companion animal currently smells a certain scent, it may be difficult to obtain the information that the companion animal smelled the scent through the video data (31). The ground truth (350), which is the correct answer to the above-described information, can be obtained through condition data (30) representing experimental conditions.

[0091] In an exemplary embodiment, the interpretation data as output (340) may further include text describing the pet's current emotions, in addition to text describing the pet's appearance and situation. Furthermore, the interpretation data as output (340) may further include text describing the pet's next action.

[0092] In an exemplary embodiment, the text of the interpretation data as output (340) may be output in a descriptive format that describes the texts interpreting the companion animal in one or more complete sentence formats, or in a question-answer format that matches the texts interpreting the companion animal as answers to the question text, depending on the contents of the extraction (38).

[0093] Based on a comparison between the interpretation data and the ground truth (350) as outputs (340), an evaluation of the large-scale language model decoder (330) can be performed. Evaluations can be performed for each adaptive PetAI model. Metrics related to sentence similarity or sentence consistency, commonly used in language models, can be used as descriptive outputs. Metrics related to multiple-choice outputs can be determined by the expected value of the scores obtained by evaluating each option differently by assigning weights based on the importance of each item. The final evaluation can represent the average of the evaluation results for each adapter.

[0094] FIG. 4 is a block diagram schematically illustrating a demagnetizer encoder according to an exemplary embodiment of the present disclosure.

[0095] Referring to Fig. 4, the demagnetizer encoder (400) can be used to interpret IMU data (42) in a high-dimensional manner. In order to use the IMU as an input to the decoder of the transformer, text information that appropriately describes the IMU data (42) is required. Although learning is not impossible, in the case of images or sounds, the content can be relatively intuitively described with text, but in the case of IMU data (42), since it is sensor data in the form of a time series, if the IMU data (42) is converted into text and described, only the data flow pattern can be described. However, in order to achieve the same effect as visually viewing a video from the IMU, the IMU data (42) is not provided as input as text, but features are extracted and fused to the decoder of the transformer. In addition, since the appearance of a pet (e.g., a dog) can be analyzed more accurately when sound information related to the pet is analyzed together with the IMU data (42) rather than analyzing the appearance of the pet with only the IMU data (42), the demagnetizer encoder (400) can receive IMU data (42) and audio data.

[0096] The deminer encoder (400) can extract features of the appearance of a companion animal from the IMU data (42) and the LFA data (43). In an exemplary embodiment, the deminer encoder (400) can use the video data (41), the IMU data (42), and the LFA data (43) through a binding method for ImageBind-LLM to find an image corresponding to the IMU data and transmit it to the large-scale language model decoder (430).

[0097] Certain sounds can elicit a strong response from a dog, and the deminer encoder (400) can estimate an image of a companion animal from LFA data (43) and / or IMU data (42) by applying a bind method for ImageBind-LLM. Since obtaining information about the appearance of a companion animal from the IMU data (42) and LFA data (43) corresponds to a high-dimensional interpretation, the deminer encoder (400) can directly provide encoded data including features of each of the IMU data (42) and LFA data (43) to a large-scale language model decoder (430). In the ImageBind-LLM method, ImageBind (image bind, or image binding) is a method that generates an encoder for each modality such as image, video, text, depth, thermal, IMU, and audio, and then trains a large-scale language model so that all features obtained from the encoder corresponding to each modality are all aligned with the image encoder (or so that the embedding space generated from the encoder corresponding to each modality is aligned with the embedding space generated by the encoder for the video), so that all modalities can be interpreted as if they were images. In ImageBind-LLM, the features generated by the encoder for the video can be aligned with the word features of the large-scale language model through the binding network. When inputs of other modalities are provided to the deminer encoder (400), the closest matching image features from the training data can be used. In an exemplary embodiment, the binding method for ImageBind-LLM may be such that the features obtained from the encoder for the IMU and LFA are all learned to align with the encoder for the video.

[0098] In an exemplary embodiment, the demultiplexer encoder (400) may include a video encoder (410), an IMU encoder (420), an LFA encoder (430), and a bind network (470).

[0099] A video encoder (410) can encode video data (41) to generate a video feature (440) (or video feature embedding) corresponding to a feature related to a video in the video data (41).

[0100] The IMU encoder (420) can encode IMU data (42) to generate an IMU feature (or IMU feature embedding) corresponding to a feature about the IMU in the IMU data (42).

[0101] The LFA encoder (430) can encode LFA data (43) to generate an LFA feature (or LFA feature embedding) corresponding to a feature related to LFA in the LFA data (43).

[0102] The synthesis unit (480) can synthesize IMU features and LFA features. The reason for synthesizing IMU features and LFA features is that it is difficult to infer video data into images with only IMU features or LFA features. The synthesized IMU and LFA features (450) can be trained to be aligned (460) with the video features (440) and can be provided to a large-scale language model decoder (430). The synthesized IMU and LFA features (450) and the video features (440) can be provided to a bind network (470) and can be aligned with the word features of the large-scale language model through the bind network (470). In an exemplary embodiment, the bind network (470) can include a cache model. The cache model is a model that collects and stores video features obtained during the training process, and when a new feature is input to the cache model, it uses the closest feature among the already stored features instead. In an exemplary embodiment, the cache model stores one or more video features (440), and when synthesized IMU and LFA features (450) are input to the cache model, the feature that is closest to the synthesized IMU and LFA features (450) among the stored video features can be provided to the large-scale language model decoder (430). That is, the feature that is closest to the synthesized IMU and LFA features (450) can be aligned so that the large-scale language model can understand it.

[0103] FIG. 5 is a block diagram schematically illustrating an audio encoder according to an exemplary embodiment of the present disclosure.

[0104] Referring to FIG. 5, an audio encoder (520) according to an exemplary embodiment can receive PPV data (54), LFA data (43), and HFA data (55) as input data.

[0105] The audio encoder (520) may include a voice identifier (521), a speech encoder (522), and a sound encoder (523).

[0106] The voice identifier (521) can identify whether the human voice is the voice of the companion animal (e.g., the owner) when there is human speech (speech). In an exemplary embodiment, the voice identifier (521) can receive PPV data (54) and LFA data (53), and identify whether the voice in the LFA data (53) corresponds to the voice of the companion animal in the PPV data (54).

[0107] The speech encoder (522) can analyze and interpret human speech. In an exemplary embodiment, the speech encoder (522) can receive LFA data (53) and analyze the speech of the LFA data (53) by encoding the LFA data (53). For example, the speech encoder (522) can analyze and interpret the words or voice commands that the owner says to the dog in the LFA data (53).

[0108] The sound encoder (523) can analyze environmental sounds, the voice of a pet (hereinafter, “pet voice”), sounds generated by the behavior of a pet (hereinafter, “pet behavior sounds”), etc. In an exemplary embodiment, the sound encoder (523) can receive LFA data (53) and analyze environmental sounds such as car sounds, dog voices, dog behavior sounds such as the sound of a dog eating, etc. from the sounds of the LFA data (53) by encoding the LFA data (53). In an exemplary embodiment, the sound encoder (523) can convert the HFA of the HFA data (55) into an LFA and then analyze the HFA as if it were an LFA. For example, the sound encoder (523) can convert the HFA of the HFA data (55) into an LFA through a pitch shift (510) for a low frequency and then analyze the HFA.

[0109] In an exemplary embodiment, the voice identifier (521) and speech encoder (522) may be used as pre-trained models. The sound encoder (523) may be a pre-trained model fine-tuned using pairs of collected LFA data (53) and annotations. In an exemplary embodiment, the sound encoder (523) may be a model fine-tuned based on LFA data (53) from the BEATs encoder of the SALMONN paper.

[0110] The results of the audio decoder (520) (e.g., the analysis results of the voice identifier (521), the analysis results of the speech encoder (522), and the analysis results of the sound encoder (523)) may be provided in a cross-attention manner in the middle of a large-scale language model decoder. Alternatively, the results of the audio decoder (520) may be converted into text tokens (540) that can be understood by a large-scale language model through a Q-FORMER (530), and the text tokens (540) may be input to the large-scale language model decoder. The Q-FORMER (530) may generate text tokens (540) based on runnable queries (550) and the results of the audio decoder (520). The Q-FORMER (530) may require training. A clear interpretation is more important than a high-level interpretation of information about the audio. Therefore, inputting text tokens (540) into a large-scale language model decoder effectively enables clear interpretation of audio information. Focusing on high-level interpretation can lead to unclear interpretation of simple parts, so using the results of a clear analysis can solve this problem.

[0111] Text tokens (540) may include whether the companion's voice is present (e.g., whether the owner's voice is present), speech content, environmental / dog sound analysis (analysis of environmental sounds, dog's voice, dog behavior sounds, etc.), and HFA analysis.

[0112] Figures 6 to 9 are block diagrams for explaining transformers according to comparative examples.

[0113] Referring to Figures 6 through 9, large language models (LLMs) can demonstrate great potential as AI assistants capable of performing complex reasoning tasks requiring specialized knowledge across a wide range of fields, including specialized areas such as programming and creative writing. LLMs enable human interaction through an intuitive chat interface, potentially leading to rapid and widespread adoption among the general public.

[0114] The training methodology for LLMs can be simple. An autoregressive transformer can be pre-trained on an extensive self-supervised data set and then tuned to human preferences using techniques such as Reinforced Learning with Human Feedback (RLHF). While the training method is simple, the high computational requirements have limited LLM development to a small number of players. Pre-trained LLMs that match closed-loop performance (e.g., BLOOM (Scao et al., 2022), LLaMa1 (Touvron et al., 2023), and Falcon (Penedo et al., 2023)) have been publicly released. While there are pre-trained competitors such as GPT3 (Brown et al., 2020) and Chinchilla (Hoffmann et al., 2022), none of these models are suitable as replacements for closed-loop product LLMs such as ChatGPT, BARD, and Claude. These closed-loop LLM products can be fine-tuned to human preferences, significantly improving their usability and security. This step can require significant computational and human annotation costs, and is often nontransparent or difficult to reproduce, limiting progress within the community to advance AI alignment research.

[0115] In this work, we present a petAI model with up to 70B parameters based on Llama 2 and Llama 2Chat, pre-trained and fine-tuned LLM families. In a series of usability and safety benchmarks tested in this disclosure, the Llama 2Chat-based petAI model generally outperformed existing open-source models. Furthermore, it appears to be comparable to some closed-source models, at least in human evaluations conducted by this disclosure. We take steps to enhance the safety of these models by using safety-related data annotations and tuning, red teaming, and iterative evaluations. Furthermore, the specification of this disclosure thoroughly describes the fine-tuning methodology and approach for improving LLM safety. Through this openness, we hope to enable the community to reproduce fine-tuned LLMs and continuously improve their safety, paving the way for more responsible LLM development.

[0116] As an example of an LLM, a transformer can be applied. The transformer can include an encoder and a decoder. The encoder of the transformer can be composed of six identical layers with two sublayers. The first sublayer of the encoder is a multi-head self-attention mechanism, and the second sublayer is a feed-forward network. Both sublayers have residual connections, followed by normalization. Rather than simply repeating operations through a specific layer, copied values ​​are fed directly, so the entire network receives both the original information and the residual portion. This reduces training difficulty, accelerates initial model convergence, and facilitates finding the global optimum. All sublayers set the output dimension to 512. The decoder of a transformer can be composed of six identical layers, each with three sublayers. Two of the sublayers are identical to those of the encoder, and one sublayer is different from those of the encoder, being a masked multi-head self-attention layer. The output of the last performed layer of the encoder is fed into the multi-head self-attention layer of the decoder. The first sublayer of the decoder performs the same task as the first sublayer of the encoder.

[0117] Figure 6 illustrates the architecture of the GPT, which consists of a decoder for a transformer, and the architecture of a petAI based on Llama 2. Referring to Figure 6, the architecture of the petAI based on Llama 2 is based on several key elements. Essentially, several improvements have been introduced to the architecture of the petAI based on Llama 2, based on the architecture of Llama 1.

[0118] Regarding the Transformer Architecture, Llama 2 uses the standard Transformer architecture proposed by Vaswani et al. (2017). This allows for efficient processing of various parts of the input sequence through self-attention and positional encoding.

[0119] Regarding pre-normalization and RMS norm (Root Mean Square Layer Normalization), Llama 2 performs pre-normalization using RMS norm. This stabilizes learning and promotes faster convergence.

[0120] Regarding the SwiGLU Activation Function, Llama 2-based petAI uses SwiGLU (Switched Gated Linear Unit) as the activation function. This further enhances nonlinearity and increases the expressive power of the model.

[0121] Regarding RoPE (Rotary Positional Embeddings), Llama 2-based petAI uses RoPE to process positional information, ensuring that each element of the input sequence can be correctly interpreted according to its position.

[0122] Regarding Grouped-Query Attention (GQA), it is an improvement over standard multi-head attention, and manages memory usage and computation time efficiently by grouping attention calculations.

[0123] These technical details are designed to enable Llama 2-based petAI to perform well on a variety of natural language processing tasks, and in particular, to efficiently handle the complex data and interactions required by large-scale language models.

[0124] Pre-normalization is the process of normalizing the inputs of each sublayer in a Transformer model. Transformer models typically adjust the mean and variance of input data and data generated within the layer to reduce numerical instability that can occur during the training process. Pre-normalization ensures that the inputs of each sublayer are normalized and processed before being passed through the layer. This approach is particularly helpful in mitigating vanishing or exploding gradients in deep networks.

[0125] RMS Norm is a variant of Layer Normalization that normalizes the inputs of each layer by calculating only the variance, rather than the mean. The core idea of ​​RMS Norm is to calculate the root mean square of each element in the input vector and use this to normalize. This method improves computational efficiency by only adjusting the scale rather than subtracting the overall mean, and more importantly, it stabilizes performance in deep networks.

[0126] The RMS norm works particularly well in transformer architectures, providing uniform scaling across layers and thus reducing numerical instability that can occur during training. This improves gradient flow in deep network structures, enabling the training of deeper models.

[0127] Pre-normalization and RMS norm both play a crucial role in improving the training speed and performance of deep learning models, especially transformer-based models. Pre-normalization normalizes inputs across the entire model, while RMS norm adjusts the scaling of each layer to ensure stable learning even in deeper networks.

[0128] SwiGLU (Switched Gated Linear Unit) is a nonlinear activation function used in neural network models. This activation function was specifically designed to improve the performance of large-scale language models, such as transformer models. SwiGLU is an improved version of the existing Gated Linear Unit (GLU), providing a structure that simultaneously improves computational efficiency and model performance.

[0129] The basic principle and operation of SwiGLU is as follows: The SwiGLU activation function consists of two parts: a linear transformation and a gate function. The basic formula is as shown in [Mathematical Formula 1] below.

[0130] [Mathematical Formula 1]

[0131]

[0132] Here, x represents the input vector, W and b represent the learnable weights and biases, and sigma represents the sigmoid activation function. The core of this function is to apply a linear transformation to the input, and then pass a portion of the input through the gate value obtained through the sigmoid function.

[0133] Regarding performance enhancement factors, there is computational efficiency. SwiGLU provides similar or better performance than GLU while reducing computational overhead. This is partly due to the use of fewer parameters. There is information flow optimization. The sigmoid gate selectively activates only important features of the input data, regulating the flow. This helps prevent the loss of unnecessary information and emphasizes important information. There is also increased model expressiveness. SwiGLU enhances the model's expressiveness by applying nonlinear gates after simple linear transformations. This allows the model to learn more complex patterns and data structures.

[0134] SwiGLU can be particularly useful in deep learning models related to natural language processing. This activation function helps models efficiently process important information and produce more accurate and detailed output. Therefore, it has the potential to be widely used not only in transformer architectures but also in various deep neural network models.

[0135] Figure 7 illustrates Absolute Position Embedding (APE) in the GPT architecture and Rotary Position Embedding (Rotary Position Embedding) in the Llama 2 architecture. Referring to Figure 7, RoPE is a novel method for encoding positional information of an input sequence, particularly in a transformer model. This technique rotates the embedding of each token according to its position, enabling the model to more clearly convey the relative positional relationship between tokens. RoPE is used to replace or complement existing positional encoding methods and is particularly useful for processing data with long sequence lengths.

[0136] In the basic principle of RoPE, RoPE encodes positional information by rotating the embedding of each token. This rotation is defined by position p and dimension d, and is applied to the embedding vector of each token as shown in [Mathematical Equation 2] below.

[0137] [Equation 2]

[0138]

[0139] Here, x is the embedding vector, and R(p) is the rotation matrix at position p. The rotation matrix R(p) is computed via the Fourier transform, providing a unique rotation for each token at each position.

[0140] RoPE's advantages include its ability to encode relative positions. RoPE effectively conveys the relative positional information of each token to the model. This helps to better understand the relationships between tokens and their context within the sequence. It also offers excellent scalability and compatibility. Because RoPE focuses on the relative rather than absolute positions of tokens, it maintains consistent performance across inputs of varying lengths. This is particularly important when the model processes long sequences. It is computationally efficient. The rotation-based approach is computationally efficient and can outperform traditional Fourier-based positional encoding. This is particularly advantageous for complex language modeling tasks.

[0141] In terms of application areas, RoPE is particularly useful in language models based on transformer architectures, sequence-to-sequence models, and deep learning models that process other types of sequential data. Its effectiveness has been proven in fields such as natural language processing, speech recognition, and machine translation.

[0142] RoPE is an important technique that improves the overall performance and interpretability of a model by helping the model more accurately understand the positional context of the input data.

[0143] GQA is a variant of the efficient attention mechanism used in Transformer architectures. GQA's mechanism aims to reduce computational costs and improve processing speed by improving upon existing multi-head attention used in large-scale language models and other deep neural networks. GQA is an attention mechanism that efficiently manages computational costs and memory usage while maintaining the robust performance of Transformer models.

[0144] The fundamental principle of GQA is that, instead of each head independently calculating attention for each input, as in traditional multi-head attention, GQA groups input sequences and calculates attention for each group. Each group contains only a portion of the sequence, and attention is calculated only within the group.

[0145] For the GQA operating mechanism, input segmentation is performed. The input sequence is divided into multiple groups. Each group can be formed based on either a continuous sequence fragment or a specific pattern. Next, attention is calculated for each group. Instead of traditional multi-head attention, attention is calculated only within each group. This significantly reduces the required computational effort and memory usage. Results are then synthesized. The attention results calculated in each group are combined to generate the final output. This process is similar to the method of merging the results of a typical multi-head attention.

[0146] GQA boasts excellent computational efficiency. By grouping inputs, the total computation required to calculate attention is reduced. This is particularly useful when processing long sequences, and enhances scalability for larger models or longer sequences. It is also memory-optimized. Because the attention map for the entire input is not computed at once, memory requirements are reduced, making large-scale model training easier. It is also easy to parallelize. Since each group can be computed independently, parallel processing improves performance.

[0147] In terms of application areas, GQA can be usefully applied to various sequence modeling tasks, particularly natural language processing. For example, it can contribute to improving processing speed and efficiency in areas such as large-scale document summarization, machine translation, and speech recognition.

[0148] In terms of architectural differences from Llama 1, the context window in Llama 2 is expanded from 2048 tokens to 4096 tokens, with respect to context length. A longer context window allows the model to process more information, which is particularly useful for supporting longer records in chat applications, performing various summarization tasks, and understanding longer documents. Comparing the performance of 2k and 4k context pretraining on long-context benchmarks, both models are trained on 150 billion tokens, maintaining the same architecture and hyperparameters and differing only in context length. This disclosure observes improvements in SCROLLS (Shaham et al., 2022), which has an average input length of 3.5k, and no performance degradation in SQUAD (Rajpurkar et al., 2018). This demonstrates that longer-context models maintain robust performance across a variety of general-purpose tasks. For Grouped Query Attention (GQA), a standard practice in autoregressive decoding is to cache key (K) and value (V) pairs for previous tokens in a sequence to speed up attention computation. However, increasing the context window or batch size significantly increases the memory cost associated with the KV cache size of the Multi-Head Attention (MHA) model. For larger models where the KV cache size becomes a bottleneck, key and value predictions can be shared across multiple heads without performance degradation (Chowdhery et al., 2022). Either the original multi-query format (MQA; Shazeer, 2019) using a single KV projection or a variant of Grouped Query Attention (GQA; Ainslie et al., 2023) using 8 KV projections can be used. Comparing the MQA and GQA variants with the MHA baseline, the present disclosure trains all models with 150B tokens while maintaining a fixed model size of 30B.To maintain a similar number of overall parameters across GQA and MQA, the dimensionality of the feedforward layer is increased to compensate for the reduction in the attention layer. For the MQA variant, the FFN dimension is increased by a factor of 1.33, while for the GQA variant, it is increased by a factor of 1.3. The results show that the GQA variant performs similarly to the MHA baseline in most evaluation tasks and, on average, outperforms the MQA variant.

[0149] To optimize latency, this disclosure hosts the largest model on a single node using eight A100s with tensor parallelism (Shoeybi et al., 2019). In this setup, sharding for MQA can no longer be performed across multiple heads, as the number of heads is smaller than the number of GPUs. An alternative is to replicate KV values ​​across all GPUs (making the KV cache size the same as GQA), or instead shard across the batch dimension (Pope et al., 2022). However, the latter option only works when the batch size is larger than the number of shards, and the additional communication cost may complicate the inference service, making it not worthwhile in all cases.

[0150] Figure 8 schematically illustrates the training method for the Llama 2Chat-based petAI. Referring to Figure 8, the process begins with pretraining Llama 2 using publicly available online resources. Supervised fine-tuning is then applied to create an initial version of Llama 2Chat. The model is then iteratively refined using the Reinforcement Learning with Human Feedback (RLHF) methodology, specifically rejection sampling and proximal policy optimization (PPO). Accumulating iterative reward modeling data in parallel with model refinement throughout the RLHF phase is crucial to ensuring that the reward model persists within the deployment.

[0151] For pre-training, we use a standard transformer architecture (Vaswani et al., 2017), apply pre-normalization using the RMS norm (Zhang and Sennrich, 2019), use the SwiGLU activation function, and pre-train using RoPE. Key architectural differences from Llama 1 include increased context length and GQA.

[0152] Regarding hyperparameters, the present disclosure was trained using the AdamW optimizer (Loshchilov and Hutter, 2017) with a1 = 0.9, a2 = 0.95, and eps = 10 5 . The present disclosure uses a cosine learning rate schedule with a 2000-step warmup, and the final learning rate can be reduced to 10% of the maximum learning rate. The present disclosure uses a weight decay of 0.1 and gradient clipping of 1.0.

[0153] Regarding the tokenizer, this disclosure uses the same tokenizer as Llama 1. The tokenizer utilizes the BPE (Byte Pair Encoding) algorithm and an implementation of SentencePiece. Like Llama 1, this disclosure breaks all numbers into individual digits and uses bytes to decompose unknown UTF8 characters. The total vocabulary size is 32,000 tokens. In the Llama 2 model family, the number of tokens refers only to pre-training data. All models use a global batch size of 4M tokens, with larger models (34B and 70B) using Grouped-Query Attention (GQA). This improves inference scalability.

[0154] In terms of fine-tuning, Reinforcement Learning with Human Feedback (RLHF) is a model training procedure applied to fine-tuned language models to better align model behavior with human preferences and instructions. This approach collects data representing empirically sampled human preferences, allowing human annotators to select their preferred output between two model outputs. This human feedback is then used to train a reward model that learns the human annotator's preference patterns and then automates preference decisions. For reward modeling, the reward model takes model responses and corresponding prompts (including the context of the previous turn) as input and outputs a scalar score representing the quality of the model generation (e.g., usefulness and safety). These response scores can be used as rewards to optimize Llama 2-Chat during RLHF to improve human preference tuning and enhance both usefulness and safety. Others have found that usefulness and safety sometimes conflict (Bai et al., 2022a). This can make it difficult for a single reward model to perform well on both. To address this issue, this disclosure trains two separate reward models: a reward model optimized for usefulness (Utility RM) and a reward model optimized for safety (Safety RM). Both models are initialized from a checkpoint of the pretrained chat model, allowing them to benefit from the knowledge gained from pretraining. Simply put, the reward model "knows" what the chat model already knows. This prevents, for example, cases where information mismatch between the two models could lead to hallucinations. The model architecture and hyperparameters are identical to those of the pretrained language model, except that the classification head for predicting the next token is replaced by a regression head for outputting a scalar reward.For training objectives, the pairwise human preference data collected to train the reward model is converted into a binary ranking label format (i.e., "select" or "reject"), and the selected response is forced to have a higher score than the corresponding response. This disclosure uses a binary ranking loss consistent with Ouyang et al. The binary ranking loss can be calculated as shown in Equation 3 below.

[0155] [Equation 3]

[0156]

[0157] Here, r-theta(x,y) is the scalar score output for prompt x and completion y with model weights theta. yc is the preferred response chosen by the annotator, and yr is the rejected response.

[0158] Based on this binary ranking loss, we further refine the following model for better utility and safety rewards. Given that the preference ratings are decomposed into a 4-point scale (e.g., "very good"), it may be useful to explicitly teach the reward model to assign more discrepancy scores to generations using this information.

[0159] The binary ranking loss with a margin component added to the loss can be calculated as shown in [Mathematical Formula 4] below.

[0160] [Equation 4]

[0161]

[0162] Here, the margin m(r) is a discrete function of the preference rating. This disclosure uses a large margin for pairs with different responses and a smaller margin for pairs with similar responses. This disclosure finds that this margin component can improve the accuracy of the utility reward model, especially in samples where the two responses are more separable.

[0163] Figure 9 illustrates model details, intended use, hardware and software, training data, and evaluation results. Referring to Figure 9, the paper on Llama 2 details the development of Llama 2, including pre-trained and fine-tuned models. These models, which have between 7 billion and 70 billion parameters and are optimized for conversational use cases, are referred to as Llama 2-Chat. Regarding pre-training and fine-tuning, Llama 2 was pre-trained on various datasets and fine-tuned using techniques such as supervised learning fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). This dual approach aims to improve performance in conversational systems by more closely matching human preferences and responses. Regarding safety and performance, a strong emphasis is placed on safety enhancements and performance evaluation. The Llama 2-Chat model underwent rigorous safety assessments and red team testing to identify and mitigate potential risks. Performance benchmarks show that the Llama 2-Chat model generally outperforms other open-source models in usability and safety metrics. Regarding community and openness, the developers of Llama 2 emphasize the importance of community engagement and open science. They have made the model and related tools available to encourage further research and advancement in the field of LLM. Regarding ethical and responsible use, the paper on Llama 2 discusses the ethical implications and responsible use of LLM. It specifically notes the need for careful implementation to avoid reinforcing bias or causing harm in applications that directly interact with humans.

[0164] FIG. 10 is a block diagram schematically illustrating a large-scale language model decoder according to an exemplary embodiment of the present disclosure, and FIG. 11 is a block diagram schematically illustrating one embodiment of the large-scale language model decoder of FIG. 10.

[0165] Referring to FIG. 10, the deminer encoder (1001) may correspond to the deminer encoder described above with reference to FIGS. 1, 3, and 4. The deminer encoder (1001) may input the feature closest to the synthesized IMU and LFA feature (450) among the video features stored in the cache model to the attention unit (1100) of the large-scale language model decoder (1000).

[0166] The audio encoder (1002) may be an embodiment of the behavioral encoder (112) of FIG. 1 and may correspond to the audio encoder described above with reference to FIG. 5. The audio encoder (1002) may output a text token, such as the text token (540) of FIG. 5. The text token and the instruction (1003) may be synthesized (1004). The synthesized token may be provided as an input token to a large-scale language model decoder (1000).

[0167] The large-scale language model decoder (1000) may include an architecture for a decoder of a transformer in Llama 2 or Llama 2Chat. For example, input tokens may be embedded, and root mean square (RMS) norming may be performed on the embedding vector. RoPE may be performed on the vector on which the RMS norming has been performed, and masked self-attention may be calculated on the result of the RoPE. The aforementioned process may be referred to as a masked self-attention unit.

[0168] The attention unit (1100) can calculate attention based on attention scores calculated by masked self-attention, an embedding vector, and features input from a demagnetizer encoder (1001).

[0169] The attention scores calculated by the attention unit (1100) can be output with probability values ​​through RMS Norm, feed forward, summation, RMS Norm, linearization, and softmax. The output with probability values ​​can correspond to the aforementioned interpretation data. The feed forward is composed of two fully connected layers with a nonlinear activation function (ReLU) in between, and the first fully connected layer can expand the input dimension, and the second fully connected layer can reduce the dimension back to the original dimension.

[0170] Referring to FIG. 11, the attention unit (1100) may include a first synthesis (1110), an RMS Norm (1120), a multi-head attention (1130), and a second synthesis (1140). Through the first synthesis (1110), attention scores calculated by masked self-attention, an embedding vector, and features input from a deminer encoder (1001) may be synthesized. An RMS Norm (1120) may be performed on the synthesized result, and a multi-head attention (1130) may be calculated on the result (e.g., an embedding) of the RMS Norm (1120) and the key, query, and value of the result. The attention scores calculated by multi-head attention (1130) and the results of performing RMS Norm (1120) can be synthesized through a second synthesis (1140).

[0171] The pre-trained multi-modal LLM according to the aforementioned embodiments can be applied as a commercially licensed model. Furthermore, the multi-modal LLM of the present disclosure can provide target outputs for describing the appearance of a companion animal (e.g., a puppy or dog). A high-performance MLLM, such as GPT-4, can be fed an image of the companion animal during an event and instructed to provide a detailed description of the companion animal's appearance. The multi-modal LLM of the present disclosure can provide target outputs for situations in which the companion animal is situated, based on provided experimental conditions, i.e., condition data.

[0172] Since each pet has a different temperament, the actual appearance of each pet can vary. Therefore, the multi-modal LLM of the present disclosure assigns and fine-tunes adapters based on LoRA (Huel et al., 2021) for each pet, enabling the model to perform more personalized inference for each pet. Adapters are generated based on socialization, activity level, trainability, and adaptability types, and there can be approximately 1,000 adapters in total. LoRA or S-LoRA may be suitable for the present disclosure due to their design, which aims to effectively learn and infer multiple adapters in a memory- and time-efficient manner. Furthermore, as the number of pets requiring optimization significantly increases, the multi-modal LLM of the present disclosure can provide interpretation information, such as the appearance or situation of the pet, by pre-training various adapters and assigning the most appropriate adapter to each new pet as it appears.

[0173] The multi-modal LLM of the present disclosure can be applied to healthcare AI. For example, it can detect respiratory disease in a companion animal from audio data and hip joint abnormalities in a companion animal from IMU data, thereby informing the user of the presence and improvement of the companion animal's condition. Furthermore, in veterinary hospitals, medical devices can collect additional data, such as electrocardiograms, electroencephalograms, and photoplethysmography (PPG), enabling the multi-modal LLM of the present disclosure to manage a wider range of diseases using healthcare AI.

[0174] FIG. 12 is a diagram illustrating a neural network according to an exemplary embodiment of the present disclosure. Specifically, FIG. 12 illustrates the structure of a convolutional neural network as an example of a neural network structure.

[0175] Referring to FIG. 12, a neural network (NN) may include a plurality of layers (L1 to Ln). Each of the plurality of layers (L1 to Ln) may be a linear layer or a non-linear layer, and in one embodiment, at least one linear layer and at least one non-linear layer may be combined and referred to as one layer. For example, the linear layer may include a convolution layer and a fully connected layer, and the non-linear layer may include a pooling layer and an activation layer.

[0176] For example, the first layer (L1) may be a convolutional layer, the second layer (L2) may be a pooling layer, and the nth layer (Ln) may be a fully connected layer serving as an output layer. A neural network (NN) may further include activation layers and may further include layers that perform other types of operations.

[0177] Each of the plurality of layers (L1 to Ln) can receive input data (e.g., an image frame) or a feature map generated from a previous layer as an input feature map, and generate a value return (QR) by operating on the input feature map. In one embodiment, the value return (QR) can be a value distribution including at least one Gaussian graph, or network parameters (e.g., a value mean value, a probability weight, a value standard deviation) corresponding to at least one Gaussian graph.

[0178] A feature map refers to data that expresses various characteristics of input data. The feature maps (FM1, FM2, FMn) may have, for example, a two-dimensional matrix or a three-dimensional matrix (or tensor) form. In one embodiment, the first feature map (FM1) input may be data corresponding to a current state. The feature maps (FM1, FM2, FMn) have a width (W) (or column), a height (H) (or row), and a depth (D), which may correspond to the x-axis, y-axis, and z-axis on a coordinate system, respectively. In this case, the depth (D) may be referred to as the number of channels.

[0179] The first layer (L1) can generate the second feature map (FM2) by convolving the first feature map (FM1) with a weight kernel (WK). The weight kernel (WK) can filter the first feature map (FM1) and may also be referred to as a filter or a map. The depth of the weight kernel (WK), i.e., the number of channels, is the same as the depth of the first feature map (FM1), i.e., the number of channels, and the same channels of the weight kernel (WK) and the first feature map (FM1) can be convolved. The weight kernel (WK) can be shifted in a manner that traverses the first feature map (FM1) using a sliding window. The amount of shifting can be referred to as a "stride length" or "stride."

[0180] During each shift, each of the weight values ​​included in the weight kernel (WK) may be multiplied and added to all pixel data in the area overlapping with the first feature map (FM1). The data of the first feature map (FM1) in the area where each of the weight values ​​included in the weight kernel (WK) overlaps with the first feature map (FM1) may be referred to as extracted data. As the first feature map (FM1) and the weight kernel (WK) are convolved, one channel of the second feature map (FM2) may be generated. Although one weight kernel (WK) is shown in Fig. 12, in reality, multiple weight maps may be convolved with the first feature map (FM1) to generate multiple channels of the second feature map (FM2). In other words, the number of channels of the second feature map (FM2) may correspond to the number of weight maps.

[0181] The second layer (L2) can generate a third feature map (FM3) by changing the spatial size of the second feature map (FM2) through pooling. Pooling may be referred to as sampling or down-sampling. A two-dimensional pooling window (PW) is shifted on the second feature map (FM2) in units of the size of the pooling window (PW), and the maximum value (or the average value of the pixel data) among the pixel data in the area overlapping with the pooling window (PW) may be selected. Accordingly, a third feature map (FM3) with a changed spatial size may be generated from the second feature map (FM2). The number of channels of the third feature map (FM3) is the same as the number of channels of the second feature map (FM2). In one embodiment, the third feature map (FM3) may correspond to the output feature map on which convolution is completed, as described above in FIG. 12.

[0182] The nth layer (Ln) can classify the class (CL) of input data by combining the features of the nth feature map (FMn). In addition, the nth layer (Ln) can generate a value return (QR) corresponding to the class. In an embodiment, the input data may correspond to data corresponding to the current state, and the nth layer (Ln) can generate a value return (QR) for determining an optimal action by extracting classes corresponding to multiple actions from the nth feature map (FMn) provided from the previous layer. The nth layer (Ln) can be performed by the fully connected module described above in FIG. 12.

[0183] According to one embodiment of the present disclosure, a value return (QR) can be expressed as a probability distribution of values ​​corresponding to each of a plurality of actions. In this specification, a neural network that generates a probability distribution for possible value returns for each current state-action pair, as described above, can be defined as a value distribution network. In one embodiment, the value distribution network can output network parameters defining the probability distribution for value returns as a deep learning result.

[0184] FIG. 13 is a diagram schematically illustrating an input feature map, a weight kernel, and an output feature map according to an exemplary embodiment of the present disclosure.

[0185] Referring to FIG. 13, input feature maps (201) include D channels, and the input feature map of each channel may have a size of H rows and W columns (D, H, W are natural numbers). Each of the kernels (202) has a size of R rows and S columns, and the kernels (202) may include a number of channels corresponding to the number of channels (or depth) (D) of the input feature maps (201) (R, S are natural numbers). Output feature maps (203) may be generated through a 3D convolution operation between the input feature maps (201) and the kernels (202), and may include Y channels depending on the convolution operation.

[0186] By performing a two-dimensional convolution operation between the input feature maps (201) of all channels and the kernels (202) of all channels, output feature maps (203) of all channels can be generated.

[0187] FIG. 14 is a flowchart illustrating an operation method of an electronic device according to an exemplary embodiment of the present disclosure.

[0188] Referring to FIG. 14, a step (S100) of encoding a behavioral data set is performed. The behavioral data set may include first audio data representing audio of a companion animal, second audio data representing the voice of the companion animal, video data representing a video of the companion animal, and IMU data representing IMU sensing results corresponding to the behavioral pattern of the companion animal. However, the present invention is not limited thereto.

[0189] A step (S200) of providing first encoded data and second encoded data to a large-scale language model decoder is performed. The first encoded data may include features of each of video data, IMU data, and first audio data. The second encoded data may include features of each of the first audio data and second audio data.

[0190] A step (S300) of calculating multi-head attention based on instructions, an embedding vector generated based on second encoded data, and first encoded data is performed. The step (S300) of calculating multi-head attention may be performed by the attention unit (1100) of FIG. 10.

[0191] A step (S400) of predicting interpretation data representing text interpreted as a companion animal based on attention scores of multi-head attention is performed.

[0192] FIG. 15 and FIG. 16 are block diagrams showing the structure of an encoder according to an exemplary embodiment of the present disclosure.

[0193] Referring to Fig. 15, individual sound encoder structures are disclosed. According to an exemplary embodiment, a Demeanor Encoder for cross-attention in a Large Language Model (LLM) decoder and a General Sound Encoder in an audio encoder that converts to text tokens can be used in each situation (General Sound Encoder 1 and 2 in Fig. 14). The encoders in Fig. 15 can produce optimized results for each learning and inference situation by individually providing encoders that can respond to each situation.

[0194] Referring to Fig. 15, a general-purpose (unified) sound encoder structure is disclosed. According to an exemplary embodiment, since the structure of the same encoder is learned, a single general-purpose sound encoder (General Sound Encoder 3 of Fig. 15) can be constructed, thereby allowing for the joint use of various general-purpose sound encoders. According to an exemplary embodiment, the general-purpose encoder structure of Fig. 15 can enable analysis for different purposes. For example, animal appearance realization and sound self-analysis can be performed simultaneously with a single encoder. However, there is a limitation that sufficient data is required, but the advantage of enabling model weight reduction is prominent.

[0195] Although the present disclosure has been primarily described with respect to electronic devices, it is not limited thereto.

[0196] While this disclosure has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will appreciate that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of this disclosure should be determined by the technical spirit of the appended claims.

Claims

1. An encoder that receives a behavioral data set including first audio data representing audio of a companion animal, second audio data representing the companion animal's voice, video data representing a video in which the companion animal was filmed, and IMU data representing an IMU sensing result corresponding to a behavioral pattern of the companion animal, encodes the behavioral data set, and outputs first encoded data including features of each of the video data, the IMU data, and the first audio data, and second encoded data including features of each of the first audio data and the second audio data; and A large-scale language model decoder that receives an instruction, the first encoded data, and the second encoded data, calculates an embedding vector generated based on the instruction and the second encoded data, and a multi-head attention based on the first encoded data, and predicts interpretation data representing a text interpreted about the companion animal as an output based on attention scores of the multi-head attention, The above encoder, An electronic device characterized by comprising a de-interlacer encoder that encodes the video data, the IMU data, and LFA data of the first and second audio data and provides the first encoded data to the large-scale language model decoder, and an audio encoder that encodes the LFA data, the second audio data, and HFA data of the first audio data and outputs the second encoded data.

2. In paragraph 1, The above encoder, An electronic device characterized by encoding the video data, the IMU data, the first and second audio data, and the all-factory data.

3. In paragraph 2, The above demagnetizing encoder is, A video encoder encoding the video data to generate at least one video feature corresponding to a feature of the video; An IMU encoder that encodes the IMU data to generate an IMU feature corresponding to a feature of the IMU; An LFA encoder that encodes the above LFA data and generates an LFA feature corresponding to a feature related to the LFA; A synthesis unit that synthesizes the IMU features and the LFA features to generate synthesized IMU and LFA features; and An electronic device characterized by comprising a bind network that receives at least one video feature and the synthesized IMU and LFA features, and provides the video feature closest to the synthesized IMU and LFA features to the large-scale language model decoder.

4. In paragraph 3, The above bind network is, An electronic device characterized in that it includes a cache model that stores video features obtained during a learning process and, when the synthesized IMU and LFA features are input, provides the video feature closest to the synthesized IMU and LFA features among the stored video features to the large-scale language model decoder.

5. In paragraph 2, The above audio encoder, A voice identifier that receives the LFA data and the second audio data and identifies whether at least a portion of the voice in the LFA data corresponds to the voice of the companion of the second audio data; A speech encoder that receives the LFA data and analyzes the voice of the LFA data by encoding the LFA data; and An electronic device characterized by including a sound encoder that receives the LFA data and the HFA data, converts the HFA of the HFA data into LFA, and analyzes environmental sounds and pet sounds from the LFA.

6. In paragraph 5, An electronic device further comprising a Q-Former that generates text tokens to be input to the large-scale language model decoder based on the analysis results of the voice identifier, the analysis results of the speech encoder, the analysis results of the sound encoder, and learnable queries.

7. In paragraph 1, The above large-scale language model decoder is, An electronic device characterized in that, based on the synthesis of the second encoded data and the instructions, an input torque is generated, embedding is performed on the input torque to generate an embedding vector, a first RMS Norm (Root Mean Square Layer Normalization) is performed on the embedding vector, a RoPE (Rotary Positional Embeddings) is performed on the vector on which the first RMS Norm is performed, masked self-attention is calculated on the embedding on which the RoPE is performed, and the multi-head attention is calculated based on attention scores of the masked self-attention, the embedding vector, and a video feature closest to the IMU and LFA features synthesized from the first encoded data.

8. In paragraph 7, The above large-scale language model decoder is, An electronic device characterized in that it synthesizes the attention scores of the masked self-attention, the embedding vector, and the video feature, performs a second RMS Norm on the synthesized result, calculates the multi-head attention for the embedding on which the second RMS Norm has been performed and the key, query, and value of the embedding, and synthesizes the attention scores of the multi-head attention and the embedding on which the second RMS Norm has been performed.

9. In paragraph 8, The above large-scale language model decoder is, An electronic device characterized in that it performs a third RMS Norm on a result synthesized from the attention scores of the multi-head attention and the embedding on which the second RMS Norm is performed, performs a feedforward on the embedding on which the third RMS Norm is performed, synthesizes the result of the feedforward and the embedding on which the third RMS Norm is performed, performs a fourth RMS Norm on a result synthesized from the result of the feedforward and the embedding on which the third RMS Norm is performed, linearizes the embedding on which the fourth RMS Norm is performed, and outputs the interpretation data using the linearized result and softmax.

10. A step of encoding a behavioral data set including first audio data representing audio of a companion animal, second audio data representing the voice of the companion animal, video data representing a video of the companion animal being filmed, and IMU data representing an IMU sensing result corresponding to a behavioral pattern of the companion animal; Providing first encoded data including low frequency band features of each of the video data, the IMU data, and the first audio data, and second encoded data including high frequency band features of each of the first audio data and the second audio data, to a large-scale language model decoder; An instruction, and an embedding vector generated based on the second encoded data, a step of calculating multi-head attention based on the first encoded data; and A method of operating an electronic device, comprising a step of predicting interpretation data representing a text interpreted about the companion animal based on the attention scores of the multi-head attention.

Images