Generative AI System for Real-Time Services by Integrating H-MoE Based High-Performance Lightweight Language Models with RAG on On-Device CPU Environments And Operation Method Thereof

Abstract

The present invention discloses a generative AI system for real-time services combining a high-performance lightweight language model based on H-MoE and RAG, executable in an on-device CPU environment, which can provide smooth and reliable real-time services in an on-device CPU environment by applying high-performance lightweight LLM-RAG technology and configuring the model as H-MoE, and providing answers or results through an appropriate Expert Model based on multimodal context recognition. The present invention comprises: an LLM module equipped with an internal database based on multimodal data including text, sensor data, and image data, and external knowledge data based on RAG, wherein semantic-based search is performed by embeddings; and a dynamic emotion recognition module in which the influence of elements constituting a space of lighting, color, and sound on user emotion is analyzed based on natural language understanding and sensor data of the LLM module.

Description

Technology Field

[0001] The present invention relates to a generative AI system, specifically a generative AI system for real-time services combining an H-MoE-based lightweight language model and RAG, and more specifically, a generative AI system for real-time services combining an H-MoE-based lightweight language model and RAG that is executed in an on-device CPU environment. More specifically, the present invention relates to a generative AI system for real-time services combining an H-MoE-based high-performance lightweight language model and RAG that is executable in an on-device CPU environment and a method of operation thereof, which applies high-performance lightweight LLM-RAG technology and configures the model as a Hierarchical Mixture of Experts (H-MoE), and provides answers or results through an appropriate Expert Model based on multimodal context recognition, thereby providing smooth and reliable real-time services in an on-device CPU environment. Background Technology

[0002] Generative AI is artificial intelligence capable of generating new content, such as text, images, audio, and code, by learning the patterns and structures of existing data. As the use of generative AI tools like ChatGTP, Gemini, and Deepseek becomes widespread, global interest in generative AI development is growing. Following the recent Deepseek incident, AI is transitioning from a cloud-centric approach to an edge- and on-device model. Furthermore, Deepseek's announcement of a low-cost, high-performance model sent shockwaves through the global market.

[0003] To complement the limitations of large-scale, high-cost, and resource-intensive generative AI models, interest in small LLMs (sLLM) is increasing, and interest in lightweight technologies applicable to on-device and edge AI is also growing. Prior art literature

[0004] Korean Registered Patent Publication No. 10-2854468 Korean Published Patent Publication No. 10-2025-0125303 Korean Published Patent Publication No. 10-2025-0124333 The problem to be solved

[0005] The present invention aims to solve the aforementioned problems by applying high-performance lightweight LLM-RAG technology and configuring the model as H-MoE, and by providing a generative AI system for real-time services combining a high-performance lightweight language model and RAG executable in an on-device CPU environment, and a method of operation thereof, which can provide smooth and reliable real-time services in an on-device CPU environment by providing answers or results through an appropriate Expert Model based on multimodal context recognition. means of solving the problem

[0006] A generative AI system for real-time services combining a high-performance lightweight language model based on H-MoE and RAG, executable in an on-device CPU environment according to an embodiment of the present invention for achieving the above-mentioned purpose, comprises: an LLM module in which semantic-based search is performed by embedding, and which is equipped with an internal database unit based on multimodal data including text, sensor data, and image data, and an external knowledge database unit based on RAG; and a dynamic emotion recognition module in which the influence of elements constituting the space of lighting, color, and sound on user emotion is analyzed based on natural language understanding of the LLM module and sensor data.

[0007] The above LLM module includes an internal database unit in which multimodal data is collected, preprocessed, and stored; an external knowledge database unit in which domain-filtered document data is stored after being sorted; an AI learning unit in which data stored in the internal database unit and the external knowledge database unit is learned; and a semantic search unit that performs search using meaning-based search by embeddings based on the content learned in the AI ​​learning unit. The LLM module is characterized by selecting and using one or more of bit-flexible quantization, knowledge distillation, and post-precision supplementation techniques as quantization technologies for lightweighting.

[0008] The above external knowledge database unit further includes at least one of a similarity sorting unit, a common field determination unit, and a data search request unit, wherein the similarity sorting unit calculates the cosine value between two vectors after the document sorting is converted into vector data and determines the similarity with a similarity score value having a similarity range of -1 to 1 for the corresponding field so that multiple documents are sorted, the above common field determination unit determines the number of documents corresponding to a common field that exceeds both the average value of the similarity score value for the first field and the average value of the similarity score value for the second field when the corresponding field is defined as having different first and second fields, and the above data search request unit performs additional data search through external search when the number of documents corresponding to the common field is less than a certain number.

[0009] For example, document data in the legal field, which is the first field, is converted into vector data and the cosine value is calculated to determine the range of similarity, and document data in the disaster safety field, which is the second field, is converted into vector data and the cosine value is calculated to determine the range of similarity, and the number of document data satisfying all conditions where the vector data in the legal field, which is the first field, exceeds a cosine value of 0.5 and the vector data in the disaster safety field, which is the second field, exceeds a cosine value of 0.5 is determined, and if the number is less than the pre-set 50, documents corresponding to the common field of the legal field, which is the first field, and the disaster safety field, which is the second field, are additionally searched to supplement the data in the common field of the external knowledge database, thereby securing a certain amount of document data regarding laws related to disaster safety rather than regarding independent topics of law and disaster safety.

[0010] The above external knowledge database unit further comprises a time similarity weighting unit, wherein if the document data is a paper, the time similarity weighting unit assigns weights to similarity score values ​​linked to the publication time of the paper, and a higher weight is assigned the closer the publication time of the paper is to the similarity judgment time.

[0011] For example, when the similarity score for Paper A is calculated as 0.5 and the similarity score for Paper B is calculated as 0.6, if Paper A was published one year ago and a weight of 1 is assigned, and Paper B was published 10 years ago and a weight of 0.8 is assigned, the weighted similarity scores are calculated as 0.5 for Paper A and 0.48 for Paper B, so the sorting based on similarity scores differs depending on the publication date. Therefore, it becomes possible to reflect similarity scores by considering the latest research trends along with relevance to the field.

[0012] The above external knowledge database unit further comprises a volume similarity weighting application unit, wherein the volume similarity weighting application unit assigns weights to similarity score values ​​linked to the volume of the document, and assigns higher weights as the volume of the document increases.

[0013] For example, when the similarity score for Document A is calculated as 0.5 and the similarity score for Document B is calculated as 0.6, the length of Document A is 10 pages and the weight is calculated as 1, and the length of Document B is 1 page and the weight is calculated as 0.7, so the sorting based on the similarity score differs depending on the document length. Therefore, it is possible to reflect the similarity score by taking into account the length of the document.

[0014] The dynamic emotion recognition module described above includes a user analysis unit in which the user's emotional state is analyzed through natural language understanding of the user's language, a spatial element analysis unit in which spatial characteristics are analyzed through the interaction between spatial components of lighting, color, and sound of the space where the user is located, and a spatial element recommendation unit in which spatial components suitable for the user are recommended through time-series analysis of changes in the user's emotional state by the user analysis unit and the spatial element analysis unit.

[0015] The above spatial element recommendation unit includes a lighting recommendation unit that recommends a color temperature according to the user's emotional state index. Here, when the emotional state index is defined as having a high emotional state index when the user is positive and a low emotional state index when the user is negative, a warm and cozy color temperature of 2700K to 3000K is recommended when the user has a high emotional state index, and a cool and lively color temperature of 40000K to 6500K is recommended when the user has a low emotional state index.

[0016] The lighting recommendation unit recommends the recommended color temperature in conjunction with the emotional state change slope, which is the change in the emotional state index over time, and the lower the color temperature value, the greater the emotional state change slope. Here, the emotional state change slope is dy / dx, where the x-axis represents time and the y-axis represents the emotional state index, and the greater the emotional state change slope, the greater the change in the emotional state over time.

[0017] For example, for User A, if the slope of the change in emotional state over 10 seconds is 10 (dy / dx=1) based on acoustic time series analysis, a color temperature of 2700K is recommended, and for User B, if the slope of the change in emotional state over 10 seconds is 5 (dy / dx=0.5) based on acoustic time series analysis, a color temperature of 3000K is recommended. Therefore, even if the average value of the emotional state index of User A and User B is the same over a certain period, a lower color temperature is recommended when the range of change in emotional state is larger, thereby enabling the recommendation of a color temperature that takes into account the change in emotion based on time series analysis.

[0018] The above LLM module is characterized by collecting data specialized for Korean search to improve Korean information retrieval performance, fine-tuning the above AI learning unit exclusively for Korean, and retraining the above AI learning unit using a hard negative mining technique.

[0019] The above LLM module is characterized by artificial intelligence being trained through ethics and sociality datasets for public service application.

[0020] In addition, a method of operation for a generative AI system for real-time services combining a high-performance lightweight language model based on H-MoE and RAG, executable in an on-device CPU environment according to an embodiment of the present invention for achieving the above-mentioned purpose, comprises: a first step in which multimodal data is collected, preprocessed, and stored; a second step in which domain-filtered document data is stored after being sorted; a third step in which the data stored in the first and second steps is learned; a fourth step in which a search is performed by semantic-based search using embeddings based on the content learned in the third step; a fifth step in which the user's emotional state is analyzed through natural language understanding of the user's language in the fourth step; a sixth step in which the user's emotion is analyzed through the interaction between spatial components such as lighting, color, and sound of the space where the user is located; and a seventh step in which spatial components suitable for the user are recommended through time-series analysis of changes in the user's emotional state. Effects of the invention

[0021] As described above, the present invention enables the reduction of computational costs and the maintenance of high performance through sLLM quantization technology, and enables the implementation of lightweight generative AI models optimized for medical, educational, manufacturing, and financial fields.

[0022] In addition, the semantic search technology based on domain-specific data of the present invention improves the accuracy of information retrieval, and the hallucination prevention technology increases the reliability of generated responses.

[0023] In addition, the edge-specific lightweight generative AI of the present invention can protect personal information and ensure real-time responsiveness by performing local inference. Brief explanation of the drawing

[0024] Figure 1 is a configuration diagram of a generative AI system for real-time services combining an H-MoE-based high-performance lightweight language model and RAG that can be executed in an on-device CPU environment according to the present invention. Figure 2 is a configuration diagram of the LLM module of the present invention. Figure 3 is a conceptual diagram showing the process of semantic-based search being performed in the semantic search unit of the present invention. Figure 4 is a conceptual diagram of knowledge distillation. Figure 5 is an example of a dataset embedded in the external knowledge database section of the present invention. Figure 6 is a configuration diagram of the dynamic emotion recognition module of the present invention. FIG. 7 is a conceptual diagram of the dynamic emotion recognition module of the present invention. FIG. 8 is an example diagram of the past context reconstruction technique of the spatial element recommendation unit of the present invention. Figure 9 is a conceptual diagram of an LLM algorithm with enhanced ethics and sociality. Specific details for implementing the invention

[0025] Expressions such as “comprising” or “may comprise” that may be used in various embodiments of the present disclosure indicate the presence of the disclosed corresponding function, operation, or component, etc., and do not limit one or more additional functions, operations, or components, etc. Furthermore, in various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0027] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components. Such expressions may be used to distinguish one component from another. For example, the first learner device and the second learner device are both learner devices and represent different learner devices. For example, without departing from the scope of the various embodiments of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0028] When it is stated that a component is "connected" or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that a new component may exist between the component and the other component. On the other hand, when it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that no new component exists between the component and the other component.

[0029] In embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0030] The terms used in the various embodiments of this disclosure are used merely to describe specific embodiments and are not intended to limit the various embodiments of this disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the various embodiments of this disclosure pertain.

[0032] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0034] Hereinafter, a generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram of a generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to the present invention. Referring to FIG. 1, the generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to the present invention may be configured to include an LLM module (100) and a dynamic sentiment recognition module (200). The generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model and RAG, executable in an on-device CPU environment according to the present invention, may include one or more processors, one or more memories, one or more storage, and one or more communication interfaces, and these may be connected to each other via a bus. In addition, the generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model and RAG, executable in an on-device CPU environment, may include hardware such as input devices and output devices. Furthermore, the generative AI system (10) for real-time services combining an H-MoE-based high-performance lightweight language model and RAG, executable in an on-device CPU environment, may be equipped with various software, including an operating system capable of running programs. Additionally, the term "module" used in the present invention should be interpreted as including software, hardware, or a combination thereof, depending on the context in which the term is used.For example, software may be machine language, firmware, embedded code, and application software. As another example, hardware may be a circuit, processor, computer, integrated circuit, integrated circuit core, sensor, Micro-Electro-Mechanical System (MEMS), passive device, or a combination thereof.

[0035] The LLM module (100) is equipped with an internal database unit (110) based on multimodal data including text, sensor data, and image data, and an external knowledge database unit (120) based on RAG, and performs semantic-based search based on embeddings. FIG. 2 is a configuration diagram of the LLM module (100) of the present invention. Referring to FIG. 2, the LLM module (100) of the present invention may be configured to include an internal database unit (110), an external knowledge database unit (120), an AI learning unit (130), and a semantic search unit (140). In the present invention, to implement on-device AI, the LLM module (100) uses an LLM (Large Language Model) or a sLLM (smaller Large Language Model). Compared to conventional large language models, sLLM has the advantage of reducing the number of parameters, thereby reducing training costs and time, while allowing it to achieve performance similar to large language models in specific desired fields through fine-tuning.

[0036] The internal database unit (110) collects and preprocesses multimodal data and stores it. Here, multimodal data includes text data, image data, voice data, video data, and various sensor data. The internal database unit (110) preprocesses the collected multimodal data for the learning of artificial intelligence. Text data preprocessing includes text normalization, which is a process to maintain the consistency of text data, removal of special characters, tokenization which separates text into token units, removal of stop words, and extraction of stems or lemmas. Additionally, image data preprocessing includes image resizing, image cropping, image transparency processing, image sharpening, image contrast enhancement, color separation, image binarization, background removal, boundary detection, and corner detection. Additionally, voice data preprocessing includes trimming, which processes sounds below 60dB as silent, and random padding, which sets a fixed size and randomly pads the front and back, as the lengths of the voice data differ, such as Melspectrogram and MFCC. In addition, image data preprocessing includes data format conversion from color to grayscale, size conversion, and pixel value conversion. Additionally, sensor data preprocessing includes sorting numeric data according to data items. The preprocessed data is stored by data embedding. Embedding is the process of generating high-dimensional vectors that reflect the meaning of words, sentences, or other data in a high-dimensional space, and through embedding, the computer can effectively understand and process the meaning of language. In particular, in the case of LLM, embedding plays a key role in converting natural language into numbers during fine-tuning or Retrieval-Augmented Generation (RAG). The internal database unit (110) is equipped with a storage memory capable of storing the embedded data.The memory may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk.

[0037] The external knowledge database unit (120) stores domain-filtered document data after sorting the documents. The LLM module (100) of the present invention can provide more accurate and up-to-date information by combining the external knowledge database unit (120) with the internal database unit (110) through Search Augmentation Generation (RAG). The external knowledge database unit (120) stores in-depth specialized knowledge of a specific domain that is domain-specific. The external knowledge database unit (120) filters the document data to match the specific domain. In the present invention, the specific domain may include fields such as news, law, medical, Korean language, disaster safety, and smart healthcare, and the data format is limited to document data having text. The external knowledge database unit (120) sorts the document data filtered to match the specific domain, and the sorting is performed by calculating a Similarity Score. Similarity includes Mean Squared Difference Similarity, Cosine Similarity, and Pearson Similarity; in one embodiment of the present invention, Cosine Similarity may be used. Mean Squared Difference Similarity is calculated as the reciprocal of the square of the difference between the MSD between User u and User v and the ratings of the documents rated by User u and User v, divided by the number of documents rated by both User u and User v; the larger the difference, the smaller the similarity value. Cosine Similarity expresses the degree of similarity between two feature vectors as a cosine value, ranging from -1 to 1, where -1 indicates completely opposite cases, 0 indicates independent cases, and 1 indicates completely identical cases. Pearson Similarity refers to the correlation coefficient between two vectors, with a value of 1 when similarity is highest and -1 when it is lowest.Since extreme low or high score criteria for specific documents can significantly affect similarity, a correlation coefficient is used to prevent this. Document data sorted according to similarity scores is embedded and stored in an external knowledge database unit (120). The external knowledge database unit (120) is equipped with a storage memory capable of storing the embedded data. The type of memory is the same as that described in the internal database unit (110) above, so it is omitted.

[0038] The AI ​​learning unit (130) learns data stored in the internal database unit (110) and the external knowledge database unit (120). The AI ​​learning unit (130) first learns the data embedded in the internal database unit (110) and the external knowledge database unit (120), and then relearns the data embedded in the external knowledge database unit (120) through fine-tuning. Since the LLM module (100) of the present invention operates based on RAG, the AI ​​learning unit (130) that has learned the data in the internal database unit (110) performs domain-specific deep learning through the external knowledge database unit (120), the deep-learned content is fine-tuned after evaluation, and the fine-tuned data in the external knowledge database unit (120) is relearned again.

[0039] The semantic search unit (140) performs a search based on the content learned by the AI ​​learning unit (130) using semantic-based search through embeddings. Semantic search refers to a search performed by understanding the questioner's intent rather than searching based on keywords of the search term. Keyword-based search has the disadvantage that it is difficult to consider similar meanings and difficult to identify the relationships between words. Semantic search produces results by matching concepts rather than keywords; it represents words as concepts through vector dimension embeddings and finds content that matches the meaning of the words included in the user query to display search results. Semantic search has the advantage of high utility and high search efficiency because it is based on vector search and can obtain intuitive search results. Vector search can search for items that are semantically similar or semantically related. This allows for the extraction of accurate search results by comparing similarities between vectors, by storing information using vectors for unstructured data such as voice and images. Since the internal database unit (110) of the present invention stores unstructured data rather than text, semantic-based search is required for efficient searching. FIG. 3 is a conceptual diagram illustrating the process of semantic-based search being performed in the semantic search unit (140) of the present invention. Referring to FIG. 3, the semantic search unit (140) embeds sentences for an input sentence and finds the closest sentence based on cosine similarity to derive a search result. The user inputs a question of desired content into an input device, and the semantic search unit (140) performs a semantic-based search of the question content and outputs an answer to the user through an output device.

[0040] The LLM module (100) may select and use one or more of bit-flexible quantization, knowledge distillation, and post-precision supplementation techniques as quantization techniques for lightweighting. The present invention enables edge computing by reducing the size of the system through lightweighting of the language model. First, unlike the conventional method of applying the same precision uniformly to all parameters, bit-flexible quantization can be applied, which applies a differential number of bits according to the importance of each parameter. This approach maintains parameters where precision is important at high precision (e.g., 8 bits) and compresses parameters of lower importance to a lower number of bits (e.g., 4 bits, 2 bits), thereby reducing the total computational load and memory usage of the model while minimizing performance degradation. Through such quantization-friendly learning and structural design, the model can be equipped with a foundation to maintain high prediction precision and efficiency even after lightweighting.

[0041] Next, knowledge distillation for improving the performance of lightweight artificial intelligence models efficiently transfers knowledge learned from a high-performance, large-scale pre-trained model (Teacher) to a small model (Student), thereby reducing computational load and memory usage while maintaining high prediction performance. In other words, it utilizes the prediction information of the high-performance Teacher model to induce the smaller Student model to mimic the Teacher's learning results. Figure 4 is a conceptual diagram of knowledge distillation.

[0042] The knowledge distillation algorithm according to one embodiment of the present invention is characterized by a method of training the Student model to mimic the probability distribution information output by the Teacher model, and unlike traditional supervised learning that uses only correct labels, it utilizes the Teacher's entire output distribution (soft target) for learning to deliver richer representational information to the Student model. Therefore, it enables learning of the Softmax probability distribution along with the correct labels. Here, the Softmax probability distribution is a probability distribution that uses the Softmax function, and the Softmax function is an activation function used to calculate probability values ​​in multi-class classification. The total loss function (L) used in this learning structure total ) is defined by Equation 1 as follows.

[0043] (Equation 1) L total = α × L hard + (1 - α) × L soft

[0044] Here, α is a hyperparameter that balances the importance of the two terms, and L hard is the cross-entropy loss for the ground truth data, and L soft is the cross-entropy loss for the smoothly processed output of Teacher.

[0045] L hard is the predicted output s of the Student model i and the actual correct label y i It is defined as the cross-entropy loss between and is given by Equation 2 below.

[0046] (Equation 2) L hard = - ∑ y i log(s i )

[0047] Also, L soft is the output distribution t of the Teacher model i (T) and the output distribution s of the Student modeli It is defined based on the Kullback-Leibler Divergence (KL Divergence) between (T) and is expressed by the following Equation 3:

[0048] (Equation 3) L soft = ∑ t i (T) log(t i (T) / s i (T))

[0049] Here, T is a temperature parameter; it is desirable to smooth the Softmax output distribution so that the Student model can learn minute probability differences from the Teacher model more precisely. This knowledge distillation technique enables the Student model to effectively transfer the Teacher model's ability to learn complex representations while maintaining a relatively small number of parameters and computational load. In particular, it can be utilized as a means to compensate for the inevitable degradation in accuracy that occurs during the lightweighting process of quantization-based models.

[0050] Next, posterior precision compensation involves applying bit-level quantization (e.g., 8-bit, 4-bit) to the trained Student model and performing fine-tuning via additional training using the Teacher output even after quantization. This compensates for the accuracy degradation that may occur during the quantization process and maintains high-precision predictions. This process is crucial for ensuring reliability and precision in real-world application environments while optimizing the real-time processing performance, inference speed, and memory usage of lightweight models.

[0051] The LLM module (100) collects data specialized for Korean search to improve Korean information search performance, fine-tunes the AI ​​learning unit (130) exclusively for Korean, and retrains the AI ​​learning unit (130) using a hard negative mining technique. Since the present invention is specialized for Korean information search among the generated AIs, the external knowledge database unit (120) collects data specialized for Korean search to improve Korean information search performance. FIG. 5 is an example of a dataset embedded in the external knowledge database unit (120) of the present invention. The AI ​​learning unit (130) is trained using data embedded in the internal database unit (110), and the AI ​​learning unit (130) is fine-tuned exclusively for Korean by the Korean search specialized data stored in the external knowledge database unit (120) so that it can be specialized for Korean search.

[0052] Hard Negative Mining is a method that identifies hard negatives among similar sentences based on similarity or statistics between embedding vectors. Here, a hard negative refers to data that is actually negative but is easily mispredicted as positive. Therefore, errors can be reduced by collecting hard negatives and retraining the model. In this invention, for hard negative mining, the advanced loss function GISTEmbedLoss(L, presented in Equation 4 below) is used. G Apply ).

[0053] (Equation 4)

[0054] Here, q i is the embedding vector, p i + is a positive embedding vector, p j - is the negative embedding vector, G B means guide model-based placement negative.

[0055] GISTEmbedLoss(LG To obtain ), the first embedding vector q from the input sentence i Generates a positive embedding vector p from a second sentence semantically similar to the input sentence. i + Generates a set of negative embedding vectors {p} from one or more hard negative sentences that are similar in expression but different in semantics to the input sentence. j - Generate} and embedding vector q i, p i +, p j - Measuring similarity between them GISTEmbedLoss(L G Calculate )

[0056] According to the above operation, by actively utilizing not only high-quality positive samples with similar meanings within the training batch but also hard negative samples where the sentences are similar but have different meanings, the model is trained to precisely distinguish actual semantic differences rather than relying solely on word similarity. GISTEmbedLoss is a loss function designed to precisely reflect semantic similarity and difference between sentences, and is optimized for learning an embedding space that can effectively separate sentences that have similar expressions but need to be distinguished. In particular, by reflecting the characteristics of languages ​​such as Korean, which have flexible word order, many morphological changes, and frequent ambiguous expressions, it strengthens classification ability based on actual meaning rather than simple sentence structure similarity. Through this, the AI ​​learning unit (130) can more accurately grasp the meaning between sentences and learns high-performance embedding expressions that can simultaneously improve the accuracy (Recall) and precision (Precision) of search results. Consequently, this method serves as a core technology for implementing a sentence embedding model with excellent semantic distinction capabilities in the Korean natural language processing environment of the AI ​​learning unit (130).

[0057] The above LLM module (100) is characterized by using a Mixture of Experts (H-MoE) to implement on-device edge computing through lightweighting. First, Mixture of Experts (MoE) is a machine learning technique in which a complex input space is divided and processed by multiple expert networks. A typical mix of experts consists of one gating network and two or more expert networks. Expert networks are specialized in specific parts of the overall task, and the gating network determines weights for the processing results of multiple expert networks. Through this, a complex task is divided into several simple tasks, and the existing complex task is solved by each expert network specializing in different tasks. A Mixture of Experts (H-MoE) can effectively process more complex tasks by organizing this hierarchically. The Mixture of Experts model solves hierarchical classification problems using a Local Classifier per Parent Node (LCPN) approach based on a Class Hierarchy that follows a taxonomy. The class hierarchy is structured as a tree, and by designating an expert network as a local classifier at each parent node, each expert network is specialized in the task of classifying an image into one of the child classes at that node. Additionally, the expert located at the parent node acts as a gating network that activates an expert at one child node based on the prediction result. The LLM module (100) of the present invention solves the hierarchical classification problem resource-efficiently through hierarchical expert mixing, and by reducing the training time and memory usage required per round through this method, it enables participation in federated learning even with edge computing that has limited computing resources. Furthermore, through hierarchical expert mixing, it provides answers or results through an appropriate expert model based on multimodal context recognition.

[0059] The dynamic emotion recognition module (200) analyzes the influence of elements constituting the space of lighting, color, and sound on user emotion based on natural language understanding and sensor data of the LLM module (100). FIG. 6 is a configuration diagram of the dynamic emotion recognition module (200) of the present invention, and FIG. 7 is a conceptual diagram of the dynamic emotion recognition module (200) of the present invention. Referring to FIG. 6, the dynamic emotion recognition module (200) of the present invention may be configured to include a user analysis unit (210), a spatial element analysis unit (220), and a spatial element recommendation unit (230).

[0060] The user analysis unit (210) analyzes the user's emotional state through natural language understanding of the user's language. The user analysis unit (210) precisely recognizes the user's emotional state through natural language semantic relationship analysis of the LLM module (100). The LLM module (100) converts the input sentence into an embedding and then adjusts the attention layer to generate a vector representation suitable for the domain. The attention layer-based emotion recognition technology has limitations in identifying semantic relationships within a sentence because it relies on class tokens for emotion classification. The user analysis unit (210) reconstructs the natural language expression information embedded in the LLM module (100) into a single special token using natural language semantic relationship analysis technology, and analyzes the user's emotional state by understanding the semantic relationships of the input sentence and maintaining consistency through class token and weight adjustment.

[0061] The spatial element analysis unit (220) analyzes the user's emotions through the interaction between spatial components of lighting, color, and sound in the space where the user is located. In lighting elements, a low color temperature induces comfort and stability, while a high color temperature induces concentration and alertness. In color elements, blue tones induce calmness, while red tones induce vitality or tension. In sound elements, low-pitched and slow-motion sounds induce relaxation, while high-pitched and fast-motion sounds induce stimulation or tension. The spatial element analysis unit (220) detects the lighting, color, and sound of the space where the user is located through sensor data collected from the internal database unit (110). Lighting is measured through an illuminance sensor, color is measured through a color sensor, and sound is measured through a sound sensor. The measured sensor values ​​are preprocessed in the internal database unit (110), and the spatial element analysis unit (220) analyzes the preprocessed data to analyze the user's emotions. In the lighting element, color temperature is analyzed primarily, in the color element, color series is analyzed primarily, and in the sound element, pitch and speed of sound are analyzed primarily. Meanwhile, since spatial elements such as lighting, color, and sound do not operate independently and complex emotional changes occur depending on their interaction, the spatial element analysis unit (220) of the present invention projects the non-linear interaction between spatial elements into a latent space and quantifies the comprehensive impact on user emotions, thereby enabling the prediction of complex emotional responses based on multidimensional interaction.

[0062] The spatial element recommendation unit (230) recommends spatial components suitable for the user through time-series analysis of changes in the user's emotional state by the user analysis unit (210) and the spatial element analysis unit (220). Since the user's emotional state is maintained or changes depending on the time point, time-series-based emotional analysis is essential, and the spatial element recommendation unit (230) infers and recognizes the user's temporal emotional changes. Sentiment analysis at a specific point in time has limitations in recognizing emotional changes between points in time and applying past points in time, and past context reconstruction technology can reconstruct user emotions from multiple points in time into a time series to maintain the direction of emotional changes and past points in time. The spatial element recommendation unit (230) reconstructs past contexts from the conversation records of the semantic search unit (140), analyzes the user's emotions in a time series based on the context embedding of the LLM, and recommends spatial components suitable for the user based on the analysis of spatial elements of the current space. The external knowledge database unit (120) stores the user's conversation history through the semantic search unit (140), so that the spatial element recommendation unit (230) can use it as a RAG dataset. FIG. 8 is an example of the past context reconstruction technique of the spatial element recommendation unit (230) of the present invention. Once the user's sentiment is analyzed in a time series, the spatial element analysis unit (220) refers to the analyzed spatial elements to recommend spatial components suitable for the user. For example, low color temperature and blue tones induce a sense of stability and calmness, making them suitable for a rest-centered space, so they can be recommended when the user needs rest. High color temperature and high-speed sound induce vitality and alertness, making them suitable for a workspace, so they can be recommended when the user is working. Red tones and high-pitched sounds induce anxiety and stress, making them unsuitable for work and rest spaces, so they are not recommended when the user is working.

[0063] The LLM module (100) can be trained on artificial intelligence through ethical and social datasets for public service application. In order to become a generative AI model suitable for public service fields such as civil complaints, administration, and welfare, it is particularly desirable to technically implement or verify the required legal, ethical, and social acceptability. To this end, it is intended to include a generative AI architecture tailored to the actual service flow of public institutions, privacy protection, and social values. It is desirable to design the generative AI to provide accurate and explainable responses to actual public queries and to secure dynamic response adjustment capabilities that satisfy both administrative objectives and social expectations. Furthermore, it includes establishing a verification framework capable of automatically evaluating the ethicality, acceptability, and reliability of response results, and conducting empirical experiments in a testbed tailored to the public service environment to quantitatively prove the social suitability and applicability of the technology. FIG. 9 is a conceptual diagram of an LLM algorithm with enhanced ethical and social capabilities. The LLM module (100) embeds and stores a dataset with enhanced ethics and sociality in an internal database unit (110) to enhance ethics and sociality, and learns the stored dataset by an AI learning unit (130). The learned AI learning unit (130) can be retrained through fine-tuning to significantly enhance ethics and sociality.

[0064] The above LLM module (100) is characterized by selecting and using one or more of the following for multimodality context recognition of the AI ​​learning unit (130): the Neural Architecture Search (NAS) technique, the Latency-pruned technique, and the Token Embedding structure technique. NAS refers to a technology in which a computer automatically searches for and finds the optimal architecture without a human directly designing the structure of a deep learning model, and includes the following three core components. The first component is the Search Space, which defines which structures the NAS can select from. For example, a range of combinations of all possible model structures is set by combining the number of layers, the type of each layer (e.g., CNN, MLP, Attention, etc.), and the connection method (e.g., sequential, parallel, whether skip connections are included).

[0065] The second element is the search strategy, which determines how to select and experiment with structures within a defined search space. Major search strategies include evolutionary algorithms, reinforcement learning-based structure generation, Bayesian optimization, greedy or random search, and also include differentiable NAS techniques that represent structures in a continuous space and optimize them through gradient descent.

[0066] The third element is the Evaluation Strategy, where the performance of the discovered model structures is evaluated efficiently. Since training all structures is time-consuming and costly, methods are employed to reduce evaluation time, such as training only a subset, proxy tasks evaluating with small datasets, weight sharing where parameters are shared across multiple structures, and utilizing surrogate models to predict performance. In particular, for NAS designed for on-device environments, it is crucial to search while reflecting constraints (latency, memory, energy, etc.) inherent in CPU-based devices. To this end, latency-aware, memory-aware, and energy-aware NAS strategies capable of predicting execution speed and resource usage, as well as multi-objective optimization methods that consider both accuracy and computational efficiency, can be applied. Examples of NAS include reinforcement learning-based ENAS, gradient-based DARTS, MNasNet which reflects actual mobile latency, and FBNet, which uses a latency prediction model. The aforementioned NAS technology automatically identifies deep learning model structures that are more precise and efficient than traditional manual designs, and it is particularly useful for developing lightweight, high-performance models in resource-constrained environments.

[0067] Next, multi-modality context-aware algorithms incorporate structural optimization based on latency-pruned techniques. This method improves overall processing speed by enhancing computational efficiency and eliminating unnecessary calculations, focusing on computationally intensive components within the model that cause latency. Specifically, it reduces the overall computational load by shrinking the attention layer of discovered lightweight models or transforming them into lightweight structures, decreasing the size of embedding dimensions for word and sentence representations, and optimizing the dimensions of internal projection operations. This structural reduction contributes to lowering model memory usage and reducing execution latency. Furthermore, it is crucial to strike a balance between practicality and accuracy through a design that maintains the model's core language understanding capabilities while reducing computational complexity (FLOPs).

[0068] Next, the multimodal context recognition algorithm includes a token embedding structure for integrated input to expand and include text, images, and various sensor data. Designed to process multimodal information of different formats integrally within a single system, it is characterized by enabling more sophisticated decision-making by simultaneously interpreting diverse input forms and identifying correlations. It is structured to allow for flexible expansion without modifying the existing system, even if new data types such as voice, GPS, and biosignals are added in the future. More specifically, to effectively fuse multimodal data, a cross-modal attention-based algorithm is applied. By automatically learning the semantic connectivity and relative importance between different modalities, the model enables itself to identify and process interactions between various types of information, such as text and images, or sensors and video. Through this, the system can make more reliable predictions and judgments based on combinations of diverse information, rather than relying on a single data point. Meanwhile, the constructed multimodal integration structure and fusion algorithm undergo actual performance verification in an environment simulating complex real-world conditions. This initial performance evaluation focuses on analyzing the model's accuracy, processing efficiency, reliability, and robustness under various data conditions, thereby enabling an empirical determination of whether the technology is sufficiently applicable in the field.

[0070] Hereinafter, with reference to the drawings, an operation method of a generative AI system (10) for real-time services that combines an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to an embodiment of the present invention will be described. FIG. 10 is a flowchart of the operation method of a generative AI system (10) for real-time services that combines an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to the present invention. Referring to FIG. 10, the operation method of a generative AI system (10) for real-time services that combines an H-MoE-based high-performance lightweight language model executable in an on-device CPU environment according to the present invention may be composed of the following 7 steps.

[0071] Step 1 (S10) is a step in which multimodal data is collected, preprocessed, and stored. Here, multimodal data includes text data, image data, voice data, video data, and various sensor data. In Step 1 (S10), the collected multimodal data is preprocessed for the learning of artificial intelligence. Text data preprocessing includes text normalization, which is a process to maintain the consistency of text data; removal of special characters; tokenization, which separates text into token units; removal of stop words; and extraction of stems or lemmas. Additionally, image data preprocessing includes image resizing, image cropping, image transparency processing, image sharpening, image contrast enhancement, color separation, image binarization, background removal, boundary detection, and corner detection. Additionally, voice data preprocessing includes trimming, which treats sounds below 60dB as silent, and random padding, which sets a fixed size and randomly applies padding to the front and back, as Melspectrogram and MFCC have different lengths when voice data have different lengths. Furthermore, image data preprocessing includes data format conversion from color to grayscale, size conversion, and pixel value conversion. Additionally, sensor data preprocessing includes sorting numeric data according to data items. The preprocessed data is embedded and stored.

[0072] Step 2 (S20) is a step in which domain-filtered document data is stored after being sorted. In Step 2 (S20), in-depth expertise specific to a particular domain is stored, and first, document data is filtered to match the specific domain. In the present invention, the specific domain may include fields such as news, law, medical, Korean language, disaster safety, and smart healthcare, and the data format is limited to document data having text. In Step 2 (S20), document data filtered to match the specific domain is sorted, and document sorting is performed by calculating a Similarity Score. Similarity includes Mean Squared Difference Similarity, Cosine Similarity, and Pearson Similarity; in one embodiment of the present invention, Cosine Similarity may be used.

[0073] Step 3 (S30) is a step in which the data stored in Step 1 (S10) and Step 2 (S20) is learned. In Step 3 (S30), the data embedded in Step 1 (S10) and Step 2 (S2) is first learned, and the data embedded in Step 2 (S20) is re-learned after fine-tuning.

[0074] Step 4 (S40) is a stage where a search is performed using semantic search based on embeddings, based on the content learned in Step 3 (S30). Semantic search refers to a search that identifies the questioner's intent rather than searching based on keywords. Keyword-based search has the disadvantage of being difficult to consider similar meanings and difficult to identify the relationships between words. Semantic search produces results by matching concepts rather than keywords; it represents words as concepts through vector-dimensional embeddings and displays search results by finding content that matches the meaning of the words included in the user query. Through Step 4 (S40), the user inputs the desired question via an input device, and the content retrieved through semantic search can be verified via an output device.

[0075] Step 5 (S50) is a stage in which the user's emotional state is analyzed through natural language understanding of the user's language in Step 4 (S40). The artificial intelligence that has gone through Step 1 (S10) to Step 4 (S40) is in a state of understanding the user's language as natural language. Steps 1 (S10) to 4 (S40) describe general generative AI, while Step 5 (S50) to 7 (S70) describe generative AI specialized for the user's emotions. In Step 5 (S50), the user's emotional state is precisely recognized through the analysis of natural language semantic relationships in Steps 1 (S10) to 4 (S40). The LLM module (100) of the present invention converts the input sentence into an embedding and then adjusts the attention layer to generate a vector representation suitable for the domain. The attention layer-based emotion recognition technology has a structure that relies on class tokens for emotion classification, which limits the identification of semantic relationships within the sentence. In the fifth step (S50), natural language expression information embedded in the LLM module (100) is reconstructed into a single special token using natural language semantic relationship analysis technology, and the semantic relationship of the input sentence is understood and consistency is maintained through class token and weight adjustment to analyze the user's sentiment state.

[0076] Step 6 (S60) is a stage in which user emotions are analyzed through the interaction between spatial components of lighting, color, and sound in the space where the user is located. In lighting elements, low color temperatures induce comfort and stability, while high color temperatures induce concentration and alertness. In color elements, blue tones induce calmness, while red tones induce vitality or tension. In sound elements, low-pitched and slow-motion sounds induce relaxation, while high-pitched and fast-motion sounds induce stimulation or tension. In Step 6 (S60), lighting, color, and sound in the space where the user is located are detected through sensor data collected in Step 1 (S10). Lighting is measured via an illuminance sensor, color is measured via a color sensor, and sound is measured via an acoustic sensor. The measured sensor values ​​are preprocessed in Step 1 (S10), and in Step 6 (S60), the preprocessed data is analyzed to analyze the user's emotions. In lighting elements, color temperature is analyzed primarily; in color elements, color series are analyzed primarily; and in acoustic elements, pitch and speed are analyzed primarily.

[0077] Step 7 (S70) is a stage where spatial components suitable for the user are recommended through time-series analysis of changes in the user's emotional state. Since the user's emotional state is maintained or changes depending on the time point, time-series-based sentiment analysis is essential; in Step 7 (S70), the user's temporal emotional changes are inferred and recognized. Sentiment analysis at a specific point in time has limitations in recognizing emotional changes between time points and applying past time points; however, past context reconstruction technology reconstructs user sentiment from multiple time points into a time series, allowing the direction of emotional changes and past time points to be preserved. In Step 7 (S70), past context is reconstructed from the conversation records stored in Step 4 (S40); the user's sentiment is analyzed in a time series based on LLM context embeddings; and spatial components suitable for the user are recommended based on the analysis of spatial elements in the current space. For example, low color temperature and blue tones induce a sense of stability and calmness, making them suitable for a relaxation-centered space; thus, they can be recommended when the user needs rest. High color temperature and high-speed sound induce vitality and alertness, making them suitable for a workspace; thus, they can be recommended when the user is working. Red tones and high-pitched sounds induce anxiety and stress, making them unsuitable for work or rest areas; therefore, they are not recommended for use while the user is working.

[0079] The system described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the system and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0080] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0081] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0082] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0083] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0084] 10: Generative AI Systems 100 : LLM Module 110: Internal Database Section 120 : External Knowledge Database Section 130 : AI Learning Department 140 : Semantic Search Section 200: Dynamic Emotion Recognition Module 210 : User Analysis Department 220 : Spatial Element Analysis Section 230 : Spatial Element Recommendation Section

Claims

Claim 1 An LLM module equipped with an internal database unit based on multimodal data including text, sensor data, and image data, and a RAG-based external knowledge database unit, and performing semantic-based search by embedding; and a dynamic emotion recognition module in which the influence of elements constituting the space of lighting, color, and sound on user emotion is analyzed based on natural language understanding and sensor data of the LLM module; wherein the LLM module includes an internal database unit in which multimodal data is collected, preprocessed, and stored; an external knowledge database unit in which domain-filtered document data is stored after being sorted; an AI learning unit in which data stored in the internal database unit and the external knowledge database unit is learned; and a semantic search unit in which search is performed by meaning-based search using embeddings based on the content learned by the AI ​​learning unit; wherein the LLM module selects and uses one or more of bit-flexible quantization, knowledge distillation, and post-precision supplementation techniques as quantization technologies for lightweighting; wherein the external knowledge database unit further includes a similarity sorting unit, a common field determination unit, and a data search request unit; wherein the similarity sorting unit sorts multiple documents by determining similarity with a similarity score value having a similarity range of -1 to 1 for the corresponding field after the document data is converted into vector data and the document sorting unit calculates the cosine value between two vectors; and wherein the common field determination unit determines that the corresponding fields are mutual When defined as having other first and second fields, the number of documents corresponding to a common field that exceeds both the average value of similarity scores for the first field and the average value of similarity scores for the second field is determined, and the data search request unit performs additional data search through external search when the number of documents corresponding to the common field is less than a certain number, and the external knowledge database unit further comprises a time similarity weighting application unit.The time similarity weighting application unit applies weights to similarity score values ​​linked to the publication time of the paper when the document data is a paper, such that a higher weight is assigned as the publication time of the paper is closer to the time of similarity judgment; the external knowledge database unit further comprises a volume similarity weighting application unit, wherein the volume similarity weighting application unit applies weights to similarity score values ​​linked to the volume of the document, such that a higher weight is assigned as the volume of the document increases; and the dynamic sentiment recognition module includes a user analysis unit in which the user's sentiment state is analyzed through natural language understanding of the user's language, a spatial element analysis unit in which spatial characteristics are analyzed through the interaction between spatial components of lighting, color, and sound of the space where the user is located, and a spatial element recommendation unit in which spatial components suitable for the user are recommended through time-series analysis of changes in the user's sentiment state over a certain period by the user analysis unit and the spatial element analysis unit; the spatial element recommendation unit comprises a lighting recommendation unit in which a color temperature is recommended according to the user's sentiment state index, and the lighting recommendation unit recommends the recommended color temperature in conjunction with the sentiment state change gradient, which is the change in the sentiment state index over time, such that the lower the sentiment state change gradient, the lower A generative AI system for real-time services combining an H-MoE-based high-performance lightweight language model and RAG executable in an on-device CPU environment, characterized in that a color temperature value is recommended, the LLM module collects data specialized for Korean search to improve Korean information retrieval performance, fine-tunes the AI ​​learning unit exclusively for Korean, and retrains the AI ​​learning unit using a hard negative mining technique, and furthermore, the LLM module is characterized in that the artificial intelligence is trained through an ethics and sociality dataset for public service application. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete

Images