Keyword-based LLM Learning and Response Optimization and Performance Verification Management Device and Method Thereof

Abstract

The present invention relates to a keyword-based LLM learning response optimization and performance verification management device and method, which enhances learning by quantitatively reflecting the importance of keywords in the learning and response stages of LLM, re-ranks generated multiple responses based on semantic and statistical criteria, and generates responses of a consistent quality by repeatedly statistically verifying the performance on the same question set. First, the keyword-based LLM learning response optimization and performance verification management device of the present invention comprises: a learning data generation module that converts received raw data into text by natural language processing, divides it into keyword units, analyzes the divided keywords according to a pre-set first evaluation criterion to calculate an importance score, and generates LLM learning data including keywords whose calculated importance score is greater than or equal to a pre-set threshold; a learning response optimization module that receives the LLM learning data generated by the learning data generation module, analyzes the keywords, sets a learning weight for each analyzed keyword, and receives candidate responses generated by training a large-scale language model with the keywords to which the learning weight is applied; and an effectiveness verification management module that analyzes the candidate responses generated by the learning response optimization module based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, determines the quality of the candidate response as good if the calculated score is greater than a pre-set quality standard score, and determines the quality of the candidate response as poor if the calculated score is less than a pre-set quality standard score. In addition, the keyword-based LLM learning response optimization and performance verification management method comprises: (a) a step in which a learning data generation module converts received raw data into text by natural language processing, divides it into keyword units, analyzes the divided keywords according to a pre-set first evaluation criterion to calculate an importance score, and generates LLM learning data including keywords whose calculated importance score is greater than or equal to a pre-set threshold; (b) a step in which the LLM learning data generated by the learning data generation module is received, the keywords are analyzed, learning weights are set for each analyzed keyword, and a large-scale language model (LLM) is trained with the keywords to which the learning weights are applied to receive candidate responses generated; and (c) a step in which an effectiveness verification management module analyzes the candidate responses generated by the learning response optimization module based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, and determines the quality of the candidate response as good if the calculated score is greater than a pre-set quality standard score, and determines the quality of the candidate response as poor if the calculated score is less than a pre-set quality standard score.

Description

Technology Field

[0001] The present invention relates to artificial intelligence (AI) technology for optimizing the training and response generation of a large-scale language model (LLM). More specifically, the present invention relates to a keyword-based LLM training response optimization and performance verification management device and method, which automatically extracts and refines core keywords during a data preprocessing step, inserts them in a context-preserving form, re-ranks responses by assigning weights during training, and improves the consistency and accuracy of the model through iterative performance verification. Background Technology

[0002] Large Language Models (LLMs) are artificial intelligence (AI) technologies that learn from vast amounts of text data to perform natural language understanding and response generation, and are being utilized in various fields.

[0003] The response quality and consistency of LLMs used in various fields show significant differences. This is because the learning process of the LLM varies greatly depending on what data is learned and how.

[0004] Many existing LLM learning systems have utilized structures that generate responses by simply iteratively learning large-scale data or through probabilistic token prediction. While this approach tends to improve performance with increasing data volume, it has several technical limitations.

[0005] First, the learning method of existing LLM learning systems optimizes the probability distribution of tokens throughout the entire sentence and fails to explicitly reflect the relative importance of important words or key concepts (keywords) in a specific context.

[0006] As a result, the model develops a bias of prioritizing the learning of simple, high-frequency words over important keywords.

[0007] Furthermore, the existing LLM response generation process relies on probabilistic selection rules such as Top-k and Top-p sampling. While this approach is advantageous for ensuring diversity, it sometimes produces contextually invalid responses because it does not sufficiently consider the semantic intent or core keywords of the query. For instance, there is a problem with low model response consistency, such as responses being generated in different directions or responses omitting key concepts occurring repeatedly even when the same question is entered multiple times. Additionally, existing model performance evaluation frameworks are centered on sentence similarity-based metrics like BLEU and ROUGE. However, these metrics fail to reflect semantic accuracy or the degree of keyword reflection.

[0008] In addition, most verifications are performed as one-time evaluations and do not guarantee performance reproducibility between repeated experiments.

[0009] Therefore, even if LLM demonstrates excellent performance in a specific domain, the problem of instability under other conditions persists, and recent research also raises the need for keyword-based optimization. Prior art literature

[0010] Republic of Korea Registered Patent No. 10-2655929 'System and method for generating conversational text data using a generative AI model' (Publication date: April 11, 2024) The problem to be solved

[0011] The objective of the present invention, devised to solve the above-mentioned problems, is to provide a keyword-based LLM learning response optimization and performance verification management device and method that improves the efficiency of learning data by quantitatively calculating the importance of core keywords to reduce unnecessary word learning or semantic distortion.

[0012] Another objective of the present invention is to provide a keyword-based LLM learning response optimization and performance verification management device and method capable of selecting the optimal response that matches the user's query intent by sorting the ranking of candidate responses by combining keyword inclusion rate, semantic similarity, human evaluation score, improvement rate, etc. of generated multiple responses.

[0013] Another objective of the present invention is to provide a keyword-based LLM learning response optimization and performance verification management device and method that can ensure the reproducibility of LLM performance improvement results by evaluating the model's response quality in objective numerical terms and reducing performance deviations between experiments through repeated measurement of keyword inclusion rate, semantic similarity, human evaluation score, improvement rate, etc., for the same question.

[0014] Furthermore, the present invention provides a keyword-based LLM learning response optimization and performance verification management device and method that assigns keyword weights according to different domains such as law, finance, and medicine, generates responses suitable for the respective domains, and enables the generation of responses of consistent quality across all domains.

[0015] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0016] The keyword-based LLM learning response optimization and performance verification management device of the present invention for achieving the above-mentioned problem comprises: a learning data generation module that converts received raw data into text by natural language processing, divides it into keyword units, analyzes the divided keywords according to a pre-set first evaluation criterion to calculate an importance score, and generates LLM learning data including keywords whose calculated importance score is greater than or equal to a pre-set threshold; a learning response optimization module that receives the LLM learning data generated by the learning data generation module, analyzes the keywords, sets a learning weight for each analyzed keyword, and receives candidate responses generated by training a large-scale language model with the keywords to which the learning weight is applied; and an effectiveness verification management module that analyzes the candidate responses generated by the learning response optimization module based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, determines the quality of the candidate response as good if the calculated score is greater than a pre-set quality standard score, and determines the quality of the candidate response as poor if the calculated score is less than a pre-set quality standard score, wherein the learning data generation module includes various A data collection unit that receives raw data in a specific format and converts it into text; a keyword determination unit that, for candidate keywords extracted by the data collection unit, quantifies them according to pre-set evaluation criteria to calculate importance and determines keywords whose calculated importance is greater than or equal to a pre-set importance threshold as core keywords; a keyword insertion unit that, after receiving the core keywords determined by the keyword determination unit, divides the original text into phrases of each sentence constituting the original text, converts each phrase into an embedding vector, calculates semantic similarity between the embedding vector of the phrase and the embedding vector of the core keyword, determines the phrase with the highest similarity among the calculated semantic similarities as an insertion candidate location, and inserts the core keyword at the corresponding insertion candidate location; and a language model that calculates the difference between the awkwardness index value of the sentence before insertion of the core keyword and the awkwardness index of the sentence with the core keyword inserted,It includes a training data generation unit that generates training data from sentences whose difference value is less than or equal to a preset awkwardness threshold.

[0017] The learning response optimization module includes a weighted learning unit that receives an importance score from the learning data generation module and adjusts the weight of the corresponding keyword within a pre-configured learning mechanism to be proportional to the importance score.

[0018] The learning response optimization module may include a response re-ranking unit that receives multiple candidate responses from a large-scale language model, calculates a frequency-based importance score, a statistical association score, or a semantic similarity score for the candidate responses, calculates a response fit score by substituting the calculated score into a pre-set weighted response calculation formula, and then reorders the responses in order of score.

[0019] The response re-ranking unit can calculate response suitability scores for all candidate responses according to a pre-set weighted response calculation formula, and then select the candidate response with the highest score as the final response.

[0020] It may include a domain tuning unit that individually sets different learning weights and response reordering weights according to the data characteristics of each domain.

[0021] When new LLM training data is input from the training data generation module after LLM training, the continuous learning unit may be included to maintain the up-to-dateness of the LLM model by comparing it with a pre-configured keyword dictionary and selectively setting only the changed or added keywords as targets for retraining, thereby updating only the necessary parts of the entire raw data.

[0022] The effectiveness verification management module may include a performance evaluation unit that analyzes candidate responses based on standard responses to pre-set question data to calculate changes in keyword inclusion rates or semantic similarity scores, calculates a comprehensive score by substituting the calculated change values ​​into a pre-set response quality calculation formula, and statistically verifies that if the comprehensive score is equal to or greater than a pre-set quality standard score, the response quality is judged to be good, and if it is less than the quality standard score, it is judged to be poor.

[0023] The effectiveness verification management module may include a statistical processing unit that calculates the performance improvement rate of a large-scale language model by reflecting the score calculated before the large-scale language model is trained by the learning response optimization module and the score calculated after the large-scale language model is trained by the learning response optimization module into a pre-set improvement rate formula.

[0024] The effectiveness verification management module includes an iterative verification unit that calculates at least one of an average score or a standard deviation by iteratively analyzing responses generated by the learning response optimization module based on pre-set question data.

[0025] The effectiveness verification management module includes a visualization reporting unit that generates visualized information for candidate responses where the improvement range of the frequency-based importance score, statistical association score, or semantic similarity score is greater than or equal to a preset visual threshold, and a domain performance verification unit that verifies the universality of the model by applying candidate responses to different domain data.

[0026] delete

[0027] The performance evaluation unit can feed back candidate responses judged to have good response quality to the learning response optimization module.

[0028] Another method for managing keyword-based LLM learning response optimization and performance verification according to the present invention for achieving the aforementioned problem to be solved comprises: (a) a step in which a learning data generation module converts received raw data into text by natural language processing, divides it into keyword units, analyzes the divided keywords according to a first evaluation criterion set in advance to calculate an importance score, and generates LLM learning data including keywords whose calculated importance score is greater than or equal to a pre-set threshold; (b) a step in which the LLM learning data generated by the learning data generation module is received, analyzes the keywords, sets a learning weight for each analyzed keyword, and receives candidate responses generated by training a large-scale language model (LLM) with the keywords to which the learning weight is applied; and (c) a step in which an effectiveness verification management module analyzes the candidate responses generated by the learning response optimization module based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, and if the calculated score is greater than a pre-set quality standard score, the quality of the candidate response is judged as good, and if the calculated score is less than a pre-set quality standard score, the quality of the candidate response is judged as poor. The method comprises, wherein step (a) includes: a step in which a data collection unit of the training data generation module receives raw data of various formats and converts it into text; and a step in which a keyword determination unit of the training data generation module calculates the importance of candidate keywords extracted by the data collection unit by quantifying them according to pre-set evaluation criteria, and determines keywords whose calculated importance is greater than or equal to a pre-set importance threshold as core keywords.The keyword insertion unit of the above-described training data generation module receives a core keyword determined by the keyword determination unit, divides the original text into phrases of each sentence constituting the original text, converts each phrase into an embedding vector, calculates the semantic similarity between the embedding vector of the phrase and the embedding vector of the core keyword, determines the phrase having the highest similarity among the calculated semantic similarities as an insertion candidate location, and inserts the core keyword at the corresponding insertion candidate location; and the training data generation unit of the above-described training data generation module calculates the difference between the awkwardness index value of the sentence before the insertion of the core keyword and the awkwardness index of the sentence with the insertion of the core keyword using a language model, and generates a sentence as training data in which the difference value is less than or equal to a preset awkwardness threshold.

[0029] (b) Step (b) may include step (b-1) in which, after the learning response optimization module receives an importance score from the learning data generation module, the weight of the corresponding keyword is adjusted in proportion to the importance score within the pre-configured learning mechanism structure.

[0030] (b) Step (b) further includes step (b-2) in which the learning response optimization module receives multiple candidate responses from a large-scale language model, calculates a frequency-based importance score, a statistical association score, or a semantic similarity score for the candidate responses, calculates a response fit score by substituting the calculated scores into a pre-set weighted response calculation formula, and selects the candidate response with the highest score as the final response. Effects of the invention

[0031] The present invention provides a keyword-based LLM learning response optimization and performance verification management device and method that improves the efficiency of learning data by quantitatively reflecting the importance of core keywords to reduce unnecessary word learning or semantic distortion.

[0032] In addition, the present invention provides a keyword-based LLM learning response optimization and performance verification management device and method that enables the automatic selection of the response most suitable for the user's query intent by re-ranking generated multiple responses according to preset criteria.

[0033] In addition, the present invention provides a keyword-based LLM learning response optimization and performance verification management device and method that can objectively evaluate the response quality of a model by measuring keyword inclusion rate, semantic similarity, human evaluation score, improvement rate, etc., for the same set of queries, and reduce performance deviation between experiments to ensure the reproducibility of LLM performance improvement results.

[0034] In addition, the present invention provides a keyword-based LLM learning response optimization and performance verification management device and method that can optimize keyword weights and response criteria for different domains such as law, finance, and medical fields, thereby enabling the maintenance of responses of consistent quality in various fields.

[0035] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0036] FIG. 1 is a conceptual diagram of a keyword-based LLM learning response optimization and performance verification management device according to one embodiment of the present invention. FIG. 2 is a block diagram of a keyword-based LLM learning response optimization and performance verification management device according to an embodiment of the present invention. Figure 3 is a block diagram of the training data generation module of Figure 2. Figure 4 is a block diagram of the learning response optimization module of Figure 2. Figure 5 is a diagram showing the operation of the weight application learning unit of Figure 4. Figure 6 is a diagram showing the operation of the response reordering unit of Figure 4. Figure 7 is a block diagram of the effectiveness verification management module of Figure 2. Figure 8 is a diagram showing the operation of the learning data generation module of Figure 2. FIG. 9 is a flowchart of a keyword-based LLM learning response optimization and performance verification management method according to an embodiment of the present invention. Specific details for implementing the invention

[0037] The advantages and features of the present invention and the apparatus for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments described below but can be implemented in various different forms. The embodiments described below are provided merely to ensure that the description of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. The scope of the claims of the present invention may be defined solely by the claims.

[0038] First, prior to the specific description of the present invention, the terms used in this specification will be briefly explained, and the described embodiments will be described in detail.

[0039] The terms used in this specification are generally accepted terms, taking into account their functions in this description; however, they may vary depending on the intent of those skilled in the relevant field, case law, the emergence of new technologies, etc. Additionally, for terms arbitrarily selected by the applicant, their meanings are described in detail in the relevant description of the invention.

[0040] Therefore, the terms used in this description are defined not merely by their names, but based on their meanings and the overall content of this description.

[0041] Furthermore, singular expressions in this specification include plural expressions unless the context clearly indicates that they are singular. Additionally, when a part of the specification is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0042] Additionally, as used herein, the terms 'module' or 'part' refer to a software or hardware component, and the 'module' or 'part' performs the roles defined for each. Such a 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run one or more processors. As an example, the 'module' or 'part' may include at least one of the following: software components, such as object-oriented software components, class components, and task components; processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables.

[0043] The functions provided within the components and 'modules' or 'parts' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'. Furthermore, the Large Language Models (LLMs) described herein are defined as language models composed of artificial neural networks having numerous parameters. Such large language models enable AI technology that analyzes user queries and provides appropriate responses.

[0044] Hereinafter, a keyword-based LLM learning response optimization and performance verification management device according to an embodiment of the present invention will be described in general terms with reference to FIG. 1.

[0045] FIG. 1 is a conceptual diagram of a keyword-based LLM learning response optimization and performance verification management device according to one embodiment of the present invention.

[0046] A keyword-based LLM learning response optimization and performance verification management device (100) according to one embodiment of the present invention improves learning efficiency by quantitatively reflecting the importance of core keywords, thereby reducing unnecessary word learning or semantic distortion and improving the efficiency of learning data, as well as generating candidate responses, calculating the quality of the generated candidate responses, classifying the candidate responses based on the calculated value as either good or bad responses, and then automatically selecting only good responses.

[0047] In particular, the present invention ensures the reproducibility of LLM performance improvement results by evaluating the model's response quality using objective numerical values ​​and reducing performance deviations between experiments through repeated measurements of keyword inclusion rates, semantic similarity, human evaluation scores, and improvement rates for pre-set question data. Furthermore, the present invention enables the optimization of keyword weights and response criteria for different domains, such as law, finance, and medicine, thereby maintaining responses of consistent quality across various fields.

[0048] A keyword-based LLM learning response optimization and performance verification management device (100) according to one embodiment of the present invention includes a learning data generation module (110) that preprocesses data and generates learning data for LLM learning, a learning response optimization module (120) that receives a response generated from a large-scale language model, and an effectiveness verification management module (130) that verifies the response quality of the large-scale language model.

[0049] Hereinafter, the components constituting the present invention will be described in detail with reference to FIGS. 2 to 8.

[0050] FIG. 2 is a block diagram of a keyword-based LLM learning response optimization and performance verification management device according to an embodiment of the present invention, FIG. 3 is a block diagram of the learning data generation module of FIG. 2, and FIG. 4 is a block diagram of the learning response optimization module of FIG. 2. FIG. 5 is a diagram showing the operation of the weight application learning unit of FIG. 4, and FIG. 6 is a diagram showing the operation of the response re-ranking unit of FIG. 4. FIG. 7 is a block diagram of the effectiveness verification management module of FIG. 2. FIG. 8 is a diagram showing the operation of the learning data generation module of FIG. 2.

[0051] The training data generation module (110) converts raw data received from the outside into text by processing it into natural language, then divides it into words and assigns importance scores to the divided words to generate LLM training data. At this time, the training data generation module (110) calculates a frequency-based importance score (first importance score) based on a frequency-based statistical analysis model as an example of a first evaluation criterion, calculates a statistical value association score (second importance score) based on a statistical association analysis model that evaluates statistical significance, and calculates a semantic similarity score (third importance score) based on a semantic embedding-based artificial intelligence model. In this process, different weights are primarily applied to each of the frequency-based importance score (first importance score), the statistical value association score (second importance score), and the semantic similarity score (third importance score). At this time, each weight (λ) is predefined for each domain, for example, law w1=0.5, w2=0.3, w3=0.2, or is dynamically calculated through a grid search technique to maximize the final importance score, ensuring that the sum of the weights remains 1. Meanwhile, the training data generation module can increase w1 to reflect the frequency-based importance score's frequency-based characteristics in the law domain, and adjust w3 to serve as a semantic complement for semantic similarity.

[0052] At the same time, the training data generation module (110) calculates a Final Importance Score. At this time, the training data generation module (110) selects the keyword with the highest Final Importance Score as the core keyword. Then, the LLM training data set can be generated by inserting the core keyword into a sentence.

[0053] The learning data generation module (110) transmits the generated core keywords and final importance to the learning response optimization module (120). This information is subsequently applied to the weight calculation according to a learning mechanism pre-set in the learning response optimization module (120), such as the Transformer self-attention mechanism. The learning data generation module (110) can improve learning efficiency and response quality by automatically generating learning data centered on important keywords by structuring and quantifying raw data prior to LLM learning.

[0054] Through this, the training data generation module (110) selects training data that is highly consistent with the overall data distribution by using a formula based on statistical consistency in the data selection stage.

[0055] A learning data generation module (110) exhibiting such characteristics may include a data collection unit (111), a keyword determination unit (112), a keyword insertion unit (113), and a learning data generation unit (114), as illustrated in FIG. 3. Various components included in the learning data generation module (110) will be described later.

[0056] The learning response optimization module (120) receives LLM training data generated by the training data generation module (110) and assigns a learning weight to each keyword. At this time, the learning response optimization module (120) assigns a learning weight in proportion to the importance assigned to the keywords of the LLM training data. Then, it trains a large-scale language model with the LLM training data to which the learning weight has been assigned and receives a response generated from the large-scale language model.

[0057] As shown in FIG. 4, such a learning response optimization module (120) includes a weighted learning unit (121), a response re-ranking unit (122), a domain tuning unit (123), and a continuous learning unit (124). Here, as shown in FIG. 5, the weighted learning unit (121) uses the importance score of the word provided by the learning data generation module (110) to adjust the attention weight of the word in proportion to the importance score within Transformer self-attention, which is an example of a pre-set learning mechanism. Then, the adjusted attention weight is assigned to each word by adjusting it according to a pre-set weight formula. At this time, the pre-set weight formula may be a formula formed as: Adjusted attention weight = Existing attention weight x Importance-based weight. Here, the weight can be set as a weighting factor in the range of 1.2 to 1.5 in proportion to the word importance score calculated by the training data generation module (110). At this time, the weight is set within the range of 1.2 to 1.5 for each domain. For example, if the importance scores for the three words “AI”, “data”, and “model” are calculated as 0.8, 0.6, and 0.3, respectively, in the training data generation module (110), the weight application learning unit (121) calculates an adjusted attention weight by multiplying the existing attention weight of each word by a weight proportional to the importance score within the Transformer self-attention mechanism. At this time, if the existing attention weight of the word “AI” is 0.25, the weight 1.5 corresponding to the importance score of 0.8 is applied, and the adjusted attention weight is calculated as 0.25 x 1.5 = 0.375. If the existing attention weight for the keyword “data” is 0.20, a weight of 1.3 corresponding to an importance score of 0.6 is applied, resulting in an adjusted attention weight of 0.20 x 1.3 = 0.26. Additionally, if the existing attention weight for the word “model” is 0.15, a weight of 1 corresponding to an importance score of 0.3 is applied.The attention weight adjusted by applying 2 is calculated as 0.15 x 1.2 = 0.18. As such, words with higher importance scores are given a relatively larger weighting factor, which increases their attention weight; consequently, the model focuses more on key words like “AI” during the training process.

[0058] In addition, the weighted learning unit (121) assigns weights to core keywords by adjusting self-attention, thereby allowing more information to be carried in the core keywords, making the learning of core concept expressions faster and reducing the learning of unnecessary vocabulary. For example, it becomes a numerical value indicating how much influence the “AI” token has when interpreting the meaning of the keyword “regulation” in the sentence “AI regulation”. In the present invention, the higher the importance of a keyword, the more its weight is amplified when calculating the semantic relationship with other words, thereby enabling learning centered on core concepts.

[0059] Therefore, the weighted learning unit (121) can more accurately reflect contextually important keywords, thereby improving the learning efficiency and response accuracy of the large-scale language model.

[0060] The response reordering unit (122) quantitatively evaluates the quality of each response when the LLM generates multiple responses to a user's question and reorders them in order of score based on the score. More specifically, as illustrated in FIG. 6, the response reordering unit (122) receives multiple candidate responses from a large-scale language model, calculates a frequency-based importance score, a vector alignment score, or a semantic similarity score for the candidate responses, and then calculates a response suitability score by substituting the calculated score into a pre-set response quality calculation formula. For example, the pre-set weighted calculation formulas may be formulas such as Score=A1Keyword Inclusion+A2Semantic Sim+A3TF-IDF Alignment and A1+B2+C3=1. Based on such response quality calculation formulas, semantic consistency is maintained even with changes in candidate data generated by the learning response optimization module, and the core is not omitted. Here, Keywordinclusion is a frequency-based importance score calculated based on the proportion of core keywords appearing in candidate responses, SemanticSim is an embedding-based semantic similarity score, and TF-IDF Alignment is a keyword side vector alignment score between the question and candidate responses. Additionally, multiple response candidates generated by the LLM for a user's question are received, and the degree of keyword reflection and semantic appropriateness of each response are evaluated to calculate a response score through a pre-set response formula.

[0061] The response re-ranking unit (122) calculates the response suitability score for all candidate responses according to a pre-set weighted response calculation formula, and then selects the candidate response with the highest score as the final response.

[0062] The domain tuning unit (123) individually sets different keyword weights, insertion rules, and response re-ranking weights according to the data characteristics of each domain. For example, in the legal domain, high weights are assigned mainly to technical terms, in the financial domain, weights are assigned based on response priority centered on numbers and indicators, and in the technology domain, weights are adjusted and set based on product names and function names. Then, the adjusted parameters and response candidate selection results are provided to the effectiveness verification management module (130).

[0063] When new LLM learning data is received from the learning data generation module (110) after LLM learning, the continuous learning unit (124) compares it with a pre-set keyword dictionary and selectively sets only the changed or added keywords as re-learning targets.

[0064] For example, the continuous learning department (124) is as shown in Table 1 below.

[0065] A pre-configured domain dictionary formed by '{ "AI regulation”: {"λ”:1.3, "domain”:"law”}, "data privacy”: {"λ”:1.2, "domain”:"tech”}, "financial compliance”: {"λ”:1.4, "domain”: "finance”}}' and ' ["AI regulation”, "data privacy”, "blockchain policy”]'

[0066] By comparing keywords extracted from the new LLM training data, only the keyword that exists only in the new LLM training data, namely “blockchain policy,” is selected as the target for retraining. Then, the continuous learning unit (124) updates only the necessary parts of the entire new LLM training data to maintain the up-to-dateness of the LLM model.

[0067] The continuous learning unit (124) reduces the total relearning time by reflecting the value calculated from the new keyword set and the existing keyword set when new data is introduced, in terms of the amount of keyword change.

[0068] The learning response optimization module (120) defines a process for adjusting learning weights so that the LLM does not learn all words equally, but rather “learns words with greater focus as they become more important.” Through this, the learning efficiency and response accuracy of the LLM are improved simultaneously.

[0069] The effectiveness verification management module (130) analyzes candidate responses based on standard responses to pre-set question data, and calculates performance indicators by comparing and analyzing candidate responses generated by the learning response optimization module (120). Here, the pre-set question data is used as a standard to compare changes in responses by repeatedly inputting the same questions whenever the model is newly trained or strengthened in the learning response optimization module (120). Simply put, it serves as test data to reproducibly compare performance. Here, the pre-set question data can be, for example, Q1: “What is AI regulation?” Q2: “Explain what data privacy means?”, and the pre-training response, i.e., the standard response, can be “AI regulation is a topic about AI.” for Q1 and “Data privacy means to protect data.” for Q2. Additionally, the candidate response for Q1 generated by the learning response optimization module (120) is “AI regulation refers to government policies managing AI technologies.” and the candidate response for Q2 is “Data privacy refers to the process of ensuring personal data is collected, stored, and used responsibly.”

[0070] The learning response optimization module (120) calculates a change in keyword inclusion rate, a change in semantic similarity score, a change in response length, or a change in human reliability, and calculates a comprehensive score by substituting the calculated change values ​​into a pre-set response quality calculation formula. The effectiveness verification management module (130) statistically processes the calculated indicators to verify the response quality of the LLM and feeds the verification results back to the learning response optimization module (120).

[0071] In addition, the effectiveness verification management module (130) determines that the response quality is good if the calculated score or the total score of the calculated scores is equal to or greater than a preset quality standard score, and poor if it is less than the quality standard score, performs multiple repeated verifications on the test sentences, and calculates the average value, standard deviation, and improvement rate of each repeated result so that a response of consistent quality is output. As illustrated in FIG. 7, such an effectiveness verification management module (130) includes an indicator calculation unit (131), a statistical processing unit (132), a repeated verification unit (133), a visualization reporting unit (134), and a domain performance verification unit (135).

[0072] The indicator calculation unit (131) can calculate indicators such as the Keyword Inclusion Rate, Semantic Similarity, and Human Rating to quantify the degree of keyword reflection and semantic appropriateness within the response. Here, the keyword inclusion rate is calculated using the formula '(responses including characters / total responses) × 100'. This keyword inclusion rate indicates the degree to which the keywords learned by the model are reflected in the actual response, and the higher the value, the better the intended topic-centered response is generated.

[0073] Semantic similarity is calculated as cosine(response vector, ground truth vector). This semantic similarity measures how semantically close the model's response is to the ground truth or expected sentence. Furthermore, semantic similarity can serve as an indicator for evaluating contextual and semantic accuracy, rather than simply checking for keyword match.

[0074] The Human Rating is a score evaluated by humans. This Human Rating is the average value of scores (1–5) directly assessed by people. This score complements the naturalness and contextual validity of sentences that are not captured by automatically calculated scores, such as keyword inclusion rates and semantic similarity.

[0075] The statistical processing unit (132) calculates the performance improvement rate of the large-scale language model by reflecting the score (first performance indicator value) calculated before the large-scale language model is trained by the learning response optimization module (120) and the score (second performance indicator value) calculated after the large-scale language model is trained by the learning response optimization module (120) into a pre-set improvement rate formula. For example, assuming that the response accuracy of the large-scale language model before it is trained by the learning response optimization module (120) is 70%, and that the response accuracy improved to 84% as a result of re-evaluating the model after applying the learning response optimization module (120) using the same test dataset, the statistical processing unit (132) calculates an improvement rate of about 20% by applying the formula ((8470) / 70)×100 to this performance change. In this context, the improvement rate does not refer to a simple comparison of absolute values ​​indicating that "the score has increased," but rather signifies the relative extent of improvement by expressing the degree of enhancement as a ratio.

[0076] As another example, if the semantic similarity of the model was 0.62 before training but improved to 0.78 after training, the statistical processing unit (132) calculates this as ((0.78 0.62) / 0.62) × 100 to obtain an improvement rate of approximately 25.8%. This numerically indicates that the model has gone beyond simply including more keywords and has produced responses that are contextually more natural and have a higher semantic similarity.

[0077] In this way, the statistical processing unit (132) can quantitatively verify the degree of performance improvement of a large-scale language model by calculating the improvement range as a percentage (%) instead of simply comparing the difference in performance before and after learning. Through this, it can be proven with objective figures that the learning response optimization module (120) actually has the effect of improving the response quality of the model.

[0078] In addition, the statistical processing unit (132) is configured to perform a statistical test procedure on multiple test samples to verify whether the effect of applying the learning response optimization module (120) is statistically significant, in addition to a simple comparison of improvement rates. For example, the statistical processing unit (132) is configured to quantitatively determine whether the improvement in the model's performance is a statistically significant improvement by performing a significance evaluation based on a Paired t-test, a significance evaluation based on a Wilcoxon signed-rank test, and a stability evaluation based on a confidence interval.

[0079] Here, the significance evaluation based on the Paired t-test verifies whether the difference between the performance indicator values ​​before and after model training on the same dataset is a change caused by chance. Accordingly, a t-statistic is calculated for the mean difference between the performance indicator values ​​before and after training. Then, the statistical significance of the performance improvement is evaluated by determining whether the calculated p-value is less than a pre-set significance level.

[0080] The Wilcoxon signed-rank test-based significance evaluation is configured to assess the significance of performance improvement in a non-parametric manner by calculating a rank-based signed sum of two performance indicator values ​​to verify the difference in performance between pre- and post-training, even when the distribution of the performance indicators does not satisfy normality, and determining whether the p-value of the calculated statistic is below a pre-set significance level.

[0081] The confidence interval-based stability evaluation is configured to verify that the magnitude of improvement in performance indicators is statistically stable by calculating a confidence interval for the difference in performance before training and the difference in performance after training, and by checking whether the calculated confidence interval does not include 0, thereby evaluating whether the improvement in the model's performance is a statistically significant and consistent change.

[0082] The iterative verification unit (133) calculates the average value, standard deviation, and improvement rate by repeatedly comparing and analyzing the responses generated by the learning response optimization module (120) N times based on the pre-set question data. That is, the iterative verification unit (133) does not evaluate a single test data only once, but verifies the stability and reliability of performance by calculating the average and standard deviation by repeatedly testing the responses generated by the model for the same question data multiple times. For example, it is assumed that a large-scale language model repeatedly generated responses five times by the learning response optimization module (120) based on the same question data, “What is AI regulation?”. If the performance indicator values ​​(e.g., semantic similarity scores) of the responses calculated by the statistical processing unit (132) in each repeated experiment appear as 0.82, 0.84, 0.81, 0.85, and 0.83, respectively, the iterative verification unit (133) first calculates the average by adding all five of these values ​​and dividing by 5. That is, the average performance indicator value is calculated as (0.82 + 0.84 + 0.81 + 0.85 + 0.83) ÷ 5 = 0.83. Next, the replication verification unit (133) calculates how far each experimental value is from the mean to obtain the degree of dispersion of the results, i.e., the standard deviation. At this time, the standard deviation is calculated using the formula √(Σ(average of each value)² ÷ 5), and when applied, √((0.820.83)² + (0.840.83)² + (0.810.83)² + (0.850.83)² + (0.830.83)²) ÷ 5) = 0.014. This result is an average of 0.83 and a standard deviation of 0.014, which are the results of five evaluations performed by the iterative verification unit (133), and these values ​​indicate that stable performance is being shown. That is, the iterative verification unit (133) shows that the response quality generated by the learning response optimization module (120) is not a one-time improvement but is consistently maintained through iterative verification.

[0083] The visualization reporting unit (134) visually highlights only the items where the improvement range of each calculated indicator is above a preset visual threshold and generates a report by visualizing performance changes in the form of a bar graph, line chart, or heat map. More specifically, if the keyword inclusion rate increases from 68% to 83%, the semantic similarity increases from 0.76 to 0.80, and the Human Rating increases from 3.9 to 4.5 in the results calculated through the statistical processing unit (132) and the iteration verification unit (133), the visualization reporting unit (134) analyzes the improvement range for each performance indicator and highlights only the items that exceed a preset visual threshold, for example, an improvement rate of 10% or more. That is, since the keyword inclusion rate increases by 15% and the human reliability score increases by 15.4%, exceeding the threshold, the visualization reporting unit (134) highlights the corresponding keyword with a bold color or highlighting, and keywords that increase by 6.6%, such as semantic similarity, and are below the threshold, do not display the highlighting or display them with a neutral color or translucent processing. In addition, based on this data, the visualization reporting unit (134) visualizes the improvement range for each indicator in the form of a bar chart and displays the trend change before and after learning using a line chart. For example, by placing indicator names (keyword inclusion rate, semantic similarity, reliability) on the horizontal axis and the scores of each indicator on the vertical axis, and displaying the bars before and after learning side by side, it is possible to identify at a glance which item had the greatest improvement effect. Furthermore, for various domains, such as law, finance, and technology, the degree of improvement for each performance indicator can be expressed in the form of a heatmap. For example, by visualizing values ​​such as keyword inclusion rate +22% and semantic similarity +6% in the legal field, and keyword inclusion rate +20% and semantic similarity +8% in the financial field by distinguishing them by color intensity, it is possible to show which part of the learning response optimization module (120) of the present invention shows the greatest effect for each domain.And visualization results can be automatically saved in PDF, HTML dashboard, and Excel report formats.

[0084] The visualization reporting unit (134) does not stop at presenting statistically calculated figures as simple numerical data, but automatically highlights items that have improved beyond a threshold and visualizes them in the form of graphs and heatmaps, thereby allowing the user to intuitively check the performance improvement trends of the large-scale language model.

[0085] The domain performance verification unit (135) applies the same procedure to different domain data, such as laws, technical manuals, and financial news, to prove universality. This domain performance verification unit (135) is a sub-component of the effectiveness verification management module (130) and performs the function of verifying the response quality of a large-scale language model (LLM) by subdividing it by domain. The domain performance verification unit (135) may be composed of a domain classifier, a domain indicator calculator, a domain comparison analyzer, etc.

[0086] The domain performance verification unit (135) analyzes meta-information included in the response data, such as document source, subject, and tags, and automatically classifies the domain to which each data belongs. The classification result is then transmitted to the next stage domain indicator calculator.

[0087] The domain performance verification unit (135) analyzes the average value and standard deviation for each domain to verify whether the LLM maintains consistent quality without being excessively biased in a specific domain. If the performance deviation is below a threshold, it can be confirmed that the LLM model has been universally trained.

[0088] The present invention enables the learning data generation module (110), the learning response optimization module (120), and the effectiveness verification management module (130) to reduce unnecessary word learning or semantic distortion, improve the efficiency of learning data, and automatically select the response most suitable for the user's question intent. Furthermore, it enables the generation of answers of consistent quality for the same question. Moreover, by optimizing keyword weights and response criteria for different domains, the quality of the answers is made consistent across various fields.

[0089] Hereinafter, with reference to FIG. 8, the data collection unit (111), keyword determination unit (112), keyword insertion unit (113), and training data generation unit (114) constituting the training data generation module will be described in detail.

[0090] Figure 8 is a diagram showing the operation of the learning data generation module of Figure 2.

[0091] The data collection unit (111) receives raw data of various formats, converts it into text through natural language processing, divides the text into sentences using a Sentence Transformer-based Sentence Boundary Detection model, and performs keyword analysis on the divided sentences. At this time, the data collection unit (111) analyzes the frequency of keywords or whether they match keywords in a pre-set domain dictionary, and evaluates the analysis results according to pre-set criteria. At this time, keywords whose frequency is greater than or equal to a pre-set minimum frequency or that match keywords registered in the domain dictionary are extracted as candidate keywords. That is, the data collection unit (111) extracts candidate keywords from the raw data.

[0092] More specifically, the data collection unit (111) is a starting module for pre-processing data and includes a natural language processor (1111) and a discriminator (1112) as illustrated in FIG. 3. More specifically, the natural language processor (1111) can collect various types of data such as PDF, Word (docx), HTML, TXT, and images (JPEG, PNG). For example, the natural language processor (1111) can parse the collected PDF data to extract text and, through a Transformer-based sentence boundary recognition model, correctly divide judgments into sentence units without semantic interruption, without mistaking the abbreviation, i.e., AI, as the sentence end in a sentence such as “Artificial Intelligence (AI) is widely used in healthcare. It improves diagnostic accuracy and treatment efficiency.”. At this time, when the natural language processor (1111) collects image data, it can convert characters within the image into text using OCR (Optical Character Recognition). In addition, text from web documents can be collected in real time using a crawler.

[0093] The discriminator (1112) tags part-of-speech information for each divided sentence and extracts candidate keywords based on nouns, proper nouns, and domain technical terms among the tagged parts of speech, such as 'Transformer', 'learning', and 'data'.

[0094] The data collection unit (111) can extract candidate keywords that have a frequency of occurrence of the keyword greater than or equal to a preset minimum frequency and match professional keywords registered in the domain dictionary. For example, if expressions such as 'Transformer', 'learning', 'LLM', and 'data' satisfy the above conditions, they are extracted as candidate keywords. For example, the data collection unit (111) can extract candidate keywords such as those in Table 2.

[0095] Candidate Keywords Frequency of appearance Domain pre-matching Transformer 5th time is available Learning Episode 7 is available data Episode 6 is available insertion 2nd time doesn't exist Preprocessing 3 times doesn't exist

[0096] At this time, the frequency of occurrence shown in Table 2 is just one example. The frequency of occurrence may vary depending on the actual operating environment. In addition, the data collection unit (111) generates keyword-centered sentence data containing core keywords by excluding unnecessary interjections such as “oh” and “wow,” particles such as “and” and “but,” and conjunctions. That is, the data collection unit (111) performs text conversion, sentence segmentation, and part-of-speech tagging, automatically excludes unnecessary interjections, particles, and conjunctions, and can increase the information density of the training data that the LLM model learns.

[0097] The keyword determination unit (112) enables the extraction of core keywords more precisely than simple frequency-based methods by considering both the statistical importance and semantic association of the keywords. More specifically, the keyword determination unit (112) calculates the importance of the keywords by quantifying the importance of the keywords according to the pre-set first evaluation criteria, such as frequency-based evaluation criteria, statistical association evaluation criteria, and semantic similarity score evaluation criteria, for candidate keywords extracted from the data collection unit (111). Then, keywords whose calculated importance is greater than or equal to a pre-set importance threshold are determined as core keywords. In addition, the keyword determination unit (112) may calculate a final importance score by assigning weights to the calculated importance, and may determine keywords whose calculated final importance score is greater than or equal to a pre-set importance threshold as core keywords.

[0098] The keyword determination unit (112) is a core module for preprocessing data and, as illustrated in FIG. 3, includes an evaluation criterion calculator (1121), a core keyword selector (1122), and a keyword refiner (1123). Here, the evaluation criterion calculator (1121) includes a frequency (Term Frequency - Inverse Document Frequency) based statistical processing model that calculates the importance of keywords by statistically analyzing the frequency of appearance of keywords in each sentence and the relative appearance ratio in the entire document set to quantify the distribution of keyword appearance into a vector, and a statistical association analysis module that evaluates the significance of keywords by verifying the statistical association between keywords and specific topics or categories. Here, the statistical association analysis module may be, for example, a chi-square statistical analysis model. Here, the frequency-based statistical analysis model of the evaluation standard calculator (1121) can calculate the frequency-based importance (TF-IDF: Term Frequency - Inverse Document Frequency) by calculating a score by dividing the frequency of a keyword within a specific document by the total number of keywords based on the evaluation standard method of frequency-based importance (TF-IDF: Term Frequency - Inverse Document Frequency), and calculate the frequency-based importance by multiplying the two scores by calculating log(N / (1+df)) by dividing the total number of documents (N) by the number of documents (df) in which the keyword appears. That is, Frequency-based importance can be calculated using a formula.

[0099] In other words, the evaluation criteria calculator (1121) calculates a frequency-based importance score by assigning a high score to a keyword when the keyword appears frequently within a document but rarely appears in the entire dataset through a method of evaluating the importance of the keyword. For example, if the entire dataset consists of 10,000 training documents related to “Artificial Intelligence (AI),” “Large-scale Language Model (LLM),” “Data Preprocessing,” “Keyword Extraction,” etc., and one of the keywords “Transformer” appears 20 times and the total number of keywords in the documents is 2,000, the evaluation criteria calculator (1121) can calculate the frequency score (Term Frequency) of the keyword as 20 / 2,000 = 0.01.

[0100] In addition, if there are 200 documents containing “Transformer” out of a total of 10,000 documents, the evaluation criteria calculator (1121) can calculate the inverse document frequency score, which is the rarity score (Inverse Document Frequency) among all documents, as log(10,000 / (1 + 200)) = 1.70.

[0101] The evaluation criteria calculator (1121) can calculate the importance of the keyword, i.e., keyword importance, as 0.01 × 1.70 = 0.017 by combining the frequency score of the keyword calculated in this way and the inverse document frequency score, and this importance can be the first importance score indicating that “Transformer” has a relatively important meaning within the document. On the other hand, if the keyword “data” is included 9,800 times in almost all documents, the score calculated as log(10,000 / (1 + 9,800)) ≈ 0.0099 is very low, so it can be classified as a keyword with low importance.

[0102] Accordingly, the evaluation criteria calculator (1121) can determine domain-specific keywords such as “Transformer,” “keyword insertion,” and “training data optimization” as core candidate keywords through such a statistical evaluation process, and classify keywords such as “data” and “model” as non-core keywords.

[0103] Additionally, the evaluation standard calculator (1121) calculates a chi-square statistical score according to the evaluation standard method of a chi-square statistical analysis model, as an example of an evaluation standard method that calculates a statistical score using a pre-set statistical analysis formula to evaluate the statistical independence or correlation between each keyword and a specific topic as illustrated in (B) of FIG. 8. That is, the evaluation standard calculator (1121) uses a chi-square statistical formula to verify the independence between each keyword and a specific topic. You can calculate the chi-square statistical score by applying the formula.

[0104] The evaluation criteria calculator (1121) can calculate a high score when there is a high correlation between the keyword and the topic, and a low score when there is a low correlation between the keyword and the topic. For example, the evaluation criteria calculator (1121) assumes that there are 100 documents corresponding to the topic “LLM training data generation,” and that there are a total of 80 documents containing the keyword “Transformer,” of which 60 belong to the topic “LLM training data generation.” In order to verify whether the actual occurrence rate of the keyword “Transformer” is significantly higher than the expected occurrence probability, the evaluation criteria calculator (1121) can calculate a chi-square statistical score calculated as (observed score - expected score)² / sum of expected scores. At this time, the calculated chi-square statistical score can be the second importance score of the keyword.

[0105] Therefore, a high chi-square score can be calculated for keywords that appear frequently in topic documents but rarely in other documents, such as “Transformer” or “keyword insertion,” while a low chi-square score can be calculated for general keywords that appear commonly in most documents, such as “data” or “model,” where the difference in frequency of appearance between topic documents and non-topic documents is small. Additionally, the evaluation criteria calculator (1121) embeds keyword units into vectors as shown in (C) of FIG. 4 to calculate the semantic similarity score between each keyword and a sentence within a document, i.e., the importance of the keyword. For example, the evaluation criteria calculator (1121) uses a semantic-based method (SBERT similarity) of a semantic embedding artificial intelligence model, such as cosine similarity. The similarity score, or the third importance score of the keyword, is calculated using a formula.

[0106] Such an evaluation criterion calculator (1121) uses a meaning-based method to generate candidate keywords (h kw ) and sentence(h sentAfter embedding the ) into a vector, the importance of the keyword is calculated by calculating the cosine similarity between the two vectors. For example, the evaluation criteria calculator (1121) receives the candidate keyword “Transformer” extracted by the data collection unit (111) for the topic “LLM training data generation,” and then inputs the phrase “In this study, a large-scale language model was trained using a Transformer structure” into a semantic embedding artificial intelligence model to vectorize it. At this time, the sentence can be divided into “In this study,” “Transformer structure,” “large-scale language model,” and “trained.”, and the evaluation criteria calculator (1121) can calculate the importance score of the keyword by calculating the cosine similarity between each phrase vector and the vector of the core keyword “Transformer.” For example, the importance of keywords can be calculated with a similarity score of 0.15 with “in this study,” 0.93 with “Transformer structure,” 0.83 with “large-scale language model,” and 0.32 with “learned.”

[0107] The evaluation criteria calculator (1121) calculates a Final Importance Score by applying different weights to each of the calculated frequency-based importance score (first importance score), statistical value association score (second importance score), and semantic similarity score (third importance score) and combining them. For example, when the frequency-based importance score (first importance score) is calculated as 0.25, the statistical value association score (second importance score) as 0.032, and the semantic similarity score (third importance score) as 0.93, the evaluation criteria calculator (1121) can calculate a Final Importance Score of (0.4×0.25)+(0.3×0.032)+(0.3×0.93)=0.4396. At this time, each weight (w1, w2, w3) is predefined for each domain (e.g., law w1=0.5, w2=0.3, w3=0.2) or is dynamically calculated through a grid search technique for the purpose of maximizing the final importance score, and the sum of the weights is maintained at 1. At the same time, the evaluation criteria calculator (1121) can increase w1 to reflect the frequency-based importance score frequency-focused characteristic in the law domain and adjust w3 to serve as a semantic complementary role for semantic similarity.

[0108] The core keyword selector (1122) determines a keyword as a core keyword if the final importance score provided by the evaluation criteria calculator (1121) is greater than or equal to a preset importance threshold. For example, if the case where the final importance score is 0.35 or higher is set as greater than or equal to the importance threshold, “Transformer” can be determined as a core keyword. More specifically, the core keyword selector (1122) receives the frequency-based importance score (first importance score), the statistical score calculated from the statistical association analysis model (second importance score), and the sentence embedding-based semantic similarity score provided by the evaluation criteria calculator (1121), applies different weights to each score, and determines a keyword as a core keyword if the calculated score (third importance score) is greater than or equal to a preset threshold. Here, the pre-set importance threshold for the base importance score calculated from the frequency-based statistical analysis model is set to 0.05 or higher and less than 0.3, the pre-set importance threshold for the statistical score calculated from the statistical association analysis model, i.e., the chi-square statistical score, is set to 0.01 or higher and less than 0.05, and the threshold for the semantic similarity score calculated from the semantic embedding-based artificial intelligence model can be set within the range of 0.7 or higher and less than 0.9.

[0109] When such a core keyword selector (1122) receives a frequency-based importance score of 0.25, a chi-square statistical score of 0.032, and a cosine similarity score of 0.88 for the keyword “Transformer” from an evaluation standard calculator (1121), it can calculate a final importance score by assigning weights W1, W2, and W3 of 0.4, 0.3, and 0.3, respectively, to each score to reflect domain-specific importance. Accordingly, the final importance score of “Transformer” can be calculated as (0.4×0.25)+(0.3×0.032)+(0.3×0.88)= 0.3736. At this time, if the calculated final importance score is greater than or equal to a preset threshold, for example 0.3, the core keyword selector (1122) can determine the keyword as a core keyword.

[0110] On the other hand, in the case of the keyword “sentiment analysis,” the frequency-based importance score (first importance score) is calculated as 0.15, the statistical score (second importance score) calculated from the statistical association analysis model is calculated as 0.012, and the sentence embedding-based semantic similarity score (second importance score) is calculated as 0.7, so the total score is calculated as (0.4×0.15)+(0.3×0.012)+(0.3×0.7)=0.2736. In this case, the core keyword selector (1122) compares the total score calculated in this way with the threshold value, and if the final score is less than the threshold value, it is excluded from the core keywords. Additionally, the core keyword selector (1122) may adopt keywords whose calculated similarity score is greater than or equal to the threshold value as core keywords. For example, if the semantic similarity score is 0.75 or higher (sim ≥ 0.75), the keyword can be judged to be a keyword that is semantically close to the sentence.

[0111] The keyword refiner (1123) receives core keywords provided by the core keyword selector (1122), calculates the embedding vector of each core keyword using a semantic embedding-based artificial intelligence model, and calculates the cosine similarity between the embedding vectors. At this time, the keyword refiner (1123) classifies keywords whose calculated semantic similarity score is greater than or equal to a preset similarity threshold (e.g., 0.85) into the same group, and generates a refined keyword set with duplicates removed by keeping only one representative keyword among the keywords belonging to the same group and integrating and removing (excluding) the remaining keywords. For example, if the core keyword set determined by the core keyword selector (1122) consists of {“Transformer”, “Transformers”, “LLM”, “Large-scale Language Model”, “Artificial Intelligence”, “AI”, “Data Preprocessing”, “Data Processing”}, the keyword refiner (1123) applies morphological similarity analysis and string normalization algorithms to integrate multiple forms of identical keywords, e.g., “Transformer”, “Transformers”, into a single base form keyword. Then, through semantic embedding-based similarity comparison, semantically similar keywords, e.g., “LLM”, “Large-scale Language Model”, “Artificial Intelligence”, “AI”, “Data Preprocessing”, “Data Processing”, are each grouped into a single cluster.

[0112] Additionally, when the keyword refiner (1123) calculates that the cosine similarity score between “LLM” and “Large-scale Language Model” is 0.92, the similarity score between “AI” and “Artificial Intelligence” is 0.95, and the similarity score between “Data Preprocessing” and “Data Processing” is 0.89, it determines these as groups representing the same concept, keeps one representative keyword, and deletes the rest. For example, it refines “LLM”, “AI”, and “Data Processing” into representative keywords and outputs them. In this way, the keyword refiner (1123) can finally generate a refined set of core keywords such as {“LLM”, “AI”, “Data Processing”} by integrating duplicate or similar keywords and semantically clustering the set of core keywords within the document.

[0113] More specifically, the keyword refiner (1123) may include a group classification unit (1231), a representative keyword determination unit (1232), a category classification unit (1233), and a core keyword generation unit (1234) as components, as illustrated in FIG. 3. Here, the group classification unit (1231) groups keywords whose similarity is greater than or equal to a preset identical classification threshold into the same group. That is, each keyword is converted into an SBERT embedding vector, and similar keywords are clustered using one or more of cosine similarity, Euclidean distance, and Jaccard similarity. For example, the group classification unit (1231) groups similar keywords such as “AI regulation”, “artificial intelligence regulation”, and “AI law” into the same group, where the cosine similarity score is 0.9 or higher. Then, the representative keyword determination unit (1232) selects the keyword with the highest weighted combined score of the TF-IDF score, chi-square statistical score, and semantic similarity score within each group as the representative keyword. At this time, the representative keyword determination unit (1232) removes unnecessarily long terms or redundant expressions during this process. The category classification unit (1233) classifies the representative keyword selected by the representative keyword determination unit (1232) into one of the semantic categories of Topic, Entity, or Emotion. For example, the category classification unit (1233) classifies “AI regulation”, “Data privacy”, etc. as Topic, “EU”, “Samsung Electronics”, etc. as Entity, and “Concern”, “Expectation”, etc. as Emotion.

[0114] The core keyword generation unit (1234) refines keywords with overlapping meanings within each category and outputs a final core keyword set. The keyword refiner (1123) includes a group classification unit (1231), a representative keyword determination unit (1232), a category classification unit (1233), and a core keyword generation unit (1234) to reduce the duplication of keywords with the same meaning and to facilitate the application of semantic weights in the subsequent training data generation stage through a categorized structure. The keyword determination unit (112) can produce a core keyword set including importance and weights as shown in Table 3 below.

[0115] { "keyword”: "artificial intelligence”,"tfidf_score”: 0.82,"chi2_score”: 5.33,"similarity”: 0.91,"category”: "TOPIC”, "weight”: 1.2, "final_score": 0.88}

[0116] The keyword determination unit (112) performs similarity-based clustering, selects representative keywords, assigns categories, and outputs refined core keywords with duplicates removed. Through this, duplicates and synonyms are organized, and the accuracy and stability of applying weights and inserting core keywords are improved. Furthermore, the keyword refiner (1123) can calculate the similarity between each keyword vector using various calculation formulas, such as a cosine similarity formula, a Euclidean distance formula, or a Jaccard similarity formula. The keyword insertion unit (113) receives the core keywords determined by the keyword determination unit (112), calculates a semantic similarity score between the embedding vector of each phrase within the sentence in the original text and the embedding vector of the core keyword, determines the phrase location with the highest calculated semantic similarity score or a similarity threshold above a preset similarity threshold as an insertion candidate location, and inserts the core keyword into the corresponding insertion candidate location. At this time, the keyword insertion part (113) inserts key keywords without damaging the meaning of the original text, maintains the naturalness of the original text, and prevents data bias caused by excessive repetition.

[0117] The keyword insertion unit (113) includes an insertion position finder (1131), a context-preserving inserter (1132), a multiple insertion interval controller (1133), an insertion sentence verifier (1134), and a bias prevention controller (1135).

[0118] The insertion location finder (1131) breaks down each sentence into morpheme units, separates them into semantic units such as noun phrases, verb phrases, and adjective phrases, i.e., divisions, and calculates an embedding vector for each division. It calculates a semantic similarity score between the calculated embedding vector and the core keyword vector, and selects the phrase with the highest similarity as the insertion candidate location.

[0119] The context-preserving inserter (1132) modifies a sentence by applying a pre-configured large-scale language model (LLM)-based sentence reconstruction technique to insert a core keyword at a candidate location selected by the insertion location searcher (1131). At this time, the context-preserving inserter (1132) calculates the semantic similarity score of the sentence embeddings before and after the core keyword insertion and confirms the insertion only if it is greater than or equal to the context-preserving threshold. For example, the context-preserving inserter (1132) can modify the sentence “In this study, data preprocessing was performed.” by inserting the core keyword “LLM” at the insertion candidate location (after ‘data’) found by the insertion location searcher (1131) to “In this study, LLM data preprocessing was performed.” At this time, the context-preserving inserter (1132) vectorizes the sentence before insertion and the sentence after insertion using a semantic embedding-based artificial intelligence model, and then calculates the cosine similarity between the two sentence vectors. In this case, insertion is confirmed only when the difference (ΔSim) between the similarity scores of sentence vectors before and after insertion is less than or equal to a pre-set context-preserving threshold, for example, 0.05. On the other hand, if the difference (ΔSim) between the similarity scores of sentence vectors before and after insertion is calculated to be greater than 0.05, it is determined that the naturalness of the context has deteriorated, so insertion is withheld or other candidate locations are searched. In this case, the context-preserving threshold can be set based on domain-specific experimental results or empirical criteria.

[0120] The multiple insertion spacing controller (1133) automatically adjusts the insertion spacing so that the same keyword is not excessively concentrated in a specific section within the document, and controls the balance of the keyword distribution throughout the document. The multiple insertion spacing controller (1133) tracks the location index of the inserted core keywords to calculate the location spacing between keywords, and if the calculated spacing is less than a preset minimum insertion spacing, it controls the insertion position so that the distance between two insertions is greater than or equal to the minimum insertion spacing. For example, the multiple insertion spacing controller (1133) controls the placement of the same keyword within the document with a spacing of at least three sentences, thereby maintaining the balance of the keyword distribution within the document. In addition, it controls the prevention of the same keyword being inserted two or more times within a single sentence. For example, if the core keyword “Transformer” is inserted in the 2nd and 4th sentences of the document, the multi-insertion interval controller (1133) can adjust the insertion of the keyword after the 5th or 6th sentence by withholding insertion for the 4th sentence and inserting the keyword thereafter, since the interval between the two insertion positions is calculated to be 2 sentences, which is less than the preset minimum insertion interval (e.g., 3 sentences). At this time, the multi-insertion interval controller (1133) can maintain the balance of the keyword distribution within the document by limiting the number of insertions so that the keyword insertion ratio does not exceed 5% of the total number of sentences, taking into account the total number of sentences within the document.

[0121] The insertion sentence verifier (1134) calculates a perplexity score using a language model for each sentence. At this time, the insertion sentence verifier (1134) measures the change in the language model perplexity score for sentences before and after insertion using a pre-set formula, for example, PPL = exp(1 / N Σ log P(Wi|context)). Here, N represents the total number of keywords or subwords included in the sentence to be analyzed, Wi represents the i-th keyword or subword unit element within the sentence, P(Wi|context) represents the probability that the i-th keyword appears under a given context, and context is defined as the sequence of all keywords prior to the i-th keyword.

[0122] Such an insertion sentence verifier (1134) calculates a first semantic similarity score and a first awkwardness index score for a sentence before the insertion of the core keyword, and calculates a second semantic similarity score and a second awkwardness index score for a sentence after the insertion of the core keyword. Then, if the difference between the first semantic similarity score and the second semantic similarity score is less than or equal to a pre-set context-preserving difference, and the difference between the first awkwardness index score and the second awkwardness index score (ΔPPL) is less than or equal to a pre-set awkwardness threshold, the sentence is confirmed as the final inserted sentence. At the same time, the change in contextual naturalness is quantitatively evaluated.

[0123] When the insertion sentence verifier (1134) calculates the semantic similarity score (relative to document vector) of the sentence “In this study, data preprocessing was performed.” before inserting the core keyword “LLM” as 0.91 and the perplexity score as 41.2, and calculates the semantic similarity score of the sentence “In this study, LLM data preprocessing was performed.” after inserting the core keyword “LLM” as 0.89 and the perplexity score as 43.5, the insertion sentence verifier (1134) determines the sentence as the final inserted sentence if the difference in semantic similarity between the two sentences |0.91-0.89| = 0.02 is within the context-preserving difference (e.g., 0.05 or less) and the difference in perplexity score |43.5-41.2| = 2.3 is within the perplexity threshold (e.g., 5 or less).

[0124] On the other hand, if a sentence in which the keyword “sentiment analysis” is inserted instead of “LLM” is transformed into “In this study, sentiment analysis data preprocessing was performed,” and the context-preserving difference number is 0.12 or higher, or the awkwardness index score difference is 10 or higher, the insertion sentence verifier (1134) determines that the inserted keyword is unnatural in context and controls the insertion to be withheld or to search for another location. At this time, the awkwardness threshold can be set according to experimental results by domain or empirical criteria.

[0125] The anti-bias controller (1135) controls the data distribution so that the same keyword is not excessively inserted into a specific section within the document, thereby preventing bias. More specifically, the anti-bias controller (1135) calculates the distance score between sentences for the inserted same keywords and adjusts the insertion position so that the calculated distance score is greater than or equal to a preset minimum insertion interval. Additionally, the anti-bias controller (1135) prevents excessive repeated insertion of specific keywords and maintains the balance of the keyword distribution within the document by controlling the ratio of the number of inserted keywords to the total number of keywords in the document to be less than or equal to a preset upper limit score for the insertion ratio (e.g., 5%). For example, if the total number of keywords in the document is 2,000 and the same keyword “Transformer” is inserted 120 times, the anti-bias controller (1135) calculates the insertion ratio as 120 / 2000 = 0.06 (6%). At this time, if the calculated score exceeds the pre-set upper limit score of 5% for the insertion ratio, the anti-bias controller (1135) detects the section that is inserted in excess and removes some of the insertion positions or relocates them to subsequent paragraphs so that the insertion ratio becomes 5% or less. Additionally, if the same keyword “Transformer” is inserted consecutively in the 2nd and 4th sentences, the anti-bias controller (1135) determines that the distance score between the two insertion positions is calculated as 2 sentences and falls short of the minimum insertion interval (e.g., 3 sentences), and then moves the insertion position to the 5th sentence or later to maintain the distance between the same keywords at or above the minimum insertion interval.

[0126] The keyword insertion unit (113) controls the minimum spacing of identical keywords and distributes them evenly throughout the document, thereby preventing keyword overcrowding and bias.

[0127] The training data generation unit (114) evaluates the quality of sentences into which core keywords are inserted by the keyword insertion unit (113), automatically selects only sentences suitable for LLM learning, and generates a final training data set. More specifically, the training data generation unit (114) quantitatively analyzes sentences into which core keywords are inserted by the keyword insertion unit (113) using a pre-set language model and calculates an awkwardness index score for the sentence. The training data generation unit (114) calculates the difference between a first awkwardness index score for the sentence before the insertion of the core keyword and a second awkwardness index score for the sentence after the insertion of the core keyword, and determines the sentence as a natural sentence and confirms the insertion only if the difference is below a pre-set awkwardness threshold. Afterward, the training data generation unit (114) saves the confirmed sentence as part of the training data set or converts it into an LLM learning format (JSON, CSV, TFRecord, etc.) to generate the final training data. For example, the training data generation unit (114) can calculate an awkwardness index score of 41.2 for the sentence “In this study, data preprocessing was performed.” before insertion, and an awkwardness index score of 43.5 for the sentence “In this study, LLM data preprocessing was performed.” after insertion of the core keyword. At this time, if the difference between the awkwardness index scores of the two sentences ΔPPL=|43.5-41.2|=2.3 is less than or equal to a preset awkwardness threshold, the training data generation unit (114) determines the sentence as a natural sentence and includes it in the LLM training data set.

[0128] On the other hand, if the keyword “sentiment analysis” is inserted instead of “LLM” and the sentence is transformed into “In this study, sentiment analysis data preprocessing was performed,” the awkwardness index score can be calculated as 61.7. The difference in the awkwardness index score according to this awkwardness index score is calculated as ΔPPL=|61.7-41.2|=20.5, and if it exceeds the threshold value, the training data generation unit (114) may not generate the sentence as training data.

[0129] As illustrated in FIG. 3, the training data generation unit (114) includes a quality evaluator (1141), a data selector (1142), and a format converter (1143) to automatically verify and select the quality of the training data, thereby minimizing unnecessary noise data during the training process and increasing the response accuracy and consistency of the LLM. Here, the quality evaluator (1141) uses a language model to simultaneously calculate an awkwardness index score for a sentence in which a core keyword is inserted by the keyword insertion unit (113), an awkwardness index score for a sentence in which a core keyword is not inserted by the keyword insertion unit (113), and a sentence semantic similarity score, and quantitatively evaluates naturalness and semantic consistency. For example, the quality evaluator (1141) can calculate the awkwardness index score of the sentence before insertion “In this study, data preprocessing was performed.” as 41.2 and the semantic similarity score as 0.92, and the awkwardness index score of the sentence after insertion “In this study, LLM data preprocessing was performed.” as 43.5 and the semantic similarity score as 0.89.

[0130] The data selector (1142) adopts only sentences as training data in which the difference in awkwardness index scores calculated by the quality evaluator (1141) is less than or equal to a preset awkwardness threshold and the difference in semantic similarity is less than or equal to a preset semantic match threshold. For example, the data selector (1142) adopts the sentence as training data when the difference in awkwardness index scores is |43.5-41.2| = 2.3 ≤ less than the awkwardness threshold and the difference in semantic similarity is |0.92-0.89| = 0.03 ≤ less than the semantic match threshold.

[0131] The format converter (1143) converts the adopted sentence into an LLM learning format such as JSON or CSV, and records the inserted key keywords and their weight information as metadata to create a learning data set.

[0132] The learning data generation module (110) of the present invention enables the data collection unit, keyword determination unit, keyword insertion unit, and learning data generation unit described above to be organically linked to ensure a certain level of data preprocessing quality, extract key keywords based on importance, and automatically generate learning data that maintains the naturalness and semantic consistency of the sentence when inserting keywords and ensures that the inserted sentence does not damage the meaning of the original text. In addition, the learning data generation module (110) of the present invention transmits the generated learning data to the learning response optimization module (120).

[0133] Hereinafter, with reference to FIG. 9, the keyword-based LLM learning response optimization and performance verification management method of the present invention will be described in detail. Here, the subject of each step of the method of the present invention becomes a component of the keyword-based LLM learning response optimization and performance verification management device described above. That is, the subject of each step of the method of the present invention and the component of the device of the present invention are identical.

[0134] Therefore, all descriptions of the keyword-based LLM learning response optimization and performance verification management device of the present invention may be applied as is to the method of the present invention.

[0135] The method of the present invention proceeds through steps (a) (S110) to (c) (S130), and when training the LLM, quantitatively calculates the importance of keywords within the training data, strengthens the responses by reflecting the calculated importance in the training and response process, re-ranks the generated responses according to semantic and statistical criteria, and improves the learning quality of the LLM by selecting and accumulating only high-quality responses through repeated verification of the same question.

[0136] Below, each step of the keyword-based LLM learning response optimization and performance verification management method is explained in detail.

[0137] The method of the present invention begins with step (a) (S110), in which a learning data generation module (110) converts received raw data into text by natural language processing, divides it into word units, and assigns importance scores to the divided words to generate LLM training data. Subsequently, the method proceeds to step (b) (S120), in which a learning response optimization module (120) receives the data generated by the learning data generation module (110), assigns learning weights according to the importance assigned to the words in the data, trains a Large Language Model (LLM) with the data to which the learning weights have been assigned, and receives a response generated from the Large Language Model.

[0138] Here, step (b) is in which the learning response optimization module (120) uses the importance score of the word provided from the learning data generation module (110) to determine the attention weight (α) of the word within the pre-set Transformer self-attention structure. ij Adjust ) to be proportional to the importance score, and the adjusted attention weights (α ' ijThe process can proceed through step (b-1), in which weights are assigned to each word by adjusting them according to a pre-set weight formula, and step (b-2), in which multiple response candidates generated by the LLM for the user's question are received, and the degree of keyword reflection and semantic appropriateness of each response are evaluated to readjust the ranking of the responses through a pre-set response formula. After step (b), the process proceeds to step (c) (S130), in which the effectiveness verification management module (130) receives the response generated by the learning response optimization module (120), calculates performance indicators by comparing and analyzing the responses before and after learning or before and after reinforcement for data of the same quality, verifies the response quality of the large-scale language model, and feeds the results back to the learning response optimization module (120).

[0139] Through these steps, the method of the present invention quantitatively calculates the importance of keywords when training the LLM, strengthens the responses by reflecting this, re-ranks the generated responses based on semantic and statistical criteria, verifies the responses to the same question, and ensures that high-quality responses are generated.

[0140] The method of the present invention proceeds through steps (a) (S110) to (d) (S140) and, in order to improve the learning efficiency of a large-scale language model (LLM), automatically selects key keywords in the data preprocessing step and inserts the selected keywords into the original text or augmented text in a manner that does not compromise the context, thereby automatically generating training data with a sentence structure optimized for LLM learning.

[0141] Hereinafter, each step of the method of the present invention will be described in detail.

[0142] The method of the present invention begins with step (a) (S110), in which a data collection unit (111) receives raw data of various formats, converts it into text, divides the text into sentence units through natural language processing, performs keyword analysis on the divided sentences, determines whether the analyzed keywords meet pre-set criteria, and extracts the matching keywords as candidate keywords. For example, the data collection unit (111) in step (a) collects the following through sentence boundary recognition: “In this study, data preprocessing is systematized and an experimental design utilizing a large-scale language model is proposed. Keyword insertion is performed so as not to harm the context, and a Transformer-based structure shows advantages in understanding long contexts. Also, sentiment analysis techniques are not directly related to the subject of this study, and a stepwise approach of pre-training and fine-tuning is compared.” S1. “In this study, data preprocessing is systematized.” S2. “An experimental design utilizing a large-scale language model is proposed.” S3. “Keyword insertion should be performed restrictively so as not to harm the context.” S4. “Transformer-based structures show advantages in understanding long contexts.” S5. “Sentiment analysis techniques are not directly related to the topic of this study.” S6. “Compares the stepwise approach of pre-training and fine-tuning.” In this way, the original text is divided into 6 sentences.

[0143] The data collection module extracts candidate keywords from the six divided sentences, focusing on nouns, proper nouns, and technical terms, as shown in Table 4 below.

[0144] Candidate Keywords Frequency of appearance Domain pre-matching Sentence similarity (maximum) Candidate Judgment Transformer 5 is available 0.9 Adoption Learning 7 is available 0.92 Adoption LLM 4 is available 0.9 Adoption data 3 is available 0.88 Adoption insertion 2 doesn't exist 0.33 hold Preprocessing 2 doesn't exist 0.41 hold Sentiment analysis 2 is available 0.7 hold model 6 is available 0.4 hold

[0145] Here, the data collection unit (111) extracts a keyword as a candidate keyword if its occurrence frequency is 2 or higher, if it matches a professional keyword registered in the domain dictionary, or if its semantic similarity score with the sentence vector is 0.75 or higher. At this time, the occurrence frequency and sentence similarity shown in Table 4 are examples. The occurrence frequency and sentence similarity may vary depending on the actual operating environment. Subsequently, the keyword determination unit (112) evaluates the candidate keywords extracted by the data collection unit (111) according to pre-set evaluation criteria to calculate an importance score, and proceeds to step (b) (S120), in which a keyword whose calculated importance score is greater than or equal to a pre-set importance threshold is determined as a core keyword. For example, the keyword determination unit (112) in step (b) extracts the importance score of the candidate keyword as shown in Table 5 below.

[0146] Keywords Frequency-based importance scores Chi-square score Cosine similarity score Transformer 0.25 0.032 0.9 LLM 0.22 0.028 0.93 Data Preprocessing 0.2 0.025 0.88 Sentiment analysis 0.15 0.012 0.7 model 0.03 0.001 0.4

[0147] At this time, the frequency-based importance score, chi-square value, and cosine similarity value shown in Table 5 are examples. They may vary depending on the frequency-based importance score, chi-square value, and cosine similarity value. Then, the keyword determination unit (112) of step (b) can determine keywords whose calculated importance score is greater than or equal to a preset importance threshold, namely “LLM”, “Transformer”, and “data preprocessing,” as core keywords. These core keywords can then be generated into a refined core keyword set such as {“LLM, “Transformer”, “data preprocessing”}. After step (b), the process proceeds to step (c) (S130), where the core keywords are inserted. The keyword insertion unit (113) of step (c) calculates a phrase-unit embedding vector from the sentence and then calculates the semantic similarity with the keywords. It then selects a location where the similarity with the core keywords is maximized or where the semantic similarity score is greater than or equal to the threshold. For example, regarding the keyword “LLM”, S1 “In this study | “In this study, data preprocessing was systematized.” At this time, the semantic similarity score for “in this study” is 0.55, the semantic similarity score for “data preprocessing” is 0.90, and the semantic similarity score for “systematized” is 0.42. The keyword insertion unit (113) of step (c) calculates these semantic similarity scores and can determine that the place before “LLM” 'data preprocessing’ is a candidate insertion location where the core keyword can be inserted. Additionally, if the semantic similarity score before inserting the core keyword is calculated as 0.92 and the semantic similarity score after inserting the core keyword is calculated as 0.9, the keyword insertion unit (113) of step (c) calculates the difference score between each calculated semantic similarity score, and if the difference score of the calculated semantic similarity score is less than or equal to a pre-set context-preserving threshold, the insertion of the core keyword is confirmed, and “in this study, data preprocessing was systematized.” can be modified to “in this study, LLM data preprocessing was systematized.”

[0148] Additionally, the keyword insertion section (113) of step (c) can ensure that the same keyword is used at intervals of at least three sentences apart. For example, if a key keyword is used in the sentence S1. “In this study, data preprocessing was systematized.”, the key keyword can be used in S4. “Transformer-based structures show advantages in understanding long contexts.” which is three sentences apart. The keyword insertion section (113) of step (c) ensures that the ratio of key keywords inserted into the entire document is 5% or less, and prevents repetitive and excessive use.

[0149] After step (c) as described above, the training data generation unit (114) quantitatively analyzes the sentence in which the core keyword is inserted by the keyword insertion unit (113) using a language model to calculate the awkwardness index score of the sentence, and allows insertion only when the difference between the awkwardness index score of the sentence before the insertion of the core keyword and the awkwardness index score of the sentence after the insertion of the core keyword is below a preset awkwardness threshold, thereby generating the sentence and proceeding to step (d) (S140), which generates the generated sentence as training data. Here, the training data generation unit (114) in step (d) calculates the difference between the awkwardness index score of the sentence before the insertion of the core keyword (first awkwardness index score) and the awkwardness index score of the sentence after the insertion of the core keyword (second awkwardness index score). Then, it calculates the difference between the first semantic similarity score for the sentence before the insertion of the core keyword and the second semantic similarity score for the sentence after the insertion of the core keyword. For example, in the case where the learning data generation unit (114) of step (d) modifies sentence S1 “In this study, data preprocessing was systematized” by inserting the core keyword “LLM” to “In this study, LLM data preprocessing was systematized,” the awkwardness index score (first awkwardness index score) of the sentence before the insertion of the core keyword “LLM” can be calculated as 41.2 and the first semantic similarity score as 0.92. Then, the awkwardness index score (second awkwardness index score) of the sentence after the insertion of the core keyword “LLM” can be calculated as 43.5 and the second semantic similarity score as 0.90.

[0150] The training data generation unit (114) of step (d) can calculate the difference in awkwardness index score |43.5-41.2| = 2.3 and the difference in semantic similarity score |0.90-0.92| = 0.02 according to the calculated score. Since the difference in this way is below a preset threshold, the sentence is adopted as training data.

[0151] The method for generating LLM training data using keyword-based data preprocessing and insertion optimization may not only repeat steps (a) (S110) through (d) (S140), but may also proceed to step (b) where keywords whose similarity to core keywords is greater than or equal to a preset similarity threshold are classified into the same group, and keywords with overlapping meanings are refined (S120-1).

[0152] Through these steps, the method for generating LLM training data using keyword-based data preprocessing and insertion optimization implements a cyclic data improvement structure that automatically verifies and supplements the quality of training data, thereby enabling the reliability, diversity, and consistency of the data to be repeatedly strengthened.

[0153] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0155] 100: Keyword-based LLM learning response optimization and performance verification management device 110: Training Data Generation Module 111: Data Collection Unit 112: Keyword Determination Unit 113: Keyword Insertion Section 114: Training Data Generation Section 120: Learning Response Optimization Module 121: Weighted Learning Unit 122: Response Re-ranking Unit 123: Domain Tuning Department 124: Continuous Learning Department 130: Effectiveness Verification Management Module 1131: Performance Evaluation Department 1132: Statistical Processing Department 1133: Iterative Verification Department 1134: Visualization Reporting Department 1135: Domain Performance Verification Department

Claims

Claim 1 A training data generation module (110) that converts received raw data into text by natural language processing, divides the data into keyword units, analyzes the divided keywords according to a first evaluation criterion set in advance to calculate an importance score, and generates LLM training data including keywords whose calculated importance score is greater than or equal to a pre-set threshold;The learning response optimization module (120) receives LLM training data generated by the learning data generation module (110), analyzes keywords, sets training weights for each analyzed keyword, and receives candidate responses generated by training a large-scale language model (LLM) with keywords to which training weights are applied; and the effectiveness verification management module (130) analyzes the candidate responses generated by the learning response optimization module (120) based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, determines the quality of the candidate response as good if the calculated score is greater than a pre-set quality standard score, and determines the quality of the candidate response as poor if the calculated score is less than a pre-set quality standard score. The learning data generation module (110) includes a data collection unit (111) that receives raw data of various formats and converts it into text, and for candidate keywords extracted by the data collection unit (111), quantifies them according to pre-set evaluation criteria to calculate importance, and the calculated importance A keyword-based LLM learning response optimization and performance verification management device characterized by comprising: a keyword determination unit (112) that determines a keyword that is greater than or equal to a preset importance threshold as a core keyword; a keyword insertion unit (113) that receives the core keyword determined by the keyword determination unit (112), divides the text into phrases of each sentence constituting the original text, converts each phrase into an embedding vector, calculates the semantic similarity between the embedding vector of the phrase and the embedding vector of the core keyword, determines the phrase having the highest similarity among the calculated semantic similarities as an insertion candidate position, and inserts the core keyword at the corresponding insertion candidate position; and a learning data generation unit (114) that uses a language model to calculate the difference between the awkwardness index value of the sentence before the core keyword is inserted and the awkwardness index of the sentence with the core keyword inserted, and generates a sentence in which the difference value is less than or equal to a preset awkwardness threshold as training data. Claim 2 A keyword-based LLM learning response optimization and performance verification management device according to claim 1, wherein the learning response optimization module (120) includes a weighting application learning unit (121) that, after receiving an importance score from the learning data generation module (110), adjusts the weight of the corresponding keyword within a pre-set learning mechanism to be proportional to the importance score. Claim 3 A keyword-based LLM learning response optimization and performance verification management device according to claim 1, wherein the learning response optimization module (120) receives a plurality of candidate responses from a large-scale language model, calculates a frequency-based importance score or a statistical association score or a semantic similarity score for the candidate responses, substitutes the calculated score into a pre-set weighted response calculation formula to calculate a response fit score, and then reorders the responses in order of score. Claim 4 In paragraph 3, the response re-ranking unit (122) calculates a response suitability score for all candidate responses according to a pre-set weighted response calculation formula, and then selects the candidate response with the highest score as the final response, keyword-based LLM learning response optimization and performance verification management device. Claim 5 A keyword-based LLM learning response optimization and performance verification management device according to claim 4, comprising a domain tuning unit (123) that individually sets different learning weights and response reordering weights according to the data characteristics of each domain. Claim 6 A keyword-based LLM learning response optimization and performance verification management device according to paragraph 3, comprising a continuous learning unit (124) that, when new LLM learning data is input from the learning data generation module (110) after LLM learning, compares the data with a pre-set keyword dictionary and selectively sets only the changed or added keywords as re-learning targets, thereby updating only the necessary parts of the entire raw data to maintain the up-to-dateness of the LLM model. Claim 7 In claim 1, the effectiveness verification management module (130) includes a performance evaluation unit (1131) that statistically verifies that candidate responses are analyzed based on standard responses to pre-set question data to calculate changes in keyword inclusion rates or semantic similarity scores, and that the calculated change values ​​are substituted into a pre-set response quality calculation formula to calculate a comprehensive score, and that if the comprehensive score is equal to or greater than a pre-set quality standard score, the response quality is judged to be good, and if it is less than the quality standard score, it is judged to be poor. Claim 8 In claim 1, the effectiveness verification management module (130) comprises a statistical processing unit (1132) that calculates the performance improvement rate of a large-scale language model by reflecting the score calculated in the state before the large-scale language model is learned by the learning response optimization module (120) and the score calculated in the state after the large-scale language model is learned by the learning response optimization module (120) into a preset improvement rate formula. This is a keyword-based LLM learning response optimization and performance verification management device. Claim 9 A keyword-based LLM learning response optimization and performance verification management device according to claim 1, wherein the effectiveness verification management module (130) includes a repetition verification unit (1133) that calculates at least one of an average score or a standard deviation by repeatedly analyzing the response generated by the learning response optimization module (120) based on the pre-set question data. Claim 10 In claim 1, the effectiveness verification management module (130) comprises a visualization reporting unit (1134) that generates information visualizing items for which the improvement range of the frequency-based importance score, statistical association score, or semantic similarity score for candidate responses is greater than or equal to a preset visual threshold score, and a domain performance verification unit (1135) that verifies the universality of the model by applying candidate responses to different domain data, a keyword-based LLM learning response optimization and performance verification management device. Claim 11 delete Claim 12 In claim 7, the performance evaluation unit (1131) feeds back candidate responses judged to have good response quality to the learning response optimization module (120), a keyword-based LLM learning response optimization and performance verification management device. Claim 13 (a) A step (S110) in which a training data generation module (110) converts received raw data into text by natural language processing, divides it into keyword units, analyzes the divided keywords according to a first evaluation criterion set in advance to calculate an importance score, and generates LLM training data including keywords whose calculated importance score is greater than or equal to a pre-set threshold; (b) A step (S120) in which the LLM training data generated by the training data generation module (110) is received, the keywords are analyzed, a training weight is set for each analyzed keyword, and a large-scale language model (LLM) is trained with the keywords to which the training weight is applied to receive the generated candidate response; (c) A step (130) in which an effectiveness verification management module (130) analyzes the candidate response generated by the training response optimization module (120) based on pre-set question data, calculates at least one of a keyword inclusion rate or a semantic similarity score, and if the calculated score is greater than a pre-set quality standard score, the quality of the candidate response is judged to be good, and if the calculated score is less than a pre-set quality standard score, the candidate response The method includes a step (S130) of determining the quality as defective, wherein step (a) comprises: a step in which the data collection unit (111) of the learning data generation module (110) receives raw data of various formats and converts it into text; and a step in which the keyword determination unit (112) of the learning data generation module (110) quantifies candidate keywords extracted by the data collection unit (111) according to a preset evaluation criterion to calculate importance, and determines keywords whose calculated importance is greater than or equal to a preset importance threshold as core keywords.A method for managing keyword-based LLM learning response optimization and performance verification, comprising: a step in which the keyword insertion unit (113) of the above-mentioned learning data generation module (110) receives a core keyword determined by the keyword determination unit (112), divides the original text into phrases of each sentence constituting the original text, converts each phrase into an embedding vector, calculates the semantic similarity between the embedding vector of the phrase and the embedding vector of the core keyword, determines the phrase having the highest similarity among the calculated semantic similarities as an insertion candidate position, and inserts the core keyword at the corresponding insertion candidate position; and a step in which the learning data generation unit (114) of the above-mentioned learning data generation module (110) calculates the difference value between the awkwardness index value of the sentence before the core keyword is inserted and the awkwardness index of the sentence with the core keyword inserted using a language model, and generates a sentence in which the difference value is less than or equal to a preset awkwardness threshold as learning data. Claim 14 In claim 13, the method for managing keyword-based LLM learning response optimization and performance verification includes step (b) in which, after the learning response optimization module (120) receives an importance score from the learning data generation module (110), the weight of the corresponding keyword is adjusted in proportion to the importance score within a pre-set learning mechanism structure (b-1). Claim 15 In claim 14, the method for managing keyword-based LLM learning response optimization and performance verification further includes step (b-2), wherein the learning response optimization module (120) receives a plurality of candidate responses from a large-scale language model, calculates a frequency-based importance score or a statistical association score or a semantic similarity score for the candidate responses, substitutes the calculated score into a pre-set weighted response calculation formula to calculate a response fit score, and then selects the candidate response with the highest score as the final response.

Images