A learning system that uses a large-scale language model with multiple functions to generate and evaluate prompts.

Abstract

Conventional prompt generation techniques rely on manual design, resulting in limitations in consistency and efficiency. Furthermore, they lack sufficient mechanisms for evaluating generated prompts and continuously improving them. They also lack mechanisms for adapting to complex tasks and for incorporating evaluation results into model retraining. [Solution] A learning system is provided that uses a teacher language model and a student language model. The student language model performs COT-style prompt generation (first function) and output generation (second function), scores the output generated by the teacher language model based on evaluation criterion data, and generates data for retraining. This dynamically and continuously improves prompt generation accuracy and model performance, and improves reliability by integrating human review results.

Description

Technical Field

[0001] The present invention relates to a technology for optimizing prompt generation and output generation by AI, and relates to a learning system in which a teacher model (teacher LLM) and a student model (student LLM) cooperate to improve generation performance. In particular, by simultaneously providing evaluation criterion data and correct answer data, "by simultaneously giving the teacher LLM a verbalizable instruction (evaluation criterion) and a non-verbalizable or difficult instruction (correct answer data), efficient learning is promoted.

Background Art

[0002] In the field of natural language processing (NLP), the use of language models, including large-scale language models (LLMs), is advancing, and task-specific AI that exhibits high-precision performance for specific tasks is widely used. This enables prompt design and data selection suitable for specific tasks, realizing the construction of efficient task-specific systems. However, there are two challenges to the widespread adoption of LLMs in society. The first is the issue of hallucination. In particular, to address this, domain-specific learning, such as in medicine, is necessary, but efficient methods for such learning are not sufficiently clear, and various engineers are resorting to trial and error. This invention is a mechanism that automates such engineers' work, enabling non-engineers to efficiently create LLMs suited to their own work. Specifically, by preparing a GPU environment in the cloud, preparing approximately 1000 examples of ground truth data including evaluation criterion data and input and output data, and preparing a teacher LLM with evaluation criterion data, a smart student LLM that has learned from the ground truth data and learned through evaluation by the teacher LLM will be automatically created. Another operational challenge in the social implementation of LLMs is electricity costs, but this problem can also be solved. Specifically, operating an LLM with a large memory size results in high electricity costs. Using this system, it becomes possible for students to create high-performance LLMs with small memory sizes that are tailored to specific purposes, and even non-engineers can create LLMs that can be used in a low-cost on-premise environment costing around a few million yen.

[0003] Furthermore, optimizing an LLM for a specific task requires careful model tuning and prompt design. Traditionally, methods have been employed to improve data generation and inference accuracy through collaboration between large-scale and small-scale LLMs. The large-scale LLM possesses the capability for high-precision inference and diverse data generation, and supports efficient learning through training the small-scale LLM. On the other hand, the small-scale LLM, while lightweight, inherits the knowledge of the large-scale LLM and plays a role in efficiently executing inference specialized for specific tasks.

[0004] Here, as a technique for transferring knowledge from a large-scale LLM to a small-scale LLM, techniques such as those described in Patent Document 1 are known. There is a method that utilizes soft labels (probability distributions of labels) generated from the teacher model to construct student models with more diverse and flexible inference capabilities. This method achieves both model weight reduction and performance improvement.

[0005] Furthermore, prompts that communicate tasks to the LLM are a crucial element that affects the model's performance. Traditionally, prompts have been manually designed using templates that reflect task specifications and label information. Among these, Patent Document 1 describes a technology that automatically generates prompts by combining task specifications and original data, thereby enabling more efficient and accurate prompt design.

[0006] In particular, chain-of-sort prompts (hereinafter referred to as COT prompts), which describe information about the thought process, are a format that explicitly describes the intermediate thought processes for problem solving, and are a mechanism that makes it easier for the model to perform correct sequential reasoning. By using COT prompts, it becomes possible to solve complex tasks by breaking them down, and the reasoning accuracy of the model is improved. Thus, COT prompts contribute to improving accuracy, especially in complex reasoning tasks.

[0007] Furthermore, to improve the performance of LLMs, evaluation and feedback on the generated prompts and output data are crucial. Traditionally, methods have been employed to quantitatively evaluate the output using evaluation criteria such as accuracy, precision, recall, and F1 score, and then improve the prompts and the model itself based on the results. This allows for step-by-step optimization of the model.

[0008] This invention relates to a technology for incorporating terminology standardization information into prompts when linking a Generative AI (LLM) with other order systems. With this technology, when ordering catheter stents in a medical institution, even if multiple product names exist, the Generative AI can present a specific name and output it as structured data usable by the system. Furthermore, in manufacturing, when ordering parts with many compatible alternatives, such as engine oil or brake pads, the LLM outputs terminology-standardized data, enabling smooth linkage with order management systems. In the logistics sector, even when linking with a warehouse inventory system, if item names are registered under multiple names, the Generative AI matches them against predefined standard terminology, preventing incorrect orders and facilitating efficient delivery arrangements. This prevents situations where different names and abbreviations coexist in various fields, facilitating information sharing between related systems and suppressing problems arising from input errors or misunderstandings. Therefore, this invention represents a new technology that brings significant convenience to the social implementation of LLM. Based on this, the invention enables order processing across a wide range of industries and contributes to cost reduction.

[0009] Patent Document 1 discloses a technology that utilizes data classification, particularly soft labels, to convey information about ambiguous cases to a model. This includes labels such as "positive" and "negative" (which provide correct label information) to text data, and soft labels indicating the probability of belonging to a specific category. For example, by assigning 80% to "positive" labels and 20% to "negative" labels, the technology allows for model learning that incorporates label ambiguity while maintaining flexibility by not being fixed to a single label, thereby improving the performance and efficiency of the model. However, this comes with the constraint that incorporating label ambiguity carries the risk of ambiguous evaluation. Nevertheless, we have not yet reached the point of inputting evaluation criteria for what constitutes ideal ambiguity into the AI. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Patent No. 7343566 [Overview of the Initiative] [Problems that the invention aims to solve]

[0011] However, conventional technologies (particularly Patent Document 1) have the following problems.

[0012] (1) Optimization of the efficiency and quality of prompt design Traditionally, prompt design has relied heavily on manual processes, resulting in limitations in terms of consistency and efficiency. Manually designed prompts are highly dependent on the designer's skills and experience, leading to inconsistent quality across tasks and datasets. COT-style prompts, in particular, require the decomposition and description of problems into intermediate steps, placing a significant burden on the design process. On the other hand, prompts generated by automated generation technologies often lack sufficient quality and task adaptability, and integrated mechanisms for evaluating and improving generated prompts are limited. Patent Document 1 discloses a method for generating prompts based on task specifications and source data, but the prompts are structured in a format that includes a text type (movie review or article), a label type (sentiment or category classification), and a label-token numerator that allows confirmation of the label's position. There is no scoring system to quantitatively evaluate the results of sentiment analysis, and although these prompts are input into a predictive completion model to generate new data and label information, the detailed mechanism for evaluating and improving the generated prompts is not disclosed. Furthermore, the selection of training data is fixed, and there is no data selection or improvement based on evaluation criteria, thus deductive learning is not possible.

[0013] (2) Use and improvement of COT prompts COT prompts are a format for breaking down complex problems into intermediate inference steps, allowing models to process them step by step. However, conventional technologies do not disclose a mechanism for efficiently generating COT prompts or continuously improving their quality. In particular, while Patent Document 1 mentions prompt generation including text types (e.g., movie reviews) and label types (emotions), it focuses on generating soft labels. However, it lacks detailed evaluation and feedback loops to improve the quality of the generated data. Furthermore, it does not directly disclose prompt generation in COT format, and is limited to data-driven learning, failing to perform deductive learning without evaluation criteria. Therefore, it does not introduce clear and deductive evaluation criteria in data generation and analysis.

[0014] (3) Lack of dynamic assessment and learning processes Conventional technologies do not adequately disclose mechanisms for evaluating the output generated by a model and incorporating those results into the model's retraining process. In particular, there is a lack of mechanisms to incorporate evaluation results and adjust the model's learning process without requiring engineer intervention. In particular, while Patent Document 1 describes a mechanism for incorporating evaluation and feedback of generated data into learning, it is limited in its mechanisms for adjusting the model based on evaluation results.

[0015] (5) Design and applicability of evaluation criteria Conventional technologies have a problem in that the criteria for evaluating model output differ for each task, making it difficult to design a unified standard. Furthermore, there is insufficient mechanism for integrating quantitative evaluation (accuracy, recall, etc.) and qualitative evaluation (contextual naturalness and logical consistency). In particular, while Patent Document 1 includes a description of evaluation criteria, it does not mention the uniformity of evaluation criteria across different tasks or domains.

[0016] (6) Challenges of dataset diversity and consistency Conventional technologies struggle to maintain consistency while improving the diversity of training datasets. Furthermore, their ability to adapt to unseen data and unknown tasks is limited. In particular, while Patent Document 1 describes a technology for generating data using original data, it does not disclose a method for simultaneously ensuring data diversity and consistency.

[0017] This invention has been made in view of these problems and provides a learning system that integrates prompt generation, evaluation, and improvement. This invention realizes a series of processes from the generation of one or more prompts from thought process information, chain of sort information, and terminology standardization information to their evaluation and the creation of retraining data / additional training data, and surpasses the prior art and Patent Document 1 in that it achieves both automation and accuracy improvement in prompt design. As a result, it is possible to simultaneously achieve improved accuracy for specific tasks and general adaptability. [Means for solving the problem]

[0018] This invention efficiently and accurately realizes the processes of prompt generation, output generation, evaluation, and retraining. Furthermore, by utilizing chain-of-sort (COT) prompts, it enables the visualization of the thought process and continuous performance improvement through dynamic evaluation. In addition, by incorporating human review results and evaluation scores as feedback, it provides a learning system that combines reliability and adaptability.

[0019] Furthermore, since the present invention is a system that integrates a specific hardware configuration (e.g., distributed processing using GPUs) and efficient data management functions (e.g., logging of evaluation results), it provides concrete means to solve technical problems that go beyond mere algorithms.

[0020] <1. Improved accuracy of COT prompt generation (Claim 1, 2, 3, 8)> The first function of the student LLM (Large-Scale Language Model) described in claims 1 and 2 generates prompts, including COT format, based on input data. These prompts are designed to break down complex tasks into intermediate inference steps, allowing the model to process them step by step. This approach makes the inference process explicit and improves answer accuracy and consistency, especially for complex tasks. In addition, in claim 3, by sequentially improving the correct data together with the COT prompt based on the evaluation result, the generation performance is continuously improved. Furthermore, in claim 8, by generating a prompt while referring to a dataset including a sample prompt, a high-quality prompt that achieves both diversity and consistency can be obtained.

[0021] <2. Improvement of Output Quality and Dynamic Evaluation (Claims 3, 6, 7)> The teacher LLM described in claim 3 evaluates the output based on each function (COT prompt generation and output generation) of the student LLM and generates relearning data or additional learning data. By repeating this process, the quality of the output data is repeatedly improved. As a result, through dynamic evaluation and continuous relearning, the performance of the model is gradually improved. In addition, in claims 6 and 7, the evaluation output result is recorded in the log as scoring information, and high-score COT prompts and output data are preferentially used as relearning data. This optimizes the selection of learning data, thereby improving the learning efficiency. Furthermore, claim 7 includes a process in which the teacher LLM generates an amendment for low-evaluation output data, thereby constructing a feedback loop for quality improvement. As a result, through feedback based on evaluation criteria and reflection of amendments, low-quality data is also effectively utilized and contributes to the improvement of the model.

[0022] <3. Utilization of Human Review and Improvement of Reliability (Claim 5)> In claim 5, the results of human review can be incorporated as feedback into the relearning or additional learning of the student LLM. By leveraging human expertise, experience, and know-how, the validity and reliability of the model are ensured, enabling its use particularly in fields such as medicine and law where high accuracy and knowledge / experience are required.

[0023] <4. Efficiency and Flexibility of Model Operation (Claims 1, 2, 8)> The collaborative system between teacher and student LLMs efficiently integrates the processes of prompt generation, output generation, evaluation, and retraining. This significantly reduces time and effort compared to traditional manual prompt design and evaluation processes, improving learning efficiency. Furthermore, the dataset configuration based on claim 8 allows the learning system to be optimized not only for specific tasks but also to quickly adapt to new tasks and domains. This results in a highly versatile system.

[0024] <5. Improved data diversity and versatility (Claim 1, 2, 8)> The COT prompt generation process defined in claims 1 and 2, and the dataset configuration in claim 8, ensure diversity in input and output data. This allows the learning system to adapt to different tasks and domains, improving its ability to handle unknown datasets.

[0025] <Meaning of Claims 6 and 7> In the learning system according to claim 6, the scoring information integrates evaluation results based on multiple evaluation criterion prompts generated by the teacher's large-scale language model and records the quality of each output data numerically or descriptively, and the scoring information is used as an indicator for determining the priority to be used for retraining the student's large-scale language model. In the learning system described in claim 7, the proposed modifications are automatically generated based on the evaluation results of the teacher's large-scale language model on the low-scoring output data, while referring to evaluation axis prompts. Furthermore, the proposed modifications are intended to improve the accuracy and consistency of the student's large-scale language model as data for retraining or additional training. Furthermore, the learning system of the present invention has technical features that reduce the manual work required by prompt designers in conventional technologies and dynamically and progressively improve generation accuracy. For example, in COT prompt generation, multiple hypotheses are automatically generated based on evaluation criterion prompts provided by the teacher model, and those with high evaluation scores are sequentially incorporated into the learning data, thereby achieving both task adaptability and efficiency simultaneously. [Effects of the Invention]

[0026] This invention provides a groundbreaking learning system that combines a teacher-led large-scale language model (teacher LLM) with a student-led large-scale language model (student LLM) having two or more functions. This system significantly improves the generation performance and versatility of large-scale language models that can handle complex tasks, and enables continuous model optimization by utilizing the evaluation and feedback functions of the teacher LLM.

[0027] Furthermore, the present invention dramatically improves the accuracy and efficiency of COT prompt generation compared to conventional retraining processes that rely on fixed data selection, by utilizing an evaluation loop via a teacher LLM. [Brief explanation of the drawing]

[0028] [Figure 1] A diagram showing the model configuration of multiple LLMs. [Figure 2] A diagram showing the system configuration and hardware arrangement of a learning system according to an embodiment of the present invention. [Figure 3] A functional block diagram of a learning system according to an embodiment of the present invention. [Figure 4] A flowchart illustrating the task processing flow by each LLM. [Figure 5] This diagram shows an overview of the task processing performed by each LLM, illustrating the process of merging user input with pre-set data. [Best Mode for Carrying Out the Invention]

[0029] Embodiments of the present invention will be described below with reference to the drawings.

[0030] Figure 1 shows a model configuration diagram of a teacher LLM and multiple student LLMs. The teacher LLM is generally a large-scale language model for general-purpose AI, such as the existing OpenAI GPT series or Google's PaLM. The teacher LLM is a large-scale language model that plays the role of transferring knowledge to student LLMs, aiming to improve the performance of student LLMs by sharing and evaluating knowledge between models. Student LLMs are multiple large-scale / small-scale language models for task-processing AI, effectively utilizing knowledge transfer from the teacher LLM to student LLMs to significantly improve the learning efficiency and performance of task-specific models. As a result, they are divided into lightweight models that process specific tasks, such as a first functional model and a second functional model, each processing different tasks. Student LLMs are configured to have multiple functions depending on their purpose for task processing. Although there are multiple student models in the diagram, one model may have multiple functions, or one LLM may be trained to have multiple functions. In addition to student LLM1 and student LLM2, further student LLMs may be added to subdivide their roles.

[0031] The network structure of the model shown in Figure 1 is a basic structure of a general neural network with input and output layers. It has multiple hidden layers (multilayer perceptrons), convolutional layers, and recurrent layers in the intermediate layers, and can be used in a wide range of fields such as text generation, speech recognition, image classification, and natural language processing. In addition to neural networks, it is possible to select the optimal network structure according to the specific task, such as feedforward neural networks, convolutional neural networks, recurrent neural networks, and transformer models. In this invention, a structure is adopted in which the teacher model and student model work cooperatively to advance prompt generation, evaluation, and optimization. The aim is to achieve gradual and stable improvement of prompt quality through a cooperative structure. On the other hand, generative opposing networks (GANs) have opposing characteristics, black box nature, and instability, so they may not be suitable for the cooperative characteristics required by this invention.

[0032] Figure 2 is a block diagram of the server 20 of the learning system. The computer of the server 20 includes a communication IF 22, an input / output IF 23, memory 25, storage 26, and a processor 29.

[0033] Communication IF22 is an interface for inputting and outputting signals so that the server 20 can communicate with external devices. Input / Output IF23 functions as an interface to input and output devices for receiving user input operations. Memory 25 is for temporarily storing programs and data processed by programs, and is a volatile memory such as DRAM. Storage 26 is a storage device for saving data, such as flash memory, HDD, or SSD. The processor 29 is hardware for executing the instruction set described in the program, and consists of an arithmetic unit, registers, peripheral circuits, etc.

[0034] Each model shown in Figure 1 is placed in either the processor 29 or memory 25. The processor 29 houses models that perform computationally intensive tasks such as inference and training, using powerful processors such as GPUs or TPUs. When a teacher LLM evaluates and updates student LLMs in real time, running them on the processor enables low-latency and highly efficient processing. On the other hand, when a small student LLM is placed in memory, because it is lightweight, it can process data in memory and efficiently execute necessary tasks without running the entire model on the processor, especially when the computational load is relatively small. Furthermore, student LLMs for task processing are specialized for specific tasks, and distributing them in separate memory locations using a distributed system improves the efficiency of parallel processing. Large-scale models require the storage of enormous amounts of data, so they are placed in high-speed, high-capacity memory. While the processing of the LLM model itself is performed on the processor, the model weights and data are stored in memory. These roles are complementary, and both the processor 29 and memory 25 can cooperate to efficiently operate the LLM.

[0035] Figure 3 is a functional block diagram of the processor 29 of the learning system. The processor 29 is a central processing unit that oversees the entire system and executes and controls the processing of each module. In Figure 3, the acquisition unit 29A integrates external data, and the inference unit 29B generates prompts.

[0036] The acquisition unit 29A collects the data necessary for the learning system to operate. It acquires input data (such as input data and correct answer data) used by student LLMs, and acquires evaluation criteria data necessary for teacher LLMs to perform evaluations. It also integrates external data by incorporating human review results and additional evaluation axis data. The integration of external data improves the learning and evaluation accuracy of both the student and teacher models by incorporating human review results and additional data from external systems.

[0037] The inference unit 29B performs inference processing regarding the behavior of the student LLM. Its first function is to generate prompts, including chain-of-sort prompts, specifically COT prompts based on the input data. In this specification, a "chain-of-sort (COT) prompt" refers to a prompt that breaks down a complex task into intermediate thought processes and explicitly structures those processes. Specifically, it describes multiple steps of a task in sequence, with the output of each step usable as input for the next step. Its second function is to create final output data through multiple hypotheses and inferences, referencing the input data and the prompts generated by the first function. This output data is used for evaluation and retraining processes by the teacher model.

[0038] The learning unit 29C performs relearning and additional training on the student LLM. It incorporates well-evaluated prompt and output data as relearning data and executes the relearning process by applying learning algorithms to improve and enhance the first and second functions. It also incorporates suggested corrections generated based on samples with low evaluation results, as well as improvement suggestions obtained from human feedback, as relearning data to improve the model's performance.

[0039] The evaluation unit 29D evaluates the output of the student LLM model using the teacher LLM. Using evaluation axis prompts as evaluation criterion data, it applies evaluation criteria to score the output results of the student model. Prompts and output data with high scores are extracted and used as training data, and the evaluation results are recorded. In addition, low-score processing is performed to generate additional feedback for low-scoring outputs, generating additional feedback prompts and creating correction suggestions. These correction suggestions are used in the retraining / additional training process.

[0040] The processing unit 29E adjusts the data format and performs pre-processing and post-processing. Input data is converted to a format suitable for learning and inference, and output data is converted to a format that is easy for evaluation and storage systems to handle. For example, it structures text data and converts it to JSON format.

[0041] The logging unit 29F records evaluation results and system operation history. It saves evaluation results in a scoring format and records scoring information. It prioritizes selecting high-scoring prompts as retraining data based on the scoring history, and manages (re)training data to support an efficient learning process.

[0042] Model Management Unit 29G manages the status and versions of teacher and student LLM models. It performs model version management to manage model updates after training and relearning. It records the performance history of each model, including accuracy, number of relearnings, and evaluation score, and uses this information for model comparison and optimization.

[0043] The prompt management unit 29H manages the generation, evaluation, and saving of prompts. It saves prompts that receive good evaluations and manages prompt samples. It records the correspondence between prompts and evaluation axes and efficiently links them using tagging and metadata management.

[0044] Figure 4 is a flowchart showing the flow of task processing performed by each LLM within the learning system, starting from the beginning of the learning process. Each LLM is a trained model; the teacher LLM is a general-purpose AI model located on the cloud, and the student LLM is a task-specific AI model installed on a standalone server 20. The retraining process (S5-S6) corresponds to claims 3 to 5.

[0045] First, the system loads the necessary input data, evaluation criteria data, and (existing) training datasets and places them in each module (step S1). Specifically, the acquisition unit 29A collects this data and integrates it with external data and human review results to prepare for the next processing. The acquired data is formatted by the processing unit 29E and converted into a format suitable for inference and evaluation.

[0046] The correct answer data (input and correct output data for each input) is set as needed and used primarily for student LLM2 training based on few-shot learning. Fine-tuning is also an option during the initial instruction. Evaluation criteria data is placed in the teacher LLM for instruction.

[0047] Next, the student LLM performs the first function (step S2). In step S2, the reasoning unit 29B of the student LLM1 generates chain-of-sort prompts that model the thought process based on the input data. In this process, knowledge-aid prompts and terminology-alignment support prompts may also be generated to improve the variety and accuracy of the prompts.

[0048] Next, the student LLM performs the second function (step S3). In step S3, the inference unit 29B creates the final output using the prompt and input data generated in the first task in step S2. This output reflects the model's inference accuracy and task-solving ability and is used for evaluation by the teacher model.

[0049] Next, the teacher LLM evaluates the output generated by the student model (step S4). In step S4, the evaluation unit 29D applies evaluation criteria using evaluation criterion prompts to score whether the generated output meets the evaluation criteria (e.g., accuracy, grammatical correctness, consistency). It selects high-scoring outputs to add to the training data, and if the output is low-scoring, an additional feedback prompt is generated and used as a suggestion for correction. The performance evaluation of student LLM1 by the evaluation unit 29D by the teacher LLM may be performed immediately after prompt generation, and the process is carried out in the order of step S2 → step S4.

[0050] Next, the teacher LLM generates (re)training data for the student model based on the evaluation results (step S5). In step S5, the learning unit 29C selects pairs of highly rated prompts and output data to construct a new training dataset. Additionally, low-rated data is modified based on the high-rated dataset to construct a new training dataset, and data including modified low-rated output data and human review results is generated and prepared for retraining.

[0051] In step S4, shown in Figure 4, the teacher LLM scores the output data generated by the student LLM2 based on the evaluation criterion prompt. A specific implementation example is the following evaluation criterion data. Finally, a total score is calculated, and the evaluation is performed based on that total score.

[0052] <1. Evaluation of compliance with the specified format> Example of evaluation criteria data: "Evaluate whether the following text conforms to the specified format. If there are format violations, point out the specific locations and clearly state the nature of the violations. The evaluation criteria are as follows: 5 points for full compliance, 4 points for some deficiencies but overall understandable, and 3 points or less for deficiencies in important areas." <Example of evaluation results by a teacher LLM> "Evaluation result: 4 points" Point of criticism: While the document as a whole generally conforms to the specified format, the order of the bullet points in the third paragraph is inappropriate. Specifically, the list items are not arranged alphabetically, reducing readability. As an example of correction, it would be desirable to place 'A. Description' at the beginning.

[0053] <2. Evaluation of consistency and coherence of terminology> Example of evaluation criteria data: "Evaluate the consistency and uniformity of terminology in the following document. Points will be deducted if different terms refer to the same concept or if the terminology is unclear. 5 points will be awarded for overall consistent terminology, and 4 points for some inconsistencies that are still understandable." Registered term to be used: Holtzmann catheter AGX12 AI, Chain of Sort, Catheter, We evaluated the consistency of terminology and the clarity of definitions on a scale of 1 to 5. We also checked for any inconsistencies in terminology within the detailed explanations. <Example of evaluation results by a teacher LLM> "Evaluation result: 3 points" Point of criticism: The terms 'catheter' and 'cardiac catheter' are used interchangeably. While they are sometimes synonymous, they give different impressions depending on the context. The Holtzmann AGX10 catheter is no longer in use, so please use the correct registered term.

[0054] <3. Evaluation of translation accuracy (if translation is included)> Example of evaluation criteria data: "Evaluate the translation on a scale of 1 to 5 based on the following criteria. If there are any unnatural translations or parts that differ in meaning from the original text, point them out specifically and clearly state the reason for the deduction." Fluency, accuracy, and consistency of the translation will each be evaluated on a scale of 1 to 5. Example of evaluation results by a teacher LLM: "Evaluation results: Fluency 4 points / Accuracy 3 points / Consistency 4 points" Point of criticism: The original text's 'clinical trial' is translated as 'medical examination,' but 'clinical trial' is generally more appropriate. While the meaning is conveyed, it could be misleading from a technical standpoint, so points were deducted. The rest of the translation is generally natural in Japanese.

[0055] <4. Evaluation of the validity of differential diagnoses and test / prescription cases in medical settings> Example of evaluation criteria data: "Evaluate whether the differential diagnosis and test prescription examples are appropriate in a clinical setting based on the following criteria. Confirm whether the target diagnosis is frequently performed and whether it conforms to the current diagnostic process, and indicate whether the tests can be performed in different medical settings such as clinics and large hospitals. A score of 10 is awarded if the frequency of diagnosis and the appropriateness in practice are high." Criterion 1: Generally, the frequency of the disease name (out of 10 points), Criterion 2: Appropriateness of the diagnosis in this situation (out of 5 points) Criterion 3: Evaluate the likelihood of diagnosis through testing in a medical setting (clinic, large hospital, etc.) (out of 5 points). Example of evaluation results by a teacher LLM: "Evaluation results: Criterion 1 (frequency) 8 points / Criterion 2 (validity) 5 points / Criterion 3 (likelihood of diagnosis) 4 points Total 17 points" Points to note: The frequency of 'chronic sinusitis' as a differential diagnosis is high and reasonable. However, at the clinic level, simple tests are mainly performed, and diagnosis based on advanced testing equipment like that found in large hospitals may be difficult. It is desirable to broaden the diagnostic options.

[0056] <Examples of actual evaluation results> <5. Evaluation of Q&A information creation based on research papers> Example of evaluation criteria data: "Create two questions from the following text and write answers to them. Evaluate the answers from the following perspectives: Perspective 1: Relevance between question and answer (out of 5 points), Perspective 2: Clinical relevance, Perspective 3: Reliability (out of 5 points), Perspective 4: Comprehensiveness (out of 5 points), Perspective 5: Clear presentation of evidence (out of 5 points), Perspective 6: Applicability to the Japanese healthcare environment (out of 5 points). Evaluate each item out of 5 points and explain why if there are any shortcomings." Example of evaluation results by a teacher LLM: "Perspective 1: 5 points (Question and answer match perfectly)" Perspective 2: 5 points (directly related to clinical practice) Perspective 3: 4 points (There are reliable sources, but the data is not up-to-date) Perspective 4: 3 points (Insufficient detail regarding multiple differential diagnoses) Perspective 5:5 points (Evidence from medical papers is clearly stated) Perspective 6: 4 points (Generally consistent with the Japanese healthcare system, but some Western standards are included) Total 27 points Points to note: The diagnostic methods are somewhat lacking in comprehensiveness; it would be desirable to add details of different diagnostic processes.

[0057] <6. Evaluation of Registry Registration> Example of evaluation criteria data: In response to the prompt, "Please evaluate the registry registration worker on a 5-point scale from the following three perspectives," the evaluation items are: completeness (are there any omissions or missing details regarding disease names or findings?), accuracy (are findings and disease names recorded correctly? are there any typos or input errors?), and appropriateness of judgment (is "unknown" appropriately indicated when findings are unclear or difficult to judge? Are there any entries based on speculation?), each on a 5-point scale from 1 to 5.

[0058] Next, these (re)training data are passed to the student model, and the student LLM uses the retraining data to improve its performance (step S6). The learning unit 29C improves the accuracy and diversity of prompt generation as a retraining of the first function, and improves the accuracy and consistency of the final output as a retraining of the second function.

[0059] Upon completion of a series of processes in the learning system, this subroutine terminates, the state of the retrained student model is saved, and it is prepared for the next learning or inference. This system has a cyclical process that continuously improves the performance of the student model by repeating output evaluation and retraining, and the teacher model is used to evaluate the data generated by the student model and provide appropriate training data. Furthermore, by appropriately utilizing high-scoring and low-scoring data, the learning efficiency can be maximized and data management can be performed efficiently. In addition, the logging unit 29F records prompts, output scores, and learning history, and the model management unit 29G manages the model version and performance history. Furthermore, the prompt management unit 29H saves the evaluated prompts and prepares them for reuse in subsequent learning and inference.

[0060] Figure 5 schematically illustrates the entire task processing of the learning system. Starting with user input, the system integrates processing based on user-provided input data, pre-set data, and externally imported data, as well as processing using data pre-loaded into the system and data from external sources, resulting in a more flexible and adaptive process. The integrated data improves the accuracy of prompt generation, output generation, and evaluation processes.

[0061] (1) Importing input data The teacher LLM and student LLM receive input data such as text data provided by the user (questions, task instructions), pre-set data (e.g., existing training datasets), and data obtained from external sources (e.g., news articles, technical documents). The processing unit 29E then adjusts the data format to convert it into a format suitable for prompt generation and inference processing (e.g., text normalization, conversion to JSON format). Data ingestion is based on user input operations or instructions from the processor.

[0062] (2) Prompt generation The prompt generation AI, which implements the first function of the student LLM, generates prompts in chain-of-sort (COT) format based on user input. Furthermore, the inference unit 29B generates prompts adapted to user input by referencing prompt design templates and samples from pre-set data stored in the prompt management unit 29H. Knowledge assistance and terminology standardization support prompts are also added as needed. Prompts are generated based on the trained COT prompt sample data and input data stored in the prompt management unit 29H.

[0063] (3) Output generation The prompt execution AI, which realizes the second function of the student LLM, generates output data based on the generated prompts, and the inference unit 29B generates output data. (4) Output evaluation The teacher LLM scores the generated output data / generated prompts and instructs the student LLM to add high-scoring data to the training data. If the score is low, it generates suggested corrections such as feedback prompts. The evaluation unit 29D performs evaluation processing based on evaluation criteria data, which consists of evaluation criterion data stored in the prompt management unit 29H and information obtained from external data sources.

[0064] (5) Retraining data generation The teacher LLM extracts high-scoring prompts and output data based on the evaluation results to generate a (re)training dataset. The high-scoring data sets are used as is, while the low-scoring data are further trained by adding feedback prompts and output modification suggestions. Alternatively, the low-scoring data is either discarded or its weighting is changed to reduce its impact during training. (6) Students' LLM relearning Based on the student LLM's relearning data, the learning unit 29C performs relearning to improve the first and second functions. [Example 1]

[0065] <Evaluation Criteria> As shown in Table 1, the evaluation criteria used in the evaluation unit 29D are a five - level criterion from 1 to 5 points that evaluates the quality of translation in terms of fluency (whether the text is natural and easy to read, or there are no awkward expressions), accuracy (whether the meaning of the original text is accurately conveyed, or there are no omissions or translation errors), and consistency (whether the terms and expressions are unified, or the style is well - organized). Also, in the example of registry registration, for the prompt "Please evaluate the registry registration operator at five levels from the following three perspectives", the evaluation items of completeness (whether there are no omissions of disease names or findings, or whether all are described without deficiencies), accuracy (whether the findings and disease names are correctly recorded, or there are no misrecords or input errors), and appropriateness of judgment (whether it is appropriately described as "unknown" when there are unclear findings or difficult judgments, or whether it is not filled in by speculation) are based on a five - level criterion from 1 to 5 points.

[0066]

Table 1

Example 2

[0067] <Data Preparation> Prepare pairs of input data and output data of about 1000 to 10000 pieces as correct labels, and prepare samples of COT prompts and evaluation axis prompts in text format. <Initial Settings of Student LLM> Based on an existing pre - trained LLM, perform minor fine - tuning referring to the texts of (A) and (B), or start from an untrained state. <COT Prompt Generation → Final Output Generation → Teacher LLM Evaluation → Feedback> · Student LLM1 generates a large number of COT prompts. · Student LLM2 makes inferences using the COT prompts and obtains the final output. · The teacher LLM evaluates the output and extracts highly - evaluated COT prompts. · Student LLM1 takes in the highly - evaluated COT prompts as re - learning data again and updates. <Iterative Learning and Improvement> Repeat the above process multiple times until finally, Student LLM1 and Student LLM2 reach an optimized state. If necessary, a human reviewer monitors and assists the evaluation process. Note that there may be cases where only Student LLM1 is trained or only Student LLM2 is trained. <Effects of the System of the Present Invention> (1) Reduce the design cost of the COT prompt and enable automatic generation and evaluation. By evaluating the COT prompt generated by Student LLM1 with the Teacher LLM and efficiently collecting and reusing the good ones, the accuracy of inference can be dramatically improved. (2) Utilize the feedback from the Teacher LLM to improve the quality of the COT prompt. Since the Teacher LLM is used as an evaluation module, in addition to human feedback, automatic evaluation by the LLM is combined to enhance learning efficiency and facilitate iterative optimization. Consequently, it contributes to the sophistication of the evaluation module. (3) Enable efficient inference with enhanced expertise through the division of roles of the Student LLM. By separating the "COT Prompt Generation Module" and the "COT Prompt Application Module", the learning pipeline is clarified, making it easier to optimize each LLM. Also, it contributes to the improvement of development efficiency through the division of roles. (4) Improve the performance of the entire system through a continuous re - learning process. Since a mechanism for incorporating high - evaluation COT prompts into the learning data of Student LLM1 is incorporated, new data and evaluation axes can be added during operation, and the system performance can be continuously updated.

[0068] <Examples of COT> <Prompt Example 2: COT Prompt Emphasizing Step - by - Step Thinking> Instruction: Please translate the following English text into Japanese. When translating, follow the steps below. 1. Decompose the components of the sentence. 2. Translate each component sequentially. 3. Please adjust the translation and word order so that the overall flow is natural. 4. If necessary, please use a free translation rather than a literal translation. 5. Please provide the final translation. 6. "Benign prostatic hypertrophy" (BPH) should be translated as "benign prostatic hypertrophy" and not "benign prostatic hypertrophy" to ensure consistent terminology. [Example 3]

[0069] <LLM for solving the National Medical Examination> The student LLM executes an algorithm that generates COT prompts, and the teacher LLM performs a learning data update process for student LLM1 based on the evaluation criteria and prompts. As a large-scale language model (LLM) that accurately answers questions in the national medical licensing examination, it utilizes Chain of Thought (COT) to clarify the thought process, particularly for questions requiring multiple steps, such as diagnostic processes and treatment plans, thereby improving medical validity and answer accuracy. This embodiment describes system operation to improve the performance of generating COT (Chain of Sort) prompts. This embodiment is based on the system configuration (see Figure 3) and processing flow (see Figures 4 and 5). In the system configuration shown in Figures 4 and 5, past questions from the national medical licensing examination are used as input data, and the inference unit 29B in Figure 3 generates COT prompts. The step in which the teacher LLM evaluates these prompts corresponds to S4 and S5 in Figure 4. <System Configuration> Student LLM1: Provide the text of the National Medical Examination as input or as the correct answer label, and generate a "COT prompt" that shows the diagnostic and treatment procedures as a thought process. Student LLM2: Output the final answer using the COT prompt. Teacher LLM: Evaluates the accuracy of the answers, compares the answers generated by Student LLM2 with the correct answer data, and evaluates the quality of the COT prompts and the thought process. Training data: Past national medical licensing examination questions (input data) and correct answer labels for the correct answers and explanations (output data). <Algorithm for student LLM to generate COT prompts> • Input: Question text from the National Medical Examination (e.g., diagnostic reasoning, treatment selection, pathology questions) • Flow of the COT prompt generation algorithm: Classify the problem: The problem statement was analyzed and categorized into "Diagnosis," "Treatment Selection," "Disease Prediction," and "Examination Plan." Question breakdown: The problem statement was divided and the following were extracted: "Symptoms," "Test Results," and "Patient Background." Pick out the necessary information from the problem statement and apply it to the following COT prompt template. COT prompt template: Step 1: Create a list of differential diagnoses based on the chief complaint and symptoms. Step 2: List any additional tests that may be needed. Step 3: Describe the definitive diagnosis and the treatment plan based on it. Step 4: Note any complications or additional precautions. • Example COT prompt (generated based on the problem statement): Example problem: "A 40-year-old male has had a fever and cough for two weeks. A chest X-ray revealed a shadow in the right lung field. What should be the next test performed on this patient?" Example COT prompt (generated result): Step 1: The main complaints are fever and cough, lasting for two weeks. A shadow is observed in the right lung field on X-ray, suggesting a respiratory infection. Step 2: Consider differential diagnoses such as pneumonia, tuberculosis, and lung tumors. Step 3: Perform sputum tests and tuberculin skin tests to rule out tuberculosis. Step 4: If tuberculosis is highly suspected, consider isolation measures. Evaluation criteria data • Evaluation criteria and example scores for Teacher LLM: Evaluate the results using the following scoring system. Validity of the diagnostic process (out of 70 points) Comprehensiveness of the differential diagnosis list Accuracy of prioritization Appropriateness of selected tests and treatments Logicality of the thought process (out of 20 points) Are the steps in order? Is there any contradiction between the content of the previous step and the next step? Application of specialized knowledge (out of 10 points) Is the expression based on medical evidence? • Example of evaluation prompts by a teacher LLM: <Input Data>: A 40-year-old male has had a fever and cough for two weeks. A chest X-ray revealed a shadow in the right lung field. <Output Data>: Sputum tests and tuberculin tests are performed to rule out tuberculosis. <Evaluation Criteria Data>: Step 1: The chief complaint and X-ray findings strongly suggest tuberculosis, so broaden the differential diagnosis. Step 2: Prioritize sputum culture and PCR testing, primarily considering tuberculosis and pneumonia. Step 3: Include information if a CT scan or chest contrast examination is necessary. <Scoring Example>: Diagnostic process: 60 / 70 Logic of thought process: 15 / 20 Application of specialized knowledge: 8 / 10 Overall rating: 83 / 100 <Process for updating student LLM1 learning data> • Flow of updating training data: The teacher LLM evaluates the COT prompt and output generated by student LLM1. Prompts with an evaluation score exceeding a certain value (e.g., 80 points or higher) are tagged as "high-quality data". Highly rated COT prompts are added to the new training dataset, and student LLM1 is retrained. Low-rated prompts are corrected in the teacher LLM and then reused as training data. • Example of updating training data: The student LLM1 is further trained with prompts that have an accuracy rate of 90% or higher. Prompts below 80% will provide feedback on the cause of the score drop and save it as additional data. example: "A PCR test was overlooked in a suspected tuberculosis case." → Add a rule to the next prompt generation: "Always include a description of the PCR test." <Advantages of this system> Improved reproducibility: By utilizing past questions from the national medical licensing examination, highly accurate COT prompts are automatically generated through multiple loops. Ensuring medical validity: Teacher LLMs check validity based on evaluation criteria to reduce diagnostic errors. Ensuring versatility: Applicable not only to the national medical licensing examination, but also to nursing and pharmacist examinations. Implementing this configuration will result in a high-precision LLM that can be applied to various medical fields beyond the national medical licensing examination.

[0070] [Table 2]

[0071] [Table 3] [Example 4]

[0072] <Disease Registry Creation LLM> <System Overview> This system is a large-scale language model (LLM) learning system designed for medical licensing examination preparation and disease registry registration. Utilizing a teacher LLM and two student LLMs, the system supports improved accuracy in medical licensing examination answers, disease summary analysis, and registry registration through a prompt generation, evaluation, and learning flow. It is a comprehensive system that includes standardization of terminology for disease names, findings, and treatments, as well as assessment of the presence and severity of complications. Student LLM1 (COT Prompt Generation): Generates Chain of Sort (COT) prompts that show the thought process and solution step by step based on case data. Student LLM2 (Summary Analysis & Registry Registration): Analyze disease summaries and structure registration data for disease registries. Teacher LLM (Evaluation Model): Evaluates registry data and uses it as training data for feedback. Learning Datasets: Dedicated datasets structured for each exam (e.g., past questions, correct answers, and explanations). For example, the medical licensing exam focuses on "diagnostic procedures," the nursing exam on "care plans," and the pharmacist exam on "pharmacokinetics." <Registry registration targets and registration information> This system is intended for patients diagnosed with COVID-19 and hospitalized at medical institutions within Japan. The information to be registered is as follows: 1) Basic patient information: Age, sex, country of birth, race Epidemiological information regarding infection Underlying medical conditions, medication history, and treatment history. 2) Information regarding hospitalization and treatment: Symptoms, length of hospitalization, treatment methods Blood and imaging test results 3) Information on infectious diseases: COVID-19 test results Other pathogen test results 4) Other considerations include the following: Pregnant women: pregnancy duration, complications during pregnancy, pregnancy outcomes Children: Birth history, vaccination status <Example of a prompt> 1. First student LLM generates COT prompt Example input data: A 50-year-old male presented with fever, fatigue, and shortness of breath, which had started a week prior. PCR testing was positive for SARS-CoV-2, and chest CT revealed ground-glass opacities in both lung fields. Create a registry for this patient. Example of a generated COT prompt: 1. Extract the patient's basic information. 2. List the symptoms and diagnosis upon admission. 3. Open the abbreviation (e.g., PCR for polymerase chain reaction, ARDS for acute respiratory distress syndrome) and write the full name. 4. Extract all synonyms for the drug names used and standardize them to the most common terminology. 5. Identify the presence or absence of complications and extract all applicable items from the following list. - Acute respiratory distress syndrome (ARDS) - Pulmonary thromboembolism (PTE) - Sepsis - Acute kidney injury (AKI) - Multiple organ failure (MODS) - Myocarditis - Cerebral infarction - Deep vein thrombosis (DVT) 6. Identify whether or not the patient has heart disease and extract all applicable items from the following list. - Acute myocardial infarction (AMI) - Heart failure - Arrhythmia - Myocarditis - Angina pectoris 7. Identify any worsening of heart disease or complications during hospitalization, and describe the treatment and course of the disease. 8. Describe the treatment method and length of hospitalization. 9. Record the results of infectious disease tests. 10. Convert to registry data format. Evaluation criteria data In response to the prompt, "Please evaluate the registry registration worker on a 5-point scale from the following three perspectives," the evaluation criteria are: completeness (are there any omissions or omissions in the diagnosis or findings? are all recorded without any shortcomings?), accuracy (are findings and diagnosis names recorded correctly? are there any typos or input errors?), and appropriateness of judgment (is "unknown" appropriately indicated when the findings are unclear or difficult to judge? Are there any entries based on speculation?) and a 5-point scale from 1 to 5 is used. 2. Summary analysis and registry registration by the second student LLM. Example summary text: The patient is a 50-year-old male. He presented with fever and fatigue for the past week, and presented with shortness of breath. A PCR test confirmed he was positive for SARS-CoV-2, and a chest CT scan revealed ground-glass opacities in both lung fields. Due to the possibility of ARDS, he is receiving oxygen therapy and antiviral medication. He is currently under observation. Example of analysis prompt: 1. Identify the age and gender. 2. Extract the chief complaint and findings. 3. Expand abbreviations (e.g., PCR for polymerase chain reaction, ARDS for acute respiratory distress syndrome) and convert them into their full names. 4. If the medication used is listed under a different name, identify the synonym and write the standard name. 5. Identify whether or not complications exist. 6. Identify and record whether or not the patient has heart disease. 7. Describe the treatment methods and hospitalization status. 8. Check and record the results of infectious disease tests. 9. Convert to registry format. Evaluation criteria data: In response to the prompt, "Please evaluate the registry registration worker on a 5-point scale from the following three perspectives," the evaluation criteria are: completeness (are there any omissions or omissions in the diagnosis or findings? are all recorded without any shortcomings?), accuracy (are findings and diagnosis names recorded correctly? are there any typos or input errors?), and appropriateness of judgment (is "unknown" appropriately indicated when the findings are unclear or difficult to judge? Are there any entries based on speculation?) and a 5-point scale from 1 to 5 is used. [Example 5]

[0073] Case report texts and examples of methods for creating correct answer data: First, divide the following case into sentences: pre-consultation text, initial consultation status, post-consultation examinations, examination results, and additional examinations. The patient was a 60-year-old male who presented with fever and dry cough lasting for one week. He was diagnosed with hypoxemia, and a simple chest CT scan revealed multiple lesions on both sides. The patient presented with predominantly peripheral ground-glass opacities and infiltrates. Polymerase chain reaction testing was positive, leading to a diagnosis of SARS-CoV-2 pneumonia. Ten days had passed since the onset of symptoms, and the patient had persistent fever and tachypnea, with oxygen demand at 8 hours. The blood flow rate was increasing from 3 L / min to 6 L / min, but oral administration was possible, so favipiravir was administered. Over the next four hours, respiratory failure worsened further, requiring oxygen administration of up to 8 L / min, and intubation and mechanical ventilation were feared. However, the increased oxygen demand and tachypnea did not worsen further, and fever began to subside 48 hours after administration, and breathing improved. The patient's condition improved and the oxygen dosage was gradually reduced. Three days after administration, a plain chest CT scan revealed an infiltrative shadow with ground-glass opacity. Constrictive changes and bronchiectasis were observed, suggesting a complication of acute respiratory spurt syndrome. Seven days after administration, respiratory... Since the number of breaths normalized and the fever subsided, we judged that the condition was improving and monitored the patient's progress without administering any additional medications. After 14 days, favipiravir was discontinued and oxygen therapy was also stopped. In this case, improvement in progressive respiratory failure was observed after administration of favipiravir, suggesting that the patient was severely ill. This case is considered an important example suggesting that this drug may be effective even in cases of exacerbation, and we report it here. Sentence examples: 1. Pre-consultation document The patient was a 60-year-old male who presented with a chief complaint of fever and dry cough that had lasted for one week. 2. Initial consultation details The patient presented with hypoxemia, and a simple chest CT scan revealed bilateral, multiple, predominantly peripheral ground-glass opacities and infiltrates. 3. Tests after the initial consultation The polymerase chain reaction test came back positive, and the patient was diagnosed with SARS-CoV-2 pneumonia. 4. Test results Ten days had passed since the onset of symptoms, and the patient was experiencing persistent fever and tachypnea, with oxygen demand increasing from 3 L / min to 6 L / min over an 8-hour period. However, oral administration was possible, and favipiravir was given. 5. Additional tests and treatment progress Four hours after administration, respiratory failure worsened further, requiring oxygen administration at a maximum rate of 8 L / min, and intubation and mechanical ventilation were feared. However, the increase in oxygen demand and tachypnea did not worsen further, and 48 hours after administration, the fever began to subside, the respiratory condition improved, and the amount of oxygen administered was gradually reduced. A plain chest CT scan three days after administration revealed infiltrative shadows with ground-glass opacities, as well as contractile changes and bronchiectasis, suggesting a complication of acute respiratory rate syndrome. Seven days after administration, the respiratory rate normalized and the fever subsided, indicating an improvement. Therefore, no further treatment with other medications was initiated, and the patient was monitored. Favipiravir was discontinued after a total of 14 days, and oxygen therapy was also discontinued. This case is considered important because it suggests that the drug may be effective even in severe and worsening cases, as it showed improvement in progressive respiratory failure after administration of favipiravir. Organizing it in this way clarifies the flow of case reports. prompt: Divide the process into the following steps and write a SOAP note for each, along with a title. The assessment should be written in more detail. When filling out the S and O sections, write the STEP name (e.g., before consultation, after consultation) and the situation. The content should be the same for the next step as it was for the previous step. Then, add any newly occurring events. Enclose the added content in brackets [ ]. Patient background information I have an allergy to penicillin. format: S: By that point, it was clearly subjective. O: Test results, etc. A: Assessment (differential diagnosis) P: Examination policy, treatment policy Step example STEP 1; Initial consultation STEP 2: Results of the initial examination STEP 3: After the consultation: Differential diagnoses and tests / treatments performed STEP 4: After the consultation: Differential diagnoses and tests / treatments performed STEP 5: After the consultation: Differential diagnoses and tests / treatments performed STPP6: Post-consultation: Differential diagnoses and tests / treatments performed STEP 7: In front of other hospitals The thought process behind assessment and planning: The assessment description Is the diagnosis correct? When considering differential diagnosis, we will discuss the necessary tests and treatments. Consider the side effects, complications, allergic reactions, and costs of the tests and treatments. As a plan: Choose the actual recommended tests and treatments. Evaluation criteria data Evaluate the following description using the following evaluation criteria. Evaluation criteria data Appropriateness of the treatment process (out of 30 points) Comprehensiveness of the differential diagnosis list Accuracy of prioritization Appropriateness of selected tests and treatments Safety suitability (out of 30 points) Are allergies and side effects taken into consideration during treatment? Logicality of the thought process (out of 10 points) Are the steps in order? Is there any contradiction between the content of the previous step and the next step? Application of specialized knowledge (out of 10 points) Is the expression based on medical evidence? Application of specialized knowledge (out of 20 points) Is the description in accordance with the SOAP format? Is the bracketing correct? Is S subjective and O objective? Is background information such as allergy information properly included in S? knowledge: The coronavirus is currently spreading. COVID-19 is often diagnosed by PCR testing. While there are also high-level COVID-19 tests available, they are not usable at this medical institution. Unified terminology The term COVID-19 will be standardized to COVID-19, which is now called COVID-19. Polymerase chain reaction (PCR) testing will be standardized to the term "PCR test." Text generated by AI Patient background information STEP 1: First consultation S: [A 60-year-old male with a history of penicillin allergy. Prior to the visit: Presented with a fever and dry cough that had lasted for one week.] O: [Before consultation: Symptoms included fever for one week and a dry cough.] A: Given that fever and a dry cough are the main symptoms, differential diagnosis should focus on infectious diseases. Considering the current COVID-19 situation, COVID-19 infection is considered highly probable. Pneumonia and other viral infections should also be considered in the differential diagnosis. P: A chest CT scan was performed to check oxygen saturation. A PCR test was conducted due to suspicion of COVID-19. STEP 2: Results of the initial examination S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Before consultation: Symptoms included fever for one week and a dry cough. [After consultation: Hypoxemia was observed, and chest CT revealed bilateral, multiple, predominantly peripheral ground-glass opacities and infiltrates. PCR testing was performed.] A: The chest CT scan revealed characteristic ground-glass opacities, strongly suggesting viral pneumonia, particularly COVID-19 pneumonia. The presence of hypoxemia indicates a high probability of a severe case. P: Oxygen therapy will be initiated, and the patient will be monitored until the PCR test results are available. STEP 3: After the consultation: Differential diagnoses and tests / treatments performed S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Before consultation: Symptoms included fever and dry cough for one week. After consultation: Hypoxemia was observed, and chest CT revealed bilateral, multiple, predominantly peripheral ground-glass opacities and infiltrates. PCR testing was performed. [Oxygen administration at 3 L / min was started. SpO2 was maintained at 92% with 3 L / min oxygen administration.] A: There is a high probability of COVID-19 pneumonia, and there are concerns about complications such as acute respiratory distress syndrome (ARDS). It is important to monitor for any deterioration in the patient's condition until the PCR test results are confirmed. P: Continue administering oxygen and adjust the oxygen dosage as needed. Once the PCR test results are available, we will decide on a treatment plan. STEP 4: After the consultation: Differential diagnoses and tests / treatments performed S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Before consultation: Symptoms included fever and dry cough for one week. After consultation: Hypoxemia was observed, and chest CT revealed bilateral, multiple, predominantly peripheral ground-glass opacities and infiltrates. PCR testing was performed. SpO2 was maintained at 92% with 3 L / min of oxygen. [[PCR test positive. Diagnosis of SARS-CoV-2 pneumonia based on the positive PCR test.] A: Diagnosed with COVID-19 pneumonia. The patient is experiencing persistent hypoxemia and is considered a severe case. There are concerns about progression of ARDS and worsening respiratory failure. P: Start administering favipiravir. Take immediate action if an increase in oxygen demand is observed. STEP 5: After the consultation: Differential diagnoses and tests / treatments performed S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Post-visit: Hypoxemia, bilateral multiple ground-glass opacities and infiltrates on chest CT. Initially, SpO2 was maintained at 92% with 3 L / min of oxygen. PCR was positive. [Oxygen demand increased to 6 L / min. Favipiravir administration was started.] A: Following the administration of favipiravir, oxygen demand has increased, strongly suggesting progression of ARDS. There is a high probability of respiratory failure progressing, and mechanical ventilation may be required. P: Increase oxygen delivery to 6 L / min and carefully monitor the patient's condition. Prepare for mechanical ventilation. STEP 6: After the consultation: Differential diagnoses and tests / treatments performed S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Post-visit: Hypoxemia, bilateral multiple ground-glass opacities and infiltrates on chest CT. Initially, SpO2 was maintained at 92% with 3 L / min oxygen administration. PCR was positive. Oxygen demand increased to 6 L / min. Favipiravir administration was started [Oxygen increased to 8 L / min. SpO2 maintained at 90%. Intubation and mechanical ventilation were avoided. Fever showed signs of resolution after 48 hours]. A: A tendency towards fever reduction was observed 48 hours after favipiravir administration, and oxygen demand decreased. It is judged that ARDS is improving. P: Continue oxygen administration and monitor the patient until their respiratory condition stabilizes. Adjust the oxygen dosage as needed. STEP 7: Before discharge S: A 60-year-old male with a history of penicillin allergy. Prior to the visit, he presented with a chief complaint of fever and dry cough that had lasted for one week. O: Post-visit: Hypoxemia, bilateral multiple ground-glass opacities and infiltrates on chest CT. Initially, SpO2 was maintained at 92% with 3 L / min oxygen administration. PCR was positive. Oxygen demand increased to 6 L / min. Favipiravir administration was started. Oxygen was increased to 8 L / min. SpO2 was maintained at 90%. Intubation and mechanical ventilation were avoided. After 48 hours, there was a tendency for fever to subside. [Subsequently, oxygen administration was reduced and SpO2 improved.] A: The symptoms have improved, and it has been determined that no further treatment is necessary. P: The plan is to continue administering favipiravir for 14 days and then discontinue oxygen therapy. Example of evaluation criteria Evaluate the following description using the following evaluation criteria. Evaluation criteria data Appropriateness of the treatment process (out of 30 points) Comprehensiveness of the differential diagnosis list Accuracy of prioritization Appropriateness of selected tests and treatments Safety suitability (out of 30 points) Are allergies and side effects taken into consideration during treatment? Logicality of the thought process (out of 10 points) Are the steps in order? Is there any contradiction between the content of the previous step and the next step? Application of specialized knowledge (out of 10 points) Is the expression based on medical evidence? Application of specialized knowledge (out of 20 points) Is the description in accordance with the SOAP format? Is the bracketing correct? Is S subjective and O objective? Is background information such as allergy information properly included in S? Evaluation results The content will be evaluated based on the following criteria. Appropriateness of the treatment process (out of 30 points) Score: 28 / 30 • Comprehensiveness of the differential diagnosis list: While focusing on COVID-19 pneumonia, it also considers ARDS and other infectious diseases, making it highly comprehensive. • Accuracy of prioritization: The state of hypoxemia is emphasized, and oxygen therapy is given high priority. The decision is appropriate considering the current situation of the COVID-19 pandemic. • Appropriateness of selected tests and treatments: The selection of chest CT, PCR test, and oxygen therapy was appropriate and aligned with the diagnostic and treatment process for COVID-19. The administration of favipiravir was also appropriate. • Reason for deduction: There is a slight lack of specific consideration of other differential diagnoses (bacterial pneumonia, influenza, etc.). Safety suitability (out of 30 points) Score: 30 / 30 • Penicillin allergy is documented at every step, and safety has been considered. Favipiravir is not a penicillin-based antibiotic, thus avoiding the risk of allergies. Increased oxygen demands were addressed quickly, successfully avoiding intubation. Logicality of the thought process (out of 10 points) Score: 10 / 10 Each step is described in a logical sequence, without any inconsistencies. Changes in oxygen administration and the effects of favipiravir administration are described in detail. Application of specialized knowledge (out of 10 points) Score: 9 / 10 The findings on the chest CT (ground-glass opacities) and the description of ARDS are accurate. The progression of COVID-19 is also described based on medical evidence. However, it would have been perfect if the detailed treatment strategy for acute respiratory spurt syndrome (ARDS) had been described in more detail. Adherence to SOAP format (20 points total) Score: 18 / 20 The documentation follows the SOAP format, with subjective information correctly placed in the "S" section and objective information in the "O" section. However, the inclusion of penicillin allergy in the "S" section at every step gives the impression of being redundant. • New events enclosed in [] are properly described.

[0074] Furthermore, the above is an example of successfully documenting changes over time by updating the STEP while clearly indicating chronological information, such as by recording clinical information in accordance with SOAP, providing knowledge support, assisting with terminology standardization, creating SOAP formats, and using brackets to distinguish between inherited data and newly added data. At the same time, it enables the use of COT and evaluation criteria to comprehensively evaluate whether the clinical aspects and format are correct, allowing the generating AI to learn from this evaluation, and also incorporates safety evaluations such as allergy assessments.

[0075] <Claim mapping> <Claim 1> <Claim 2> The acquisition unit 29A has the function of collecting input data and external data, and the inference unit 29B generates a COT prompt. The first function and second function of the student LLM described in claim 1 correspond to the inference unit 29B and the function of generating output, respectively. <Claim 3> In step S4 of Figure 4, the teacher LLM evaluates the output of the student LLM based on the evaluation criteria and generates data for iterative improvement, thereby repeatedly improving the quality of the output data. <Claim 5> As shown in Figure 5, human review results are incorporated into the training LLM and used to generate corrective suggestions for low-scoring outputs and for retraining. Combining human feedback or review results eliminates erroneous intermediate inferences. <Claims 6 and 7> The logging unit 29F records scoring information and score management information, and in step S6 of Figure 4, high-scoring data is preferentially used as retraining data. <Claim 8> The supervised large-scale language model evaluates multiple output data sets by referring to evaluation criteria and extracts chain-of-sort prompts and their corresponding output data sets that are judged to have a good evaluation. The quality of the evaluation is determined by the set values ​​of the evaluation criteria or by criteria modified by the supervised LLM. [Industrial applicability]

[0076] The learning system of the present invention is applicable to any industrial field that utilizes large-scale language models. It is useful as a means of improving inference accuracy using COT prompts in areas where high accuracy is required, such as finance (risk assessment, prediction models), healthcare (diagnostic support, registry creation), and education (exam preparation systems). Furthermore, in software development, it is expected to be used in a variety of tasks by adjusting the learning dataset and evaluation criteria, such as the automatic generation of specifications and design documents in natural language. In addition, the COT prompt generation technology of this system is particularly effective when used in situations requiring complex decision-making, providing rapid and highly accurate output. [Explanation of symbols]

[0077] 20: Server 22: Communication Interface (Communication IF) 23: Input / Output Interface (Input / Output IF) 25: Memory 26: Storage 29: Processor 29A: Data Acquisition Unit 29B: Inference Unit 29C: Learning Unit 29D: Evaluation Unit 29E: Processing Unit 29F: Logging Unit 29G: Model Management Unit 29H: Prompt Management Unit

[0078] The program of the present invention can be installed in computer terminals having computer functions such as CPU, memory, and storage, as well as mobile devices such as smartphones, tablets, and wearable devices, digital home appliances such as smart TVs, smart speakers, and smart home appliances, recording media such as USB memory, SD cards, hard disk drives (HDDs) / solid state drives (SSDs), dedicated equipment and terminals such as POS terminals, vending machines, ATMs, and medical equipment, and game consoles (home and portable). When this program is installed in medical equipment, it may be linked with the hospital's electronic health record (EHR / EMR) system.

[0079] [Note C1] A learning method obtained by operating a computer equipped with a processor and memory, A learning method that uses a teacher-large-scale language model and a student-large-scale language model, and has the function of accepting evaluation criterion data and user input data, The first step of the student large-scale language model is to generate a prompt that includes one or more of the following based on the input data: thought process information, chain of sort information, and terminology consistency information. The second step of the student large-scale language model is to generate output data based on the generated prompts and input data, A learning method characterized in that the teacher-large-scale language model performs a learning step in which it updates the parameters related to prompt generation used in the first step of the student-large-scale language model by using the evaluation results from the evaluation step and the input data that produced the evaluation results as learning data. [Note C2] A learning method obtained by operating a computer equipped with a processor and memory, A learning method that uses a teacher-large-scale language model and a student-large-scale language model, and has the function of accepting evaluation criterion data, user input data, and correct answer data, The first step of the student large-scale language model is to generate a prompt that includes one or more of the following based on the input data: thought process information, chain of sort information, and terminology consistency information. The second step of the student large-scale language model is to generate output data based on the generated prompts and input data, A learning method characterized in that the teacher-large-scale language model performs a learning step in which it updates the parameters related to prompt generation used in the first step of the student-large-scale language model by using the evaluation results from the evaluation step and the input data that produced the evaluation results as learning data. [Note C3] [Appendix C2] In the learning method described, A learning method characterized by performing the step of generating retraining data or additional training data for executing a learning process that updates the correct answer data and updates at least one of the first function and the second function, based on the evaluation results by the large-scale teacher language model. [Note C4] In the learning method described in [Appendix C1] or [Appendix C2], The learning method is characterized in that the prompt includes a multi-step intermediate thinking process as chain-of-sort information, and the intermediate thinking process is successively improved based on the evaluation result. [Note C5] In any of the learning methods described in [Appendix C1] to [Appendix C3], The learning method is characterized by performing a step in the learning process of retraining or additionally training the aforementioned large-scale student language model, which involves incorporating human review results as feedback. [Appendix C6] In any of the learning methods described in [Appendix C1] to [Appendix C3], A learning method characterized by recording the evaluation results from the teacher-large-scale language model as scoring information, and using one or more of the high-scoring thought process information, chain-of-sort information, and terminology consistency information to train the student-large-scale language model. [Note C7] In the learning method described in [Appendix C6], The aforementioned large-scale teacher language model includes a function to identify output data with high scoring information and to create additional feedback prompts for generating outputs with high scoring information, Using the aforementioned additional feedback prompts, proposed revisions to the evaluation criteria and chain-of-sort prompts are generated. A learning method characterized by incorporating the generated revised proposals as additional training data for the large-scale student language model. [Note C8] In any of the learning methods described in [Appendix C1] to [Appendix C3], The system comprises training data containing multiple (approximately 1,000 to 10,000) examples of the aforementioned correct answer data, and text data including sample prompts and evaluation criterion data in chain-of-sort format, (a) The first function of the student large-scale language model generates a chain of sort prompts for the input data while referring to the dataset, (b) The second function of the student large-scale language model generates multiple output data using multiple chain-of-sort prompts and the input data, (c) The large-scale trained language model evaluates a plurality of output data by referring to the evaluation axis prompts, and extracts the chain of sort prompts and their output data sets that have received a good evaluation (or are determined to have received a good evaluation), (d) A learning method characterized in that the student large-scale language model incorporates the extraction results as retraining data or additional training data, thereby improving the chain-of-sort prompt generation performance as the first function.

Claims

1. A learning system comprising a teacher language model and a student language model having at least two functions, The learning system has the function of receiving evaluation criterion data and user input data. The first function of the student language model is to generate prompts that include one or more of the following based on the input data: thought process information, chain of sort information, and terminology consistency information. The second function of the student language model includes a function to generate output data based on the generated prompts and input data, The teacher language model includes evaluation means for evaluating the input data and the output data generated by the student language model based on the evaluation criteria data. The learning system is characterized in that the student language model performs a learning process to update the first function of the student language model by using the evaluation results from the evaluation means and the input data that produced the evaluation results as learning data.

2. A learning system comprising a teacher language model and a student language model having at least two functions, The learning system has the function of receiving evaluation criterion data, user input data, and correct answer data. The first function of the student language model is to generate prompts that include one or more of the following based on the input data: thought process information, chain of sort information, and terminology consistency information. The second function of the student language model includes a function to generate output data based on the generated prompts and input data, The teacher language model includes evaluation means for evaluating the input data and the output data generated by the student language model based on the evaluation criteria data. The learning system is characterized in that the student language model performs a learning process to update the first function of the student language model by using the evaluation results from the evaluation means and the input data that produced the evaluation results as learning data.

3. In the learning system described in claim 2, A learning system characterized by comprising a module that generates retraining data or additional training data for executing a learning process that updates the correct answer data and updates at least one of the first function and the second function based on the evaluation results by the teacher language model.

4. In the learning system according to claim 1 or 2, The learning system is characterized in that the prompt includes a chain of sort information consisting of multiple intermediate thinking steps, and the intermediate thinking steps are successively improved based on the evaluation results.

5. In the learning system according to any one of claims 1 to 3, A learning system characterized in that the learning process for relearning or additionally learning the aforementioned student language model incorporates human review results as feedback.

6. In the learning system according to any one of claims 1 to 3, A learning system characterized by recording the evaluation results from the teacher language model as scoring information, and using one or more of the high-scoring thought process information, chain of sort information, and terminology consistency information to train the student language model.

7. In the learning system described in claim 6, The aforementioned training language model includes a function to identify output data with high scoring information and to create additional feedback prompts for generating outputs with high scoring information. Using the aforementioned additional feedback prompts, proposed revisions to the evaluation criteria and chain-of-sort prompts are generated. A learning system characterized by incorporating the generated revised proposals as additional training data for the student language model.

8. In the learning system according to any one of claims 1 to 3, The system comprises training data containing multiple sets of the aforementioned correct answer data, and text data including sample prompts and evaluation criterion data in chain-of-sort format, as a dataset. (a) The first function of the student language model is to generate a chain of sort prompts for the input data while referring to the dataset, (b) The second function of the student language model generates multiple output data using the chain of sort prompts and the input data, (c) The training language model evaluates the multiple output data by referring to the evaluation axis prompts, and extracts the chain of sort prompts and their output data sets that have received a good evaluation. (d) A learning system characterized in that the student language model incorporates the extraction results as retraining data or additional training data, thereby improving the chain-of-sort prompt generation performance as the first function.

Images