Method and system for text generation

Abstract

A system for text generation according to one embodiment may comprise one or more processors that select one summary candidate from among summary candidates obtained by providing, to a machine-learning-based summary generation model, summary-generation content instructing the machine-learning-based summary generation model to summarize a plan, obtain a weight by providing, to a machine-learning-based classification model, a prompt including query content containing a query inputted by a user and input content (220) including the plan and the selected summary candidate, obtain a query vector by providing the prompt to a machine-learning-based encoder model, generate one or more pieces of reference data by using pre-written plans and rule documents defining rules retrieved from a database on the basis of the query vector, calculate, for each token recognizable by a large language model (LLM), a first generation probability based on the prompt, a second generation probability based on the one or more pieces of reference data and the prompt, and a generation probability according to the weight by providing the one or more pieces of reference data and the prompt to the machine-learning-based generation model, and generate output text by sequentially selecting generation tokens from among a plurality of tokens on the basis of generation probabilities individually calculated for the plurality of tokens.

Description

Text generation method and system

[0001] Below, a technique is provided for generating text corresponding to summaries and answers from plans and queries based on a machine learning-based Large Language Model (LLM).

[0002] With the advancement of machine learning-based large language models (LLMs), technologies for generating various contents using large language models are being actively researched.

[0003] In particular, ChatGPT, a conversational AI chatbot based on the Transformer architecture, demonstrated outstanding performance, providing detailed and sophisticated responses to natural language prompts. This enabled the generation of two-query content based on appropriate prompt input. The emergence of ChatGPT has led to a surge in research on two-query content generation using LLM.

[0004] For example, there is research on generating answers to questions using LLMs. In particular, technology for generating answers to questions is rapidly advancing, as LLMs trained with more parameters, such as GPT-4, continue to emerge. However, generating answers by referencing a specific database with less computational effort using LLMs remains a challenge.

[0005] The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the present application.

[0006] A method of text generation performed by one or more processors according to one embodiment comprises the steps of: selecting one of summary candidates obtained by providing summary generation content instructing a summary to a machine learning-based summary generation model; obtaining a weight by providing a prompt including query content including a query input from a user and input content including the plan and the selected summary candidate to a machine learning-based classification model; obtaining a query vector by providing the prompt to a machine learning-based encoder model; generating one or more reference data using rule documents defining rules retrieved from a database based on the query vector and pre-written plans; calculating a first generation probability based on the prompt, a second generation probability based on the one or more reference data and the prompt, and a generation probability according to the weight for each token recognizable in a large language model (LLM) by providing the one or more reference data and the prompt to a machine learning-based generation model; And it may include a step of generating an output text by sequentially selecting a generated token from among the plurality of tokens based on individually calculated generation probabilities for the plurality of tokens.

[0007] In a text generation method performed by one or more processors according to one embodiment, each of the pre-written plans includes a deliberation result indicating the contents of deliberation on each of the pre-written plans, and the step of generating the one or more reference data may include the step of obtaining database vectors by providing the rule documents of the database and the pre-written plans to a machine learning-based encoder model; and the step of generating the one or more reference data by calculating similarities between the obtained database vectors and the query vector.

[0008] In a text generation method performed by one or more processors according to one embodiment, the step of calculating the generation probability may include: calculating the first generation probability by providing the prompt to the generation model; calculating the second generation probability by providing the generated one or more reference data and the prompt to the generation model based on the calculated similarities; and calculating the generation probability by calculating a weighted sum of the first generation probability and the second generation probability based on the weights.

[0009] In a text generation method performed by one or more processors according to one embodiment, the step of selecting one of the summary candidates may include: generating the summary candidates by providing the summary generation content to the summary generation model; obtaining candidate vectors by providing the generated summary candidates and the plan to a machine learning-based encoder model; and selecting one of the summary candidates by providing the obtained candidate vectors to machine learning-based expert models and machine learning-based gate models.

[0010] A method of generating text, performed by one or more processors according to one embodiment, further comprises: updating the database based on at least one or a combination of new medical laws, new hospital regulations, or new pre-written plans; and training at least one or a combination of the classification model, the encoder model, or the generative model based on the updated database, wherein the rule documents may include at least one of medical laws directing laws related to the medical field or hospital regulations directing rules related to the management of hospitals.

[0011] A computing system for generating text according to one embodiment may include one or more processors for selecting one of the summary candidates obtained by providing summary generation content instructing a summary to a machine learning-based summary generation model, obtaining a weight by providing a prompt including query content including a query input from a user and input content including the plan and the selected summary candidate to a machine learning-based classification model, obtaining a query vector by providing the prompt to a machine learning-based encoder model, generating one or more reference data using rule documents defining rules retrieved from a database based on the query vector and pre-written plans, and providing the one or more reference data and the prompt to the machine learning-based generation model, thereby calculating a first generation probability based on the prompt, a second generation probability based on the one or more reference data and the prompt, and a generation probability according to the weight for each token recognizable in a large language model (LLM), and generating an output text by sequentially selecting a generation token from among the plurality of tokens based on the generation probabilities individually calculated for the plurality of tokens.

[0012] In a computing system for generating text according to one embodiment, each of the pre-written plans includes a deliberation result indicating the contents of deliberation on each of the pre-written plans, and the one or more processors can obtain database vectors by providing the rule documents of the database and the pre-written plans to a machine learning-based encoder model, and generate one or more reference data by calculating similarities between the obtained database vectors and the query vector.

[0013] In a computing system for generating text according to one embodiment, the processor may calculate the first generation probability by providing the prompt to the generation model, calculate the second generation probability by providing the generated one or more reference data and the prompt to the generation model based on the calculated similarities, and calculate the generation probability by calculating a weighted sum of the first generation probability and the second generation probability based on the weights.

[0014] In a computing system for generating text according to one embodiment, the one or more processors may generate summary candidates by providing the summary generation content to the summary generation model, obtain candidate vectors by providing the generated summary candidates and the plan to a machine learning-based encoder model, and select one of the summary candidates by providing the obtained candidate vectors to machine learning-based expert models and machine learning-based gate models.

[0015] In a computing system for generating text according to one embodiment, the rule documents include at least one of medical laws directing laws related to the medical field or hospital regulations directing rules related to the management of a hospital, and the one or more processors can update the database based on at least one or a combination of new medical laws, new hospital regulations or new pre-written plans, and train at least one or a combination of the classification model, the encoder model or the generative model based on the updated database.

[0016] FIG. 1 is a diagram schematically illustrating a text generation method according to one embodiment.

[0017] FIG. 2 is a diagram illustrating the generation of a summary and an answer by a text generation system according to one embodiment.

[0018] FIG. 3 is a flowchart illustrating an operation of generating a summary from a plan according to one embodiment.

[0019] Figure 4 is a flowchart illustrating an operation for generating an answer from a prompt according to one embodiment.

[0020] FIG. 5 is a diagram illustrating an operation of generating a summary from a plan according to one embodiment.

[0021] FIG. 6 is a diagram schematically illustrating an operation of generating an answer from a prompt according to one embodiment.

[0022] FIG. 7 is a diagram illustrating in more detail the operation of generating an answer from a prompt according to one embodiment.

[0023] Figure 8 is a diagram illustrating training of a generative model according to one embodiment.

[0024] FIG. 9 is a diagram illustrating the detailed structure and training of an encoder model used to generate a query vector according to one embodiment.

[0025] FIG. 10 is a diagram illustrating an update of a text generation system according to one embodiment.

[0026] FIG. 11 is a block diagram illustrating a text generation system according to one embodiment.

[0027] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0028] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0029] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0030] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0032] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0033] FIG. 1 is a diagram schematically illustrating a text generation method according to one embodiment.

[0034] The text generation system (110) can generate a summary (165) from a plan (155) and perform a method (e.g., a text generation method) for generating an answer to a query of a reviewer (160).

[0035] In step (120), the text generation system (110) can generate a summary (165) from the plan (155). The plan (155) is a document containing content indicating the plan, and may be, for example, a document written by a researcher (150). The text generation system (110) may include a computing system. The computing system may generate a summary (165) for the plan (155) and generate a response to a query from a reviewer (160). The computing system may include a user terminal and a server.

[0036] In step (130), the text generation system (110) can generate a response to a query from a reviewer (160). At this time, the text generation system (110) can reference a database (e.g., a rule database (175), a plan database (180)). The database can include documents related to the review process of the plan (155). The database can include a rule database (175) and a plan database (180). The generation of a response by the text generation system (110) referencing the database is described in detail in FIGS. 4, 6, and 7 below.

[0037] In step (140), the text generation system (110) may update the database based on the results of the deliberation committee (170). For example, the text generation system (110) may receive content (or text, document) (e.g., deliberation result report) indicating the results created by the deliberation committee (170). The text generation system (110) may update the database based on the received document. In addition, the text generation system (110) may update models (e.g., encoder model, generation model, etc.) of the text generation system (110) based on the updated database.

[0038] The text generation system (110) can generate a more accurate summary (165) of the plan (155) and provide it to the reviewer (160) before the review process begins. The text generation system (110) can also provide more accurate answers to specific queries (e.g., queries related to medical laws and regulations, queries related to hospital regulations) regarding the plan (155) based on reference to a database (e.g., a rules database (175), a plan database (180)). The text generation system (110) can update the database based on the results of a new review committee (170) or changes in laws and regulations, and can train models (e.g., encoder models, etc.) of the text generation system (110).

[0039] The text generation system (110) may include a user terminal and a server. The user terminal may include one or more computing devices. The server may include one or more computing devices. The computing devices included in the user terminal and the server may transmit and receive data to and from each other via a network. The text generation system (110) may include a memory in which models (e.g., an encoder model, a generation model, etc.) for performing a text generation method are stored. The text generation system (110) is described in detail with reference to FIG. 11 below.

[0040] A plan (155) is a document scheduled for or currently under review by a committee, and may contain content that directs the proposed plan for research. For example, a plan (155) may be a document proposing a plan for research related to the medical field.

[0041] A reviewer (160) may be a person who reviews a plan (155) during the review process. The reviewer (160) may be a member of the review committee (170). The text generation system (110) may receive a query for the individual review process (161) from the reviewer (160). The text generation system (110) may generate and provide a response in response to the query from the reviewer (160).

[0042] The summary (165) may include a document summarizing the contents of the plan (155). The reviewer (160) can more easily understand the plan (155) by referring to the summary (165) during the review process.

[0043] The review committee (170) may be an organization that includes reviewers (160) to determine whether to approve the plan (155). When the review committee (170) determines whether to approve the plan (155), the review result containing the decision may be entered into the text generation system (110). The entered review result may be stored in a database (e.g., a plan database (180)) along with the reviewed plan (155).

[0044] In step (140), the text generation system (110) can update the database. The text generation system (110) can train models of the text generation system (110) based on the updated database. The text generation system (110) can update the plan database (180) by storing the reviewed plan (155) and the review results in the plan database (180) in response to the resolution of the review committee (170). In addition, the text generation system (110) can update the text generation system (110) by training models of the text generation system (110) based on the updated plan database (180). The updating of the database and models according to step (140) is described in detail in FIG. 11 below.

[0045] The rule database (175) may be a database containing rule documents. Rule documents may include documents representing rules. For example, rule documents may include medical laws and hospital regulations. Medical laws may include documents that dictate laws related to the medical field. Medical laws may specify matters necessary for national healthcare (e.g., the authority and responsibilities of medical professionals). Hospital regulations may include documents that dictate rules related to the operation and management of hospitals. For example, hospital regulations may specify the authority and responsibilities of hospital personnel (e.g., employees). Although not illustrated in FIG. 1, the text generation system (110) may update the rule database (175) in response to changes in medical laws or hospital regulations. Furthermore, the text generation system (110) may train models thereof based on the updated rule database (175).

[0046] The plan database (180) may be a database containing pre-written plans and review results for the pre-written plans. The pre-written plans may include plans (155) that have already been reviewed by the review committee (170). The review results may include documents indicating the content of the review of the pre-written plans.

[0047] FIG. 2 is a diagram illustrating the generation of a summary and an answer by a text generation system according to one embodiment.

[0048] In step (120), the text generation system (110) may generate a summary (165) from the plan (155) and provide it to the user (260). For example, the text generation system (110) may receive the plan (155) from the user (260), and in response, the text generation system (110) may output the generated summary (165) to provide the summary (165) to the user (260). The user (260) may be a person who uses the text generation system (110). For example, the user (260) may include a reviewer (160).

[0049] In step (130), the text generation system (110) can generate and provide an answer to a query of a user (260) by referring to the plan (155), the summary (165), and the database (240). For example, the user (260) can input a prompt (230) including query content (210), input content (220), and the database (240) into the text generation system (110), and the text generation system (110) can provide an answer to the user (260) by outputting output text (250) in response thereto. The query content (210) can include content representing a query that the user (260) inputs into the text generation system (110). The content can include text, images, and audio. The input content (220) can be content related to the query content (210). For example, the input content (220) may include at least one of a plan (155) and a summary (165). The input content (220) may include not only the plan (155) and the summary (165), but also other documents or contents related to the query content (210) input from a user (260) or stored in the text generation system (110). The output text (250) may include text indicating an answer corresponding to the query content (210). The prompt (230) may include at least one of the query content (210) and the input content (220). In addition, the database (240) may be input into the text generation system (110) in advance. In this case, since the operation regarding the database (240) is processed in advance, the text generation system (110) can output the output text (250) in response to the input of the prompt (230) more quickly. Additionally, in this case, the database (240) or a portion thereof may be stored in the text generation system (110).

[0050] Although step (120) and step (130) are illustrated as being performed in conjunction with each other in FIG. 2, this is not a limitation, and step (120) and step (130) may be performed independently of each other. For example, the input content (220) of step (130) may be separate content that does not include the plan (155) and summary (165) of step (120).

[0051] FIG. 3 is a flowchart illustrating an operation of generating a summary from a plan according to one embodiment.

[0052] In step (120), the text generation system (110) can generate summary candidates from the plan (155) and generate a summary (165) by selecting one of the generated summary candidates. The summary candidate is a document summarizing the contents of the plan (155) generated from the summary generation model, and may be a document that is a candidate to be generated as the summary (165). The summary candidate can be output as the summary (165) by being selected by the text generation system (110). The summary generation model can include a machine learning-based neural network for summarizing the plan (155). For example, the summary generation model can include a large-scale language model based on a pre-trained Transformer Architecture (e.g., Generative Pre-trained Transformer 3 (GPT-3)).

[0053] In step (310), the text generation system (110) can generate summary candidates by providing summary generation content including the plan (155) to the summary generation model. At this time, the summary generation model can output summary candidates as output values ​​in response to input that includes the summary generation content as input values. The summary generation content can include content that instructs the plan (155) to be summarized.

[0054] In step (320), the text generation system (110) can obtain candidate vectors by providing the generated summary candidates and the plan (155) to the encoder model. At this time, each of the summary candidates and the plan (155) can be provided in combination. In addition, the text generation system (110) can select one of the summary candidates by providing the generated candidate vectors to expert models and gate models. The encoder model can include a machine learning-based neural network for obtaining an embedding vector corresponding to the content from the content. For example, the encoder model can include a pre-trained large-scale language model based on a transformer architecture (e.g., Bidirectional Encoder Representations from Transformers (BERT), A Robustly Optimized BERT Pretraining Approach (RoBERTa)). The embedding vector can include a vector corresponding to the content. The candidate vector can include an output value of the encoder model obtained by providing the plan (155) and the summary candidate to the encoder model. For example, the candidate vector may include an embedding vector corresponding to the combined plan (155) and summary candidate. In addition, the candidate vector may include not only the output value of the encoder model obtained by providing the plan (155) and summary candidate to the encoder model, but also the value obtained by transforming the output value (e.g., feature extraction using a neural network). The expert model may include a machine learning-based neural network configured to evaluate the candidate vector from the input of the candidate vectors. The gate model may include a machine learning-based neural network configured to generate weights between the output values ​​of the expert models. The generation of the summary (165) according to steps (310) and (320) is described in detail in FIG. 5 below.

[0055] Figure 4 is a flowchart illustrating an operation for generating an answer from a prompt according to one embodiment.

[0056] In step (410), the text generation system (110) may obtain weights by providing the prompt (230) to a classification model. The classification model may include a machine learning-based neural network (e.g., RoBERT) for obtaining weights from the prompt (230). The classification model may be configured to output a value (e.g., a weight) in response to an input of the prompt (230) indicating whether the query content (210) included in the prompt (230) can be answered solely by referring to the input content (220) included in the prompt (230). The weight may include, for example, a weight value to be applied to an output value (e.g., a second generation probability) generated based on a document retrieved from the database (240) in step (460). For example, the weight may have a value greater than or equal to 0 and less than or equal to 1.

[0057] In step (420), the text generation system (110) can obtain a query vector by providing a prompt (230) to the encoder model. The query vector may include a vector output by inputting the prompt (230) to the encoder model. The query vector may be an embedding vector corresponding to the prompt (230).

[0058] In step (430), the text generation system (110) may generate one or more reference data by calculating similarities between database vectors and query vectors. The database vectors may include vectors obtained by providing each of rule documents, pre-written plans, and deliberation results contained in the database (240) to the encoder model. The similarity may be a value indicating the degree to which two documents or two vectors are similar. For example, the similarity may include cosine similarity, inner product, Euclidean distance, etc. The reference data may represent the text of a document retrieved from the database (240). The retrieved document may be text included in a document retrieved based on the similarity with the prompt (230) among a plurality of documents contained in the database (240). The retrieved document may be a predetermined number of documents. For example, the retrieved document may be retrieved by selecting, in descending order, a document corresponding to a database vector having a high similarity with the query vector among the documents in the database (240). The similarity between the prompt (230) and each document can be calculated as the similarity between the query vector extracted from the prompt (230) and the embedding vector (e.g., database vector) extracted from the corresponding document.

[0059] In step (440), the text generation system (110) may calculate a first generation probability by providing the prompt (230) to the generation model. The generation model may include a machine learning-based neural network for obtaining probabilities (e.g., first generation probability, second generation probability) for each token recognizable in the LLM from the content. For example, the generation model may include a pre-trained large-scale language model (e.g., GPT-3) based on a transformer architecture. A token may include a unit that constitutes a text. The first generation probability is the probability that a token is generated from the prompt (230), and may include the probability of a token being recognizable in the LLM generated without referencing the database (240). For example, the first generation probability may be a vector having a size corresponding to the total number of tokens recognizable in the LLM. For example, a token recognizable in the LLM may be a token included in the vocabulary of a tokenizer used by the LLM.

[0060] In step (450), the text generation system (110) may calculate a second generation probability by providing one or more reference data and a prompt (230) to the generation model. At this time, each of the one or more reference data and the prompt (230) may be combined and then provided to the generation model. The second generation probability is a probability that a token is generated from the prompt (230) and the one or more reference data, and may include a probability for a token that is recognizable in an LLM generated by referencing a database (240). The second generation probability may be calculated by multiplying an output value obtained by inputting the prompt (230) and the reference data into the generation model by a normalized similarity of the corresponding reference data, and summing the result for all retrieved reference data. The normalized similarity may include a normalized value for the similarities calculated between the query vector and the database vectors. For example, the second generation probability may be a vector having a size corresponding to the number of tokens that are recognizable in the LLM.

[0061] In step (460), the text generation system (110) can calculate the generation probability by calculating the weighted sum of the first generation probability and the second generation probability based on the weights. For example, the text generation system (110) can calculate the first generation probability as 1- The value multiplied by the second generation probability The generation probability can be calculated by summing the values ​​multiplied by . The generation probability can include the probability of a recognizable token in the LLM generated by referencing the database (240) only as much as necessary. For example, the text generation system (110) can calculate the generation probability according to the following mathematical expression 1.

[0062] [Mathematical Formula 1]

[0063]

[0064] In the above mathematical expression 1, x may represent query content (210) and d may represent input content (220). represents the i-th token to be generated in the output text (250) (e.g., the i-th generated token). can represent the tokens generated from the first to the i-1th (e.g., the 1st to the i-1th generated tokens). represents a weight, and D' can represent a set of generated reference data. zi can represent reference data included in D' and to be referenced in generating the ith token. is x, d, From The probability that x, d, will be selected (e.g. x, d, represents the output value of the generative model corresponding to the input, i.e., the first probability) can represent the probability that zi is selected from the prompt (230), i.e., the normalized similarity (e.g., the cosine similarity between the prompt (230) and zi). is x, d, zi, From The probability that is selected (e.g. x, d, zi, ) represents the output value of the generative model corresponding to the input, can represent the second probability for the token to be generated i-th. x, d, calculated considering the weights From The probability of being generated, i.e., the generation probability, can be represented. i can represent an integer greater than or equal to 2. As shown in the mathematical expression 1 above, the generation probability for each token can be calculated as a weighted sum of the first generation probability and the second generation probability for each token based on the weight.

[0065] In step (470), the text generation system (110) may sequentially select generated tokens from among a plurality of tokens based on individually calculated generation probabilities for a plurality of tokens recognizable in the LLM, thereby generating output text (250). The generated tokens may include tokens generated by being selected from among a plurality of tokens recognizable in the LLM. The output text (250) may be text including the generated tokens. The output text (250) may be text representing an answer corresponding to a query of a user (260).

[0066] For example, the text generation system (110) may select the token with the highest generation probability among the calculated generation probabilities for each token as the generated token. In this case, the output text (250) may be obtained by sequentially selecting the generated tokens.

[0067] The text generation system (110) can generate generated tokens not only by the above method but also by various methods such as top-k search or beam search. For example, beam search can be a method of obtaining candidates for the output text (250) by sequentially generating a preset number of generated tokens while maintaining a preset number of candidates, and selecting one of the generated candidates for the output text (250). When using the method of generating candidates for the output text (250) and selecting one of them, the output text (250) can be generated by selecting the candidate for the output text (250) that has the highest probability of selecting the candidate for the output text (250), which is expressed by the product of the generation probabilities for each token of the candidates for the output text (250). In this case, the probability of selecting the candidate for the output text (250) can be expressed as in the following mathematical expression 2.

[0068] [Equation 2]

[0069]

[0070] In the above mathematical expression 2, y may represent a candidate of the output text (250), and x may represent the query content (210). D' may represent a set of reference data selected from the query vector. d may represent the input content (220), and z may represent reference data included in D'. yi may represent the ith generated token. can represent the generated tokens from the first to the i-1th. can represent the first generation probability of the i-th generated token, can represent the second generation probability of the i-th generated token. can represent a weight, and L can represent the number of generated tokens that the output text (250) candidate has. can represent the probability of selecting a candidate for the output text (250). i can be an integer greater than or equal to 2. The generation of the output text (250) in response to the input of the prompt (230) is described in detail in FIGS. 6 and 7 below.

[0071] FIG. 5 is a diagram illustrating an operation of generating a summary from a plan according to one embodiment.

[0072] In step (510), the text generation system (110) can generate multiple summary candidates by providing summary generation content that instructs the summary generation model (530) to summarize the plan (155). The resulting summary candidates can be generated in a predetermined number, M, as illustrated in FIG. 5. The generated summary candidates can be combined with the plan (155).

[0073] In step (520), the text generation system (110) can obtain candidate vectors (550) by providing the combined plan (155) and summary candidates to the encoder model (540). At this time, the number of candidate vectors (550) corresponding to each of the M summary candidates generated may also be M. The text generation system (110) can select one of the summary candidates by providing each of the obtained candidate vectors (550) to a corresponding expert model (560) and a gate model (570). At this time, the number of expert models (560) corresponding to each of the M candidate vectors (550) may also be M. The expert model (560) may be configured to output a score, which is a value obtained by evaluating the candidate vector (550), which is an input value. The gate model (570) may be configured to output weighted values ​​to be assigned to the scores of the expert models (560). For example, the gate model (570) may be configured to output a one-dimensional vector having a size of M. The text generation system (110) may select a summary candidate corresponding to the candidate vector (550) having the highest value among the scores to which the weighted value of the gate model (570) is applied. The selected summary candidate may be provided to the reviewer (160) as a summary (165). The reviewer (160) may use the summary (165) provided in the deliberation process to more easily understand the plan (155). For reference, the summary generation model (530) and the encoder model (540) are pre-trained models and may be used without additional training. The parameters (e.g., connection weights) of the summary generation model (530) and the encoder model (540) may be fixed. However, the expert models (560) and the gate models (570) may be additionally trained. The parameters of the expert models (560) and the gate models (570) can be updated. In this case, the text generation system (110) can generate a summary (165) from the training plan (155) through steps (510 and 520).Thereafter, the text generation system (110) can train expert models (560) and gate models (570) to reduce the generated objective function by comparing the generated summary (165) with the actual summary (165) corresponding to the training plan (155) and evaluating it based on various criteria.

[0074] FIG. 6 is a diagram schematically illustrating an operation of generating an answer from a prompt according to one embodiment.

[0075] A text generation system (110) according to one embodiment can obtain output weights by inputting a prompt (230) including at least one of a query content (210), a summary (165), and a plan (155) of a reviewer (160) into a classification model (610).

[0076] In addition, the text generation system (110) can obtain the top k reference data (630) by providing the rule database (175), the plan database (180), and the prompt (230) to the encoder model (640). At this time, the database (240) can be input in advance, and the documents of the database (240) can be input individually. Based on the similarities between the database vectors output by inputting the database (240) to the encoder model (640) and the query vectors output by inputting the prompt (230) to the encoder model (640), the top k rule documents, pre-written plans, and deliberation results can be searched in descending order from the highest similarity. The aforementioned searched documents can be input to the generation model (620) as the top k reference data (630).

[0077] The top k reference data (630) can be combined with the query content (210) and the input content (220) and inputted into the generation model (620). The text generation system (110) can obtain the output text (250) by providing the prompt (230) and the top k reference data (630) to the generation model (620) based on the weight.

[0078] For reference, although FIG. 6 depicts that the database (240) and the prompt (230) are input to the same encoder model (640), this is not limiting, and the encoder model (640) into which the database (240) is input and the encoder model (640) into which the prompt (230) is input may be different. In addition, although FIG. 6 depicts that the prompt (230) includes all of the query content (210), the summary (165), and the plan (155), this is not limiting, and the prompt (230) may include one or both of the query content (210), the summary (165), and the plan (155).

[0079] FIG. 7 is a diagram illustrating in more detail the operation of generating an answer from a prompt according to one embodiment.

[0080] A text generation system (110) according to one embodiment can obtain an output value by inputting a prompt (230) including query content (210) and input content (220) into a classification model (610). At this time, the output value of the classification model (610) can be a value (e.g., a weight) greater than or equal to 0 and less than or equal to 1.

[0081] According to one embodiment, the text generation system (110) can obtain database vectors (720) as output values ​​by inputting a database (240) including a rule database (175) and a plan database (180) into an encoder model (640). In addition, the text generation system (110) can obtain a query vector (710) as an output value by inputting a prompt (230) into the encoder model (640). For example, the text generation system (110) can obtain a query vector (710) by inputting query content (210) into the encoder model (640).

[0082] Thereafter, the text generation system (110) can obtain k reference data (730) by calculating the similarity between each of the query vector (710) and the database vectors (720). For example, the text generation system (110) can calculate the cosine similarity between each of the query vector (710) and the database vectors (720), and obtain k database vectors (720) in descending order of the calculated similarities from the highest similarity. In addition, the text generation system (110) can obtain k reference data (730), which are documents or texts of the database (240) corresponding to the obtained k database vectors (720). The above-described similarity calculation is an example, and the similarity can be calculated by various other methods in addition to the cosine similarity.

[0083] As described above, the text generation system (110) can obtain the first probability as an output value by inputting the prompt (230) into the generation model (620). In addition, the text generation system (110) can obtain the output value by providing k reference data (730) and the prompt (230) to the generation model (620), and can calculate the second probability from the output value and the calculated similarity described above in the composer (740). The composer (740) can represent a post-processing operation of the output values ​​of the generation model (620). The text generation system (110) can calculate the generation probability by calculating a weighted sum of the first probability and the second probability based on the weight in the composer (740). The text generation system (110) can generate the output text (250) by sequentially selecting a generation token from among the tokens that the LLM can recognize based on the generation probability for each token that the calculated LLM can recognize. For example, the output text (250) can be generated by sequentially selecting tokens with the highest generation probability. In addition, for example, a predetermined number of candidates for the output text (250) can be obtained by sequentially selecting a predetermined number of tokens in descending order from the highest generation probability by beam search, and the output text (250) can be generated by selecting a candidate for the output text (250) with the highest probability of being selected, which is expressed by the product of the generation probabilities of each token of the obtained candidates (see mathematical equation 2). The above-described method is an example, and the output text (250) can be generated from the generation probability calculated by various other methods. The text generation system (110) can provide an answer to a query to a user (260) by outputting the generated output text (250).

[0084] Figure 8 is a diagram illustrating training of a generative model according to one embodiment.

[0085] The generative model (620) may be a pre-trained, large-scale language model, but may have difficulty generating accurate outputs in response to prompts (230) related to specific topics (e.g., medical research, medical laws, etc.). Therefore, the accuracy of the output generated in response to input of prompts (230) related to specific topics can be improved by performing additional training (e.g., fine-tuning) using a dataset related to specific topics (e.g., training prompts, training answer content (820), etc.).

[0086] The text generation system (110) can train the generation model (620) based on a training prompt and a training answer content (820) including at least one of a training query content (810), a training summary (830), and a training plan (840). The training query content (810) can include a query content (210) for training. The training summary (830) can include a summary (165) that summarizes a training plan (840) corresponding to the training query content (810). The training plan (840) can include a plan (155) corresponding to the training query content (810). The training answer content (820) can be a content that represents an answer to the training query content (810) as an answer for training. The training prompt can include at least one of the training query content (810), the training summary (830), and the training plan (840).

[0087] For example, the text generation system (110) can obtain a temporary output text (850) as an output value by inputting a training prompt into the generation model (620). The text generation system (110) can calculate a first objective function between the training answer content (820) and the temporary output text (850). The first objective function can include an objective function for training the generation model (620). The first objective function can include, for example, cross-entropy as illustrated in FIG. 8. The generation model (620) can be trained to reduce the first objective function.

[0088] FIG. 9 is a diagram illustrating the detailed structure and training of an encoder model used to generate a query vector according to one embodiment.

[0089] While the encoder model (910) may be a pre-trained, large-scale language model, it may struggle to generate accurate outputs in response to prompts (230) related to specific topics (e.g., medical research, medical legislation, etc.). Therefore, by additionally training the encoder model using a dataset (e.g., training prompts) related to specific topics, the accuracy of output generation in response to inputs of prompts (230) related to specific topics can be improved.

[0090] An encoder model (640) according to one embodiment may include at least one of a first encoder model (910) and a second encoder model (920). The first encoder model (910) may obtain a query vector (710) from a prompt (230) including query content (210). The second encoder model (920) may obtain a document vector from a summary (165) and a plan (155). Here, the document vector may be included in the database vectors (720) when calculating the similarity of FIG. 6. In addition, the second encoder model (920) may generate the database vectors (720) from the database (240).

[0091] According to one embodiment, the text generation system (110) can obtain a temporary output query vector (930) as an output value by inputting the training query content (810) into the first encoder model (910). In addition, the text generation system (110) can obtain a temporary output document vector (940) as an output value by inputting the training summary (830) and the training plan (840) into the second encoder model (920), and can obtain database vectors (720) as an output value by inputting the database (240) including the rule database (175) and the plan database (180) into the second encoder model (920). The text generation system (110) can calculate a second objective function based on the temporary output query vector (930), the temporary output document vector (940), and the database vectors (720). The second objective function can include an objective function for training the first encoder model (910). For example, the text generation system (110) can calculate an n-pair loss as a second objective function, where the temporary output document vector (940) is a positive vector vp and each database vector (720) is a negative vector vn. The positive vector can represent a vector that is highly related to the temporary output query vector (930). The negative vector can represent a vector that is low in relatedness to the temporary output query vector (930). In this case, the n-pair loss can be expressed as in the following mathematical expression 3.

[0092] [Equation 3]

[0093]

[0094] In the above mathematical expression 3, L represents an n-pair loss, and vq may represent a temporary output query vector (930). vp may represent a positive vector, and vn may represent a negative vector. The first encoder model (910) may be trained to reduce the second objective function.

[0095] The generative model (620) and the encoder model (910) can be trained separately or together. For example, the text generation system (110) can train the classification model (610), the encoder model (910), and the generative model (620) together based on the training prompt, the training answer content (820), and the database vectors (720). The third objective function used in this case can be calculated based on the first objective function and the second objective function, for example. An exemplary third objective function can be expressed as shown in the following mathematical expression 4.

[0096] [Equation 4]

[0097]

[0098] In the above mathematical formula 4 can represent the third objective function can represent the first objective function. In addition, may represent a second objective function. x may represent query content (210) and y may represent output text (250). d may represent input content (220) and D' may represent a set of related data.

[0099] As will be described later in FIG. 10, the classification model (610), the first encoder model (910), and the generation model (620) can be trained to reduce the third objective function based on the new database (240) in response to updates to the database (240), etc. For reference, in the text generation method, the second encoder model (920) can be used without additional training. That is, the parameters of the second encoder model (920) can be fixed. However, the first encoder model (910) can be additionally trained. That is, the parameters of the first encoder model (910) can be updated.

[0100] FIG. 10 is a diagram illustrating an update of a text generation system according to one embodiment.

[0101] A text generation system (110) according to one embodiment can update a database (240) based on at least one of new medical laws, new hospital regulations, new pre-written plans, and new review results.

[0102] For example, the text generation system (110) can update the plan database (180) based on the newly determined review committee dataset (1020). The newly determined review committee dataset (1020) can include new pre-written plans and new review results. The new pre-written plan can include a pre-written plan that has been newly decided by the review committee (170). The new review results can include a review results document indicating the content of the review of the new pre-written plan. The text generation system (110) can update the plan database (180) by adding the newly determined review committee dataset (1020) to the plan database (180). For example, the text generation system (110) can update the plan database (180) in response to a predetermined number or more of pairs of new pre-written plans and new review results being stored in the newly determined review committee dataset (1020). After the update, the newly determined review committee dataset (1020) may be reset (e.g., pre-written plans and review results used in the update are deleted from the newly determined review committee dataset (1020)).

[0103] For example, the text generation system (110) can update the rule database (175) based on a modified rule database (1010) in response to changes in hospital regulations or medical laws. The modified rule database (1010) can include a database (240) containing new hospital regulations and new medical laws. The text generation system (110) can update the rule database (175) by modifying the contents of the existing rule database (175) with the contents of the modified rule database (1010).

[0104] In one embodiment, the text generation system (110) can train models of the text generation system (110) (e.g., trainable models of the text generation system (110), such as the classification model (610), the generation model (620), and the first encoder model (910)) in response to an update criterion being met. The update criterion can include conditions that must be met in order to train the models of the text generation system (110). For example, the update criterion can include an update of the database (240) in response to a change in hospital regulations or medical laws, or a predetermined number or more of documents being stored in a newly determined review committee dataset (1020). That is, the text generation system (110) can train the models of the text generation system (110) based on the updated database (240) in response to the database (240) being updated. For example, the text generation system (110) may train at least one or a combination of the classification model (610), the generation model (620), or the first encoder model (910) in response to an update of the database (240). Additionally, the update criteria may include a training instruction from a user (260). For example, if the performance of the text generation system (110) is determined to have deteriorated (e.g., a decrease in the accuracy of the output text (250) for a query), the user (260) (e.g., a reviewer (160), a review committee (170)) may train the models of the text generation system (110) by instructing the text generation system (110) to train. Additionally, the update criteria may include various predetermined criteria.

[0105] FIG. 11 is a block diagram illustrating a text generation system according to one embodiment.

[0106] As described above, the text generation system (110) may include a user terminal (1110) and a server (1130).

[0107] The user terminal (1110) may include one or more computing devices (e.g., desktop computers, laptops, mobile phones, and tablet PCs) that input or output data. The computing devices may include electronic devices that perform logical operations and / or storage. The computing devices may include one or more processors. The computing devices may include one or more memories. The user terminal (1110) may receive data (e.g., plan (155), query content (210), etc.) from a user (260). The user terminal (1110) may transmit the received data to the server (1130). The user terminal (1110) may receive data (e.g., summary (165), output text (250), etc.) transmitted by the server (1130). The user terminal (1110) may output the received data. Transmission and reception of data between the user terminal (1110) and the server (1130) may be performed via a network (1120).

[0108] The network (1120) may include communication devices and transmission lines (e.g., Internet lines) used to interconnect devices (e.g., user terminals (1110) and servers (1130)) that are located apart from each other so that data can be exchanged between these devices. The network (1120) may be used by one or more computing devices included in the user terminals (1110) and servers (1130) to transmit and receive data to and from other computing devices.

[0109] The server (1130) may include one or more computing devices that provide services (e.g., providing a summary (165), providing an output text (250)) to a client (e.g., a user terminal (1110)). The server (1130) may include one or more memories that store data required for a text generation method. For example, the server (1130) may include one or more memories that store models used in the text generation method (e.g., expert models (560), generation models (620), encoder models, etc.). In addition, the server (1130) may include one or more memories that store commands used in the text generation method. In this case, the aforementioned data may be stored together in one memory or may be stored in two or more memories. The aforementioned memories may be included in one computing device or may be included in two or more computing devices. In addition, the server (1130) may include one or more processors that perform the text generation method. At this time, the text generation method may be performed solely by one processor included in the server (1130), or two or more processors included in the server (1130) may perform each detailed step of the text generation method separately.

[0110] Although the input and output through the user terminal (1110), the storage of data through the server (1130), and the execution of the text generation method have been described, they are not limited thereto. That is, one or more memories of the user terminal (1110) may store data required for the text generation method. One or more processors of the user terminal (1110) may perform the text generation method. In addition, one or more memories included in the user terminal (1110) and one or more memories included in the server (1130) may separately store data required for the execution of the text generation method. One or more processors included in the user terminal (1110) and one or more processors included in the server (1130) may separately execute each detailed step of the text generation method.

[0111] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0112] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0113] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0114] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0115] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0116] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A text generation method performed by one or more processors, A step of selecting one of the summary candidates obtained by providing summary generation content instructing a machine learning-based summary generation model to summarize the plan; A step of obtaining weights by providing a prompt including query content including a query input from a user and input content including the plan and the selected summary candidate to a machine learning-based classification model; A step of obtaining a query vector by providing the above prompt to a machine learning-based encoder model; A step of generating one or more reference data using rule documents and pre-written plans defining rules retrieved from a database based on the above query vector; By providing the one or more reference data and the prompt to a machine learning-based generative model, a step of calculating a first generation probability based on the prompt, a second generation probability based on the one or more reference data and the prompt, and a generation probability according to the weight for each recognizable token in a large language model (LLM); and A step of generating an output text by sequentially selecting a generated token from among a plurality of tokens based on individually calculated generation probabilities for the plurality of tokens. Including method.

2. In paragraph 1, Each of the above pre-written plans includes a deliberation result document indicating the contents of the deliberation on each of the above pre-written plans, The step of generating one or more of the above reference data comprises: A step of obtaining database vectors by providing the rule documents and the pre-written plans of the database to a machine learning-based encoder model; and A step of generating one or more reference data by calculating similarities between the acquired database vectors and the query vector. method.

3. In paragraph 2, The step of calculating the above generation probability is: A step of calculating the first generation probability by providing the above prompt to the above generation model; A step of calculating the second generation probability by providing the generated one or more reference data and the prompt to the generation model based on the calculated similarities; and A step of calculating the generation probability by calculating a weighted sum of the first generation probability and the second generation probability based on the weights. method.

4. In paragraph 1, The step of selecting one of the above summary candidates is: A step of generating the summary candidates by providing the above summary generation content to the above summary generation model; A step of obtaining candidate vectors by providing the above-generated summary candidates and the plan to a machine learning-based encoder model; and A step of selecting one of the summary candidates by providing the obtained candidate vectors to machine learning-based expert models and machine learning-based gate models. method.

5. In paragraph 2, The above regulatory documents include at least one of the medical regulations that direct laws related to the medical field or the hospital regulations that direct rules related to the management of hospitals. Updating said database based on at least one or a combination of new medical legislation, new hospital regulations, or new pre-written plans; and A step of training at least one or a combination of the classification model, the encoder model, or the generative model based on the updated database. Including more method.

6. A computer-readable recording medium having recorded thereon a program for executing the method of paragraph 1.

7. In a computing system for generating text, The present invention comprises one or more processors for selecting one of the summary candidates obtained by providing summary generation content instructing to summarize a plan to a machine learning-based summary generation model, obtaining a weight by providing a prompt including query content including a query input from a user and input content including the plan and the selected summary candidate to a machine learning-based classification model, obtaining a query vector by providing the prompt to a machine learning-based encoder model, generating one or more reference data using rule documents defining rules searched from a database based on the query vector and pre-written plans, and providing the one or more reference data and the prompt to a machine learning-based generation model, thereby calculating a first generation probability based on the prompt for each token recognizable in a large language model (LLM), a second generation probability based on the one or more reference data and the prompt, and a generation probability according to the weight, and generating an output text by sequentially selecting a generation token from among the plurality of tokens based on the generation probabilities individually calculated for the plurality of tokens. Computing system.

8. In paragraph 7, Each of the above pre-written plans includes a deliberation result document indicating the contents of the deliberation on each of the above pre-written plans, The one or more processors obtain database vectors by providing the rule documents of the database and the pre-written plans to a machine learning-based encoder model, and generate one or more reference data by calculating similarities between the obtained database vectors and the query vector. Computing system.

9. In paragraph 8, The processor calculates the first generation probability by providing the prompt to the generation model, calculates the second generation probability by providing the generated one or more reference data and the prompt to the generation model based on the calculated similarities, and calculates the generation probability by calculating a weighted sum of the first generation probability and the second generation probability based on the weight. Computing system.

10. In paragraph 7, The one or more processors generate the summary candidates by providing the summary generation content to the summary generation model, obtain candidate vectors by providing the generated summary candidates and the plan to a machine learning-based encoder model, and select one of the summary candidates by providing the obtained candidate vectors to machine learning-based expert models and machine learning-based gate models. Computing system.

11. In paragraph 8, The above regulatory documents include at least one of the medical regulations that direct laws related to the medical field or the hospital regulations that direct rules related to the management of hospitals. The one or more processors update the database based on at least one or a combination of new medical laws, new hospital regulations, or new pre-written plans, and train at least one or a combination of the classification model, the encoder model, or the generative model based on the updated database. Computing system.

Images