Method and apparatus for generating biomarker test result from medical record by using large language model

Abstract

According to one aspect of the present disclosure for realizing the above-described task, disclosed is a method, performed by a computing device, for generating a biomarker test result from a medical record by using a large language model (LLM). The method may comprise the steps of: receiving a medical record text; generating, from the medical record text, a test result for at least one biomarker related to a target disease, by using a pre-trained first LLM; and storing the test result as a data structure of a common data model (CDM). The representative drawing is FIG. 3.

Description

Method and apparatus for generating biomarker test results from medical records using a large-scale language model

[0001] The present disclosure relates to a technology for generating biomarker test results, and specifically to a method and apparatus for generating biomarker test results from medical records using a large-scale language model.

[0002] Large Language Models (LLMs) are artificial intelligence models containing billions of parameters trained on a vast amount of training data, demonstrating significant performance capabilities across various tasks. In particular, they outperform not only humans but also existing machine learning models in tasks such as extracting necessary information from text data or restructuring it through summarization, and this trend continues even in specialized fields like medicine or law.

[0003] With the rapid advancement of LLM, the potential to utilize unstructured text data, which was previously unprocessable due to practical limitations such as time, cost, and difficulty, has expanded. LLM demonstrates exceptional capabilities in unstructured data analysis and contributes to the analysis of complex patterns and trends that were difficult to identify using conventional methods. As such, LLM is garnering attention as a core technology that expands the scope of data analysis and enhances the efficiency and accuracy of data-driven decision-making.

[0004] Meanwhile, biomarkers are indicators representing specific human conditions, the presence or absence of disease, or responses to treatment, and are utilized as indicators to assess the body's current state. Biomarkers encompass various types, including genetic biomarkers, protein biomarkers, and imaging biomarkers, and can be employed in a series of disease-related processes, such as disease prediction, diagnosis, and evaluation of treatment response. In particular, research on biomarkers is actively underway in the field of cancer.

[0005] Korean Patent Publication No. 10-2022-0106419 (published July 29, 2022) discloses a method for medical domain-specific named entity identification using a language model.

[0006] The present disclosure is devised in response to the aforementioned background technology and addresses the problem of generating biomarker test results from medical records using a large-scale language model. Specifically, the present disclosure addresses the problem of generating test results for biomarkers related to target diseases from medical record text using a large-scale language model.

[0007] Meanwhile, the technical problem that the present disclosure aims to solve is not limited to the technical problem mentioned above, and various technical problems may be included within the scope obvious to a person skilled in the art from the contents described below.

[0008] According to one aspect of the present disclosure for realizing the aforementioned task, a method for generating biomarker test results from a medical record using a large language model, performed by a computing device, is disclosed. The method may include: receiving a medical record text; generating test results for at least one biomarker associated with a target disease from the medical record text using a pre-trained first large language model (LLM); and storing the test results in a data structure of a common data model (CDM).

[0009] In one embodiment, the method further comprises: a step of identifying at least one name from the medical record text using a pre-trained artificial intelligence-based name identification model prior to the step of generating the inspection result; a step of identifying personal information excluding the name from the medical record text using a regular expression; and a step of de-identifying personal information including the at least one name identified using the name identification model and the regular expression in the medical record text, wherein the name identification model may correspond to an artificial intelligence model pre-trained to identify at least one name included in the input text from the input text using a training dataset comprising a plurality of names generated by combining N surnames and M given names.

[0010] In one embodiment, the method may further include: a step of identifying a first text and a second text that are duplicates of each other included in the medical record text prior to the step of generating the inspection result; and a step of deleting the second text from the medical record text, wherein the recording time is after the recording time of the first text.

[0011] In one embodiment, the step of generating the test result may include: generating a first input prompt comprising the medical record text, examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a second instruction text instructing the generation of a test result for biomarkers associated with the target disease; and inputting the first input prompt into the first large-scale language model to generate the test result.

[0012] In one embodiment, the step of generating the test result may include: generating a second input prompt comprising the medical record text, an example of at least one biomarker associated with the target disease, and a second instruction text instructing the generation of a test result for the biomarker associated with the target disease; and inputting the second input prompt into the first large-scale language model to generate the test result.

[0013] In one embodiment, examples of the at least one biomarker include the name of the at least one biomarker and classification criteria for each of the at least one biomarker, and the test result may include classification results for each of the at least one biomarker classified according to the classification criteria for each of the at least one biomarker.

[0014] In one embodiment, the classification criteria for each of the at least one biomarker correspond to a dichotomous classification method that classifies as positive or negative, and may include a preset threshold value for each of the at least one biomarker.

[0015] In one embodiment, the preset threshold may include at least one preset threshold for each of at least one inspection methods.

[0016] In one embodiment, the first large-scale language model may correspond to a large-scale language model that is pre-trained to output a test result for at least one biomarker associated with the target disease by using a training dataset that includes training medical record text and biomarker information associated with each of a plurality of diseases, with the medical record text and the target disease as inputs.

[0017] In one embodiment, the step of generating the test result may include: a step of extracting text about biomarkers from the medical record text using a pre-trained second large-scale language model; and a step of generating the test result from the text about biomarkers using the first large-scale language model.

[0018] In one embodiment, the step of extracting text about biomarkers from the medical record text using a pre-trained second large-scale language model may include: generating a third input prompt comprising the medical record text, examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a third instruction text instructing the extraction of text about biomarkers related to the target disease; and inputting the third input prompt into the second large-scale language model to generate text about at least one biomarker related to the target disease.

[0019] In one embodiment, the step of extracting text for a biomarker from the medical record text using a pre-trained second large-scale language model may include: generating a fourth input prompt comprising the medical record text, an example of at least one biomarker associated with the target disease, and a third instruction text instructing the extraction of text for the biomarker associated with the target disease; and inputting the fourth input prompt into the second large-scale language model to generate text for at least one biomarker associated with the target disease.

[0020] In one embodiment, the second large-scale language model may correspond to a large-scale language model that is pre-trained to output text for at least one biomarker associated with the target disease by using a training dataset that includes training medical record text and biomarker information associated with each of the plurality of diseases, with the medical record text and the target disease as inputs.

[0021] In one embodiment, the step of storing the test results in a data structure of a common data model may include: a step of classifying the test results for each of the at least one biomarker based on a pre-set classification criterion for each of the at least one biomarker; and a step of storing the classified test results in a data structure of the common data model.

[0022] In one embodiment, the step of generating the test results may include the step of generating a list of candidate diseases that may occur in a patient corresponding to the medical record text and the test results from the medical record text using the first large-scale language model.

[0023] In one embodiment, the candidate disease list may include the name of at least one candidate disease and the probability that each of the at least one candidate disease may occur in a patient.

[0024] In one embodiment, the first large-scale language model may correspond to a large-scale language model that is pre-trained to output a list of candidate diseases and a test result for at least one biomarker associated with the target disease, using a training dataset that includes a training medical record text, biomarker information associated with each of the multiple diseases, and the prevalence of each of the multiple diseases according to the biomarker test result, with the medical record text and the target disease as inputs.

[0025] In one embodiment, the target disease may correspond to the disease with the highest probability of occurring in the patient among at least one candidate disease included in the candidate disease list.

[0026] According to one aspect of the present disclosure, a computing device is disclosed for generating biomarker test results from medical records using a large language model. The computing device comprises a processor including at least one core; memory; and a network unit, wherein the processor receives medical record text, generates test results for at least one biomarker associated with a target disease from said medical record text using a pre-trained first large language model (LLM), and stores said test results in a data structure of a common data model (CDM).

[0027] According to one aspect of the present disclosure, a computer program stored on a computer-readable storage medium is disclosed. The computer program enables a processor of a computing device to perform a method for generating biomarker test results from a medical record using a large language model, the method may include: receiving medical record text; generating test results for at least one biomarker associated with a target disease from said medical record text using a pre-trained first large language model (LLM); and storing said test results in a data structure of a common data model (CDM).

[0028] According to some embodiments of the present disclosure, biomarker test results can be generated from medical records using a large-scale language model.

[0029] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0030] Various aspects are now described with reference to the drawings, wherein similar reference numbers are used to collectively refer to similar components. In the following embodiments, for illustrative purposes, a number of specific details are presented to provide a comprehensive understanding of one or more aspects. However, it will be apparent that such aspect(s) may be practiced without these specific details. In other examples, known structures and devices are illustrated in block diagram form to facilitate the description of one or more aspects.

[0031] FIG. 1 is a block diagram of a computing device that generates biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure.

[0032] FIG. 2 is a schematic diagram showing a network function according to one embodiment of the present disclosure.

[0033] FIG. 3 is a flowchart illustrating a method in which a computing device generates biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure.

[0034] FIG. 4 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure generating biomarker test results from medical record text using a first large-scale language model.

[0035] FIG. 5 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure generating biomarker test results from medical record text using a first large-scale language model and a second large-scale language model.

[0036] FIG. 6 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure storing generated biomarker test results in a data structure of CDM.

[0037] FIG. 7 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0038] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0039] As used herein, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0040] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the terms "or" and "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0041] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0042] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0043] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0044] The description of the presented embodiments is provided to enable those skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0045] Terms expressed as N, such as the first, second, or third in the present disclosure, are used to distinguish at least one entity. For example, the entities expressed as the first and second may be identical or different from each other.

[0046] In the present disclosure, the term "biomarker" may refer to an indicator representing a specific state of the human body, the presence or absence of disease, or a response to treatment. That is, biomarkers related to cells, blood vessels, proteins, genes, etc., within the body can be utilized as indicators to identify changes within the body. For example, based on the subject being tested, biomarkers may include genetic biomarkers, protein biomarkers, metabolite biomarkers, imaging biomarkers, etc.

[0047] In one embodiment, a Common Data Model (CDM) may refer to a data model that establishes the same data structure and specifications to facilitate the management and analysis of multiple data with different data structures. Through this, data held by each of multiple entities (e.g., medical institutions) can be processed collectively.

[0048] In the present disclosure, the term "medical record text" is used for illustrative purposes for convenience of explanation and may be interpreted to encompass medical text data related to the medical field. Accordingly, those skilled in the art may interpret the methodology for generating biomarker test results for medical record text in the present disclosure as a methodology for generating biomarker test results for medical text data. Hereinafter, embodiments of the present disclosure will be described using the term "medical record text" for illustrative purposes.

[0049] FIG. 1 is a block diagram of a computing device that generates biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure.

[0050] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0051] A computing device (100) according to some embodiments of the present disclosure may be a device for generating biomarker test results from medical records using a large-scale language model. The computing device (100) may include any type of server and / or any type of device. The server may include any type of computer system or computer device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a portable device, or a device controller.

[0052] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0053] The processor (110) may be composed of one or more cores and may include a processor for performing operations related to data processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of a computing device (100). The processor (110) may read a computer program stored in memory (130) and perform data processing to generate biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure. That is, a computer program stored in a computer-readable storage medium (e.g., memory (130)) may cause one or more processors (e.g., processor (110)) to perform operations to generate biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure. Additionally, the processor (110) may implement any component for performing data processing to generate biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure.

[0054] According to one embodiment of the present disclosure, the processor (110) may perform operations for training a neural network. For example, the processor (110) may perform operations for training a neural network, such as processing input data for training in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the training of a network function. For example, the CPU and GPGPU may together process the training of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices may be used together to process the training of a network function and data classification using the network function.

[0055] The processor (110) can typically control the overall operation of the computing device (100). The processor (110) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. that are input or output through components included in the computing device (100) or by running applications stored in memory (130).

[0056] In one embodiment, the processor (110) may receive medical record text. The processor (110) may generate test results for at least one biomarker associated with a target disease from the medical record text using a pre-trained first Large Language Model (LLM). The target disease in this disclosure is a disease subject to analysis associated with a biomarker, and may correspond to at least one disease expected to occur in a patient and / or at least one disease that has already occurred. The target disease may be extracted from the medical record text or pre-set. For example, if the target disease is extracted from the medical record text, the Large Language Model may receive the medical record text as input and extract the target disease for the patient corresponding to the medical record text. If the target disease is pre-set, the computing device (100) may determine a selected disease as the target disease based on user input. The processor (110) may store the test results for at least one biomarker associated with the target disease in a data structure of a common data model.

[0057] In the present disclosure, a biomarker related to a target disease may refer to a biomarker used to determine the likelihood of occurrence, onset (diagnosis), course, and degree of response to a drug for treating the target disease. For example, biomarkers related to lung cancer may include PD-L1, which is used to predict the therapeutic response to an immunotherapy drug for lung cancer treatment, and EGFR, which is used for targeted therapy of lung cancer (e.g., non-small cell lung cancer) based on mutated exons.

[0058] In one embodiment of the present disclosure, the memory (130) may store any form of information generated or determined by the processor (110) and / or any form of information received by the network unit (150). In one embodiment, the memory (130) may store medical record texts of a plurality of patients, biomarker test results and / or a CDM database. The database may be a set of data stored in a form that can be processed by the computing device (100).

[0059] In one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and / or an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto. The memory (130) may be operated by a processor (110).

[0060] A network unit (150) according to one embodiment of the present disclosure may include any wired or wireless communication network capable of transmitting and receiving any type of data and signals, etc., as described in the present disclosure. The technologies described in this specification may be used not only in the networks mentioned above but also in other networks.

[0061] FIG. 2 is a diagram showing a network function according to one embodiment of the present disclosure.

[0062] Throughout this specification, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0063] A neural network can be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting neural networks may be interconnected by one or more links.

[0064] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0065] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight can be variable and can be varied by the user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0066] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0067] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0068] In one embodiment of the present disclosure, a set of neurons or nodes may be defined by the expression a layer.

[0069] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0070] A neural network according to one embodiment of the present disclosure may have the same number of nodes in the input layer as the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the same number of nodes in the input layer as the number of nodes in the output layer as the number of nodes increases as it progresses from the input layer to the hidden layer. Furthermore, a neural network according to yet another embodiment of the present disclosure may have the same number of nodes in the input layer as the number of nodes in the output layer as the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer is greater than the number of nodes in the output layer.

[0071] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network allows for the identification of latent structures in data. Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), Restricted Boltzmann Machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0072] The artificial intelligence-based model of the present disclosure may be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

[0073] A neural network that can be used in an artificial intelligence-based model of the present disclosure may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, federated learning for distributed deep learning, or incremental learning. Training of a neural network may be a process of applying knowledge to the neural network to perform a specific operation.

[0074] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In supervised learning, training data is used where the correct answer is labeled for each data point (i.e., labeled training data), whereas in unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning regarding data classification, the training data may consist of data where each training data point is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data.

[0075] As another example, in the case of unsupervised learning for data classification, errors can be calculated by comparing the input training data with the neural network output. The calculated errors are backpropagated within the neural network (i.e., from the output layer to the input layer), and this backpropagation updates the connection weights of each node in each layer of the neural network. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of errors can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations in the neural network's learning cycle. For instance, a high learning rate can be used during the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used during the later stages to improve accuracy.

[0076] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed by the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a type of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0077] According to one embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed. The data structure described above may be stored in a memory (130) in the present disclosure, executed by a processor (110), and transmitted and received by a network unit (150).

[0078] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data for solving specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of its resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0079] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0080] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge may include a line connecting two different vertices. Graph data structures may include tree data structures. A tree data structure may be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it may be a data structure that does not form a loop in a graph data structure.

[0081] The data structure may include a neural network. The data structure including the neural network may be stored on a computer-readable medium. The data structure including the neural network may also include data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for learning the neural network, etc. The data structure including the neural network may include any of the components disclosed above. That is, the data structure including the neural network may be composed of all or any combination thereof, such as data preprocessed for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. In addition to the aforementioned components, the data structure including the neural network may include any other information that determines the characteristics of the neural network. Additionally, the data structure may include all forms of data used or generated during the computational process of the neural network, and is not limited to the foregoing. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. The neural network may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one node.

[0082] A data structure may include data input to a neural network. A data structure including data input to a neural network may be stored on a computer-readable medium. Data input to a neural network may include training data input during the neural network learning process and / or input data input to a neural network after training is complete. Data input to a neural network may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into a neural network. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0083] The data structure may include weights of the neural network. (In this specification, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0084] As an example rather than a limitation, weights may include weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Weights that vary during the neural network learning process may include weights at the start of the learning cycle and / or weights that vary during the learning cycle. Weights for which neural network learning is completed may include weights for which the learning cycle is completed. Accordingly, a data structure containing the weights of a neural network may include a data structure containing weights that vary during the neural network learning process and / or weights for which neural network learning is completed. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of a neural network. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0085] Data structures containing the weights of a neural network may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network are not limited to serialization. Furthermore, data structures containing the weights of a neural network may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0086] The data structure may include hyperparameters of the neural network. The data structure including the neural network hyperparameters may be stored on a computer-readable medium. The hyperparameters may be variables that are varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle iterations, weight initialization (e.g., setting the range of weight values ​​subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0087] An AI-based model according to one embodiment of the present disclosure may include a Large Language Model (LLM). In the present disclosure, a Large Language Model may refer to an AI-based model trained using a vast amount of training data to perform Natural Language Processing. The Large Language Model may include a Transformer, a Transformer encoder family model, and / or a Transformer decoder family model. A Transformer encoder family model may correspond to an AI model that uses a Transformer encoder structure. A Transformer decoder family model may correspond to an AI model that uses a Transformer decoder structure. For example, the AI-based model of the present disclosure may include a GPT family Large Language Model, a BERT (Bidirectional Encoder Representations from Transformers) family Large Language Model, etc.

[0088] In one embodiment, the transformer may be composed of an encoder that encodes input data and a decoder that decodes the encoded data. The transformer may have a structure that takes a series of input data as input, undergoes encoding and decoding steps, and outputs a series of output data. In one embodiment, the series of input data may be processed into a form that the transformer can compute. The process of processing the series of input data into a form that the transformer can compute may include a tokenizing process and an embedding process. The tokenizing process may refer to the process of dividing the series of input data into tokens of a certain unit. For example, the certain unit may include word units. The embedding process may refer to the process of converting at least one token tokenized from the series of input data into an embedding vector.

[0089] In one embodiment, the transformer can obtain an embedding vector to be input to an encoder by combining a token embedding vector that embeds at least one token corresponding to a series of input data, a segment embedding vector that distinguishes sentences containing the tokens for each token, and a position embedding vector that reflects the position of the token. The encoder family model and the decoder family model of the transformer can also obtain embedding vectors by performing the same method.

[0090] In one embodiment, for the transformer to encode and decode a series of input data, the encoder and decoder within the transformer may utilize an attention algorithm. An attention algorithm may refer to an algorithm that, for a given query, calculates similarity by applying a softmax function to an attention score obtained by matrix multiplying the query with a key, and calculates an attention value for the query by matrix multiplying the calculated similarity with a value.

[0091] In one embodiment, the self-attention algorithm may refer to an attention algorithm that uses a query, a key, and a value generated by multiplying the same embedding vector by a query weight, a key weight, and a value weight, respectively. The cross-attention algorithm may refer to an attention algorithm that uses a query generated by multiplying a first embedding vector by a query weight, and a key and a value generated by multiplying a second embedding vector by a key weight and a value weight, respectively. The query weight, the key weight, and the value weight may be trainable parameters that are updated through the training process of a large-scale language model.

[0092] In one embodiment, the encoder of the transformer may include an embedding layer, a self-attention layer that applies a self-attention algorithm to an embedding vector, a normalization layer, and a feed-forward neural network (FFN). Additionally, the encoder may have a form in which N unit structures including a self-attention layer, a normalization layer, and a feed-forward neural network are connected. The decoder of the transformer may include an embedding layer, a masked self-attention layer, a normalization layer, a cross-attention layer that applies a cross-attention algorithm, and a feed-forward neural network. Additionally, the decoder may have a form in which N unit structures including a masked self-attention layer, a normalization layer, a cross-attention layer, and a feed-forward neural network are connected. The masked self-attention layer may correspond to a layer that calculates an attention value for each of the sequences in which words are sequentially included among a plurality of words included in a series of input data.

[0093] A transformer may include additional components such as a linear layer and a softmax layer, in addition to an encoder and a decoder. The encoder family model and the decoder family model of a transformer may also each include the aforementioned additional components in addition to the encoder and decoder. A method for constructing a transformer using an attention algorithm may include the method disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0094] In one embodiment, attention layers such as a self-attention layer, a masked self-attention layer, and a cross-attention layer may correspond to a multi-head attention layer that includes a plurality of attention layers in parallel. The multi-head attention layer may concatenate the attention values ​​output from each of the plurality of attention layers and output an output attention value by matrix multiplying the concatenated matrix by an output weight. The output attention value output from the multi-head attention layer may have the same size as the attention value output from a single attention layer.

[0095] In one embodiment, the transformer may be trained through a Masked Language Model (MLM) process, a Next Sentence Prediction (NSP) process, etc. The MLM process may refer to a training process that predicts a masked word through a series of training data in which some words are masked. The NSP process may refer to a training process that determines whether two sentences are connected sentences in a series of training data containing any two sentences.

[0096] In one embodiment, a large-scale language model can process various data formats, such as natural language text as well as image data, audio data, and video data. The large-scale language model can embed data to convert data of various data formats into a series of computationally operable data. The large-scale language model can process additional data that represents the relative positional or topological relationships between a series of input data. Alternatively, a series of input data may be embedded by additionally reflecting vectors that represent the relative positional or topological relationships between the input data. In one example, the relative positional relationships between a series of input data may include, but are not limited to, word order within a natural language sentence, the relative positional relationships of each segmented image, and the temporal order of segmented audio waveforms. The process of adding information that represents the relative positional or topological relationships between a series of input data may be referred to as positional encoding.

[0097] An example of a large-scale language model for processing image data (Vision Transformer, ViT) is disclosed in Dosovitskiy, et al., AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE, which is incorporated herein by reference.

[0098] An artificial intelligence model according to one embodiment of the present disclosure may include a multimodal large-scale language model. A multimodal large-scale language model may refer to a large-scale language model capable of understanding and processing relationships between different data formats, such as natural language text data, image data, audio data, and video data. A multimodal language model may include a plurality of encoders that encode input data corresponding to each data format. A multimodal language model may be trained to calculate similarity between embedding vectors encoded by encoders of each data format through training data containing data of different data formats, such that similarity between identical pairs is calculated to be higher and similarity between different pairs is calculated to be lower.

[0099] An example of a large-scale multimodal language model that understands and processes the relationship between image data and natural language text data (Contrastive Language-Image Pre-training, CLIP) is disclosed in Alec Radford, et al., *LEARNING TRANSFERABLE VISUAL MODELS FROM NATURAL LANGUAGE SUPERVISION*, which is incorporated herein by reference.

[0100] In the following, according to one embodiment of the present disclosure, a method is disclosed in which a computing device (100) generates biomarker test results from medical records using a large-scale language model. In the following, operations performed by the processor (110) of the computing device (100) are described as being performed by the computing device (100).

[0101] FIG. 3 is a flowchart illustrating a method in which a computing device generates biomarker test results from medical records using a large-scale language model according to one embodiment of the present disclosure.

[0102] In step S310, a computing device (100) according to one embodiment may receive medical record text. For example, the medical record text may correspond to a single patient. For example, the medical record text may correspond to a medical record for a first patient recorded at a first medical institution. Or, the medical record text may correspond to medical records for a first patient recorded at multiple medical institutions. The computing device (100) may receive medical record texts corresponding to each of multiple patients. The medical record text is text recording any information related to the patient's health status, and may include patient information, medical history, examination findings, test results, disease name, course of disease, treatment, patient's health progress, summary at discharge, etc. For example, the test results may include test results for biomarkers performed on the patient.

[0103] Hereinafter, a methodology for a computing device (100) to generate biomarker test results from a single medical record text (i.e., a medical record text corresponding to a single patient) will be described as an example. However, not limited thereto, the computing device (100) may generate biomarker test results corresponding to each of a plurality of medical record texts from a plurality of medical record texts through the embodiments described below, and store the generated plurality of biomarker test results in the data structure of a CDM model.

[0104] In step S320, a computing device (100) according to one embodiment can generate a test result for at least one biomarker related to a target disease from medical record text using a pre-trained first large-scale language model. The computing device (100) can generate a test result for at least one biomarker included in the medical record text using the first large-scale language model. That is, the first large-scale language model is a large-scale language model for Named Entity Recognition (NER), and can take the medical record text as input to generate a test result for all biomarkers recorded in the medical record text and / or a test result for at least one biomarker related to a target disease among the test results for all biomarkers recorded in the medical record text.

[0105] In one embodiment, the test result for a biomarker may include the test result value of the biomarker recorded in the medical record text (e.g., numerical value, concentration, level, positive / negative, etc.) and / or a classification result in which the recorded test result value of the biomarker is classified. For example, the test result value of the biomarker may be classified according to a pre-set classification criterion for each biomarker. The first large-scale language model may receive not only the medical record text but also additional classification criteria for each biomarker as input, and by additionally performing classification of the test result value of the biomarker according to the classification criterion, generate a test result including a classification result for the biomarker.

[0106] In one embodiment, the computing device (100) may generate a first input prompt comprising medical record text, examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a second instruction text instructing the generation of test results for biomarkers associated with the target disease. The first large-scale language model may, through inference on the medical record text, extract at least one disease expected to occur in the patient corresponding to the medical record text and / or at least one disease that has already occurred as a target disease based on a series of medical records contained in the medical record text. For example, the target disease may be the same as the disease recorded in the medical record text. Or, the target disease may not be explicitly recorded in the medical record text, but may be a target disease inferred by the first large-scale language model from a series of medical records containing biomarker test results. The computing device (100) may input the first input prompt into the first large-scale language model to generate test results for at least one biomarker associated with the target disease.

[0107] In one embodiment, the computing device (100) may generate a second input prompt including medical record text, an example of at least one biomarker associated with the target disease, and a second instruction text instructing the generation of a test result for the biomarker associated with the target disease. That is, the computing device (100) may present a predetermined target disease to the first large-scale language model without instructing the extraction of the target disease through the second input prompt including the name of the target disease. The computing device (100) may generate a test result for at least one biomarker associated with the target disease by inputting the second input prompt into the first large-scale language model.

[0108] In one embodiment, the computing device (100) may generate an input prompt including medical record text, an example of at least one biomarker, and instruction text instructing the generation of test results for the biomarker. The computing device (100) may instruct a first large-scale language model to generate test results for all biomarkers included in the medical record text. The computing device (100) may generate test results for at least one biomarker recorded in the medical record text by inputting the input prompt generated in the first large-scale language model.

[0109] In one embodiment, an example of at least one biomarker included in the input prompt may include the name of at least one biomarker and a classification criterion for each of at least one biomarker. Additionally, the example of the biomarker may further include a method for testing the biomarker. The test result for at least one biomarker may include a classification result for each of at least one biomarker classified according to the classification criterion for each of at least one biomarker.

[0110] For example, a single biomarker may have multiple classification criteria. For instance, a single biomarker may have different classification criteria depending on the testing method. Depending on the testing method, the classification criteria may include different classification methods such as numerical values, levels, categories, and positive / negative.

[0111] For example, the classification criteria for each of at least one biomarker may correspond to a dichotomous classification method that classifies as positive or negative. In this case, the classification criteria may include a preset threshold for each of at least one biomarker. For example, if the input prompt includes a classification criterion for a dichotomous classification method for the first biomarker, the first large-scale language model may output a test result for the first biomarker classified as positive if the value of the first biomarker recorded in the medical record text is greater than or equal to the preset threshold, and as negative if it is less than. The preset threshold may also be set differently depending on the test method for the biomarker. For example, the threshold may be set differently for each of the first test method and the second test method for testing the first biomarker. That is, the preset threshold for each of at least one biomarker may include at least one preset threshold for each of at least one test method.

[0112] For example, the input prompts entered into the first large-scale language model may include instruction text such as “Biomarkers associated with lung cancer patients include PD-L1, EGFR, etc. Extract test results for biomarkers associated with lung cancer patients recorded in the medical record text.” along with medical record text. Additionally, as a classification criterion for biomarker test results, it may include classification criteria such as “Output: {PD-L1(+): positive, PD-L1(-): negative, PD-L1(100%): positive, PD-L1(low): negative, PD-L1(high): positive}”.

[0113] For the example above, if the target disease is lung cancer, biomarkers related to lung cancer may include PD-L1, EGFR, etc. PD-L1 is a protein on the surface of cancer cells and can be used as a biomarker to predict the therapeutic effect of anticancer drugs. The PD-L1 testing method may include different types of tests depending on the type of PD-L1 antibody used in the test, such as SP142, SP263, and 22C3. Additionally, the PD-L1 testing method may include different test targets depending on the target cells, such as measuring PD-L1 expression levels in tumor cells or measuring PD-L1 expression levels in immune cells. The classification criteria for PD-L1 may include different classification criteria for each testing method (test type and / or test target). Examples of PD-L1 included in the input prompt may include PD-L1 testing methods and / or classification criteria for each testing method.

[0114] EGFR stands for Epidermal Growth Factor Receptor and can be used as a biomarker to predict the efficacy of anticancer drugs by determining the presence or absence of mutations in the EGFR gene. EGFR testing methods may include different test targets depending on the type of exon (Exon 18, 19, 20, 21, etc.). Classification criteria for EGFR may include different criteria depending on the testing method (e.g., test targets such as the type of exon). For example, EGFR classification criteria may be set to classify into different mutation categories depending on the type of exon in which the mutation occurs. For example, if a mutation occurs in Exon 21, the EGFR test result may be generated as “L858R mutation.” Alternatively, EGFR classification criteria may be set to classify as positive if a mutation occurs in any exon, regardless of the type of exon. Examples of EGFR included in the input prompt may include EGFR testing methods and / or classification criteria by testing method.

[0115] Additionally, examples of biomarkers may include an output format for biomarker test results to be output by the first large-scale language model. For example, examples of biomarkers may include data formats such as JSON or dictionary as output formats, where biomarkers are keys and test results are values.

[0116] In one embodiment, the computing device (100) can identify the gender and age of a patient corresponding to the medical record text from the medical record text. For example, the computing device (100) can additionally obtain the gender and age of the patient as well as the test results for biomarkers from the medical record text using a first large-scale language model. The computing device (100) can obtain the average test results for at least one biomarker according to gender and age. The computing device (100) can generate a comparison result by comparing the test results for at least one biomarker based on the medical record text with the average test results.

[0117] In one embodiment, the test results for at least one biomarker may include at least one recording time and a test result corresponding to each of the at least one recording time recorded in the medical record text for each of the at least one biomarkers. For example, the medical record text may include at least one test result for the same biomarker. In this case, the medical record text may include information about the recording time at which the test result was recorded, along with the test result. The computing device (100) can obtain the test results of the biomarker for each recording time from the medical record text. Through this, the technical effect of being able to verify the change in the test result over time for a single biomarker can be achieved.

[0118] In step S330, a computing device (100) according to one embodiment may store test results for at least one biomarker related to a target disease in a data structure of a common data model. The computing device (100) may store test results for at least one biomarker output from a first large-scale language model in a data structure of a common data model. A methodology for storing test results for biomarkers in a data structure of a common data model will be described in detail through FIG. 6.

[0119] Through the embodiments of the present disclosure, the computing device (100) can achieve the technical effect of being able to store structured biomarker test results from unstructured medical record text using a large-scale language model. In particular, although biomarkers are important factors in diagnosing a patient's condition or formulating a treatment plan in most cancers, they have been recorded as individual text rather than being stored and managed in a standardized data format due to the variety of their types and results. Therefore, by accurately extracting and structuring test results for biomarkers from medical record text, the technical effect of being able to store, manage, and analyze test results for various biomarkers in a common data format can be achieved.

[0120] In addition, by instructing a large-scale language model to generate test results for at least one biomarker related to the target disease among the biomarkers included in medical record text, a technical effect can be achieved in which only the biomarker related to the target disease can be analyzed.

[0121] FIG. 4 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure generating biomarker test results from medical record text using a first large-scale language model.

[0122] In one embodiment, a computing device (100) may receive a medical record text (410). The computing device (100) may input the medical record text (410) into a first large-scale language model (430) to obtain test results (440) for each of at least one biomarker related to a target disease among the biomarkers included in the medical record text (410). The computing device (100) may store the test results (440) for each of at least one biomarker in a CDM table (450) having a data structure of a common data model.

[0123] In one embodiment, the first large-scale language model (430) may correspond to a pre-trained large-scale language model that takes the medical record text and the target disease as input and outputs a test result for at least one biomarker related to the target disease, using a training dataset that includes the medical record text and biomarker information related to each of the plurality of diseases. For example, the biomarker information may include biomarker test results such as whether there is a relationship with each disease, the test result value of the biomarker according to the presence or absence of the disease, and / or classification results. Additionally, the first large-scale language model (430) may correspond to a pre-trained large-scale language model that takes the medical record text as input and outputs a test result for at least one biomarker included in the medical record text, using a training dataset that includes the medical record text and a test result for the biomarker included in the medical record text.

[0124] The first large-scale language model (430) being pre-trained with a training dataset containing training record texts may include fine-tuning using the training dataset containing training record texts in the pre-trained large-scale language model (420). That is, the first large-scale language model (430) may correspond to the large-scale language model that is fine-tuned to generate biomarker test results in the pre-trained large-scale language model (420).

[0125] In one embodiment, a computing device (100) may use a first large-scale language model (430) to generate a list of candidate diseases that may occur in a patient corresponding to the medical record text (410) and test results (440) for each of at least one biomarker from the medical record text (410). For example, the computing device (100) may add instruction text to an input prompt to be input to the first large-scale language model (430) to instruct the output of a list of candidate diseases from the medical record text. The list of candidate diseases may include diseases recorded in the medical record text. Additionally, the list of candidate diseases may include at least one disease expected to occur in the patient and / or at least one disease that has already occurred, inferred from a series of medical records (e.g., test results for biomarkers, etc.) recorded in the medical record text.

[0126] In one embodiment, the candidate disease list may include the name of at least one candidate disease and the probability that each of at least one candidate disease may occur in a patient. For example, the probability that a candidate disease may occur in a patient may correspond to a prevalence rate, which is the ratio of patients with the candidate disease in a specific group having the same test results for a specific biomarker. That is, the computing device (100) may add instruction text to an input prompt to be input to a first large-scale language model, instructing to output the prevalence of the candidate disease according to the test results for each biomarker as the probability that the candidate disease may occur in a patient.

[0127] In one embodiment, the first large-scale language model (430) may correspond to a pre-trained large-scale language model that takes medical record text and target disease as input and outputs a candidate disease list and test results (440) for each of at least one biomarker associated with the target disease, using a training dataset that includes a training medical record text, biomarker information related to each of the multiple diseases, and the prevalence of each of the multiple diseases according to the biomarker test results. In this case, the first large-scale language model (430) may be pre-trained to output a candidate disease list that includes the names of the candidate diseases and the probability that the candidate diseases may occur in the patient.

[0128] In one embodiment, the target disease may correspond to the disease with the highest probability of occurring to the patient among at least one candidate disease included in the candidate disease list. For example, the computing device (100) may add instruction text to an input prompt to be entered into the first large-scale language model (430) instructing the candidate disease with the highest prevalence according to the biomarker test results to be set as the target disease. The first large-scale language model (430) may calculate the prevalence of each candidate disease based on the biomarker test results recorded in the medical record text and set the disease with the highest prevalence as the target disease.

[0129] FIG. 5 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure generating biomarker test results from medical record text using a first large-scale language model and a second large-scale language model.

[0130] In one embodiment, the computing device (100) can generate test results (440) for each of at least one biomarker from medical record text (410) using two large-scale language models. That is, the computing device (100) can use a second large-scale language model (530) to extract text (540) about the biomarker, which is text related to the biomarker such as the name of the biomarker, the method of testing the biomarker, and the test results for the biomarker, from the medical record text (410), and then use a first large-scale language model (430) to generate test results (440) for the biomarker from the text (540) about the biomarker. The computing device (100) can store the test results (440) for the biomarker in a CDM table (450) having a data structure of a common data model.

[0131] In one embodiment, the computing device (100) may repeat the inference operation of each of the first large-scale language model (430) and the second large-scale language model (530). For example, the computing device (100) may repeat the process of inputting medical record text (410) and / or medical record text different from the previously input medical record text into the second large-scale language model (530) to output text (540) for biomarkers. Additionally, the computing device (100) may repeat the process of inputting text (540) for biomarkers into the first large-scale language model (430) to output test results (440) for each of at least one biomarker. Additionally, the computing device (100) may repeat the process of fine-tuning the first large-scale language model (430) and / or the second large-scale language model (530) using a training dataset containing training medical record text.

[0132] In one embodiment, the computing device (100) may perform preprocessing on the medical record text (410) before inputting the medical record text (410) into the first large-scale language model (430) or the second large-scale language model (530). The computing device (100) may identify at least one name from the medical record text (410) using a pre-trained artificial intelligence-based name identification model. The name identification model in the present disclosure may correspond to an artificial intelligence model that identifies a name included in the input text from the input text. The name identification model may correspond to an artificial intelligence model that is pre-trained to identify at least one name included in the input text from the input text using a training dataset containing a plurality of names generated by combining N surnames and M given names. For example, the name identification model may include a large-scale language model, a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM), etc.

[0133] In one embodiment, the computing device (100) can identify personal information excluding names from medical record text (410) using a regular expression. For example, personal information excluding names may include patient number, resident registration number, mobile phone number, email address, address, age, etc. The computing device (100) can de-identify personal information including at least one name identified using a name identification model and a regular expression in the medical record text (410). That is, the computing device (100) can de-identify personal information excluding at least one name identified by the name identification model and the name identified using a regular expression in the medical record text (410). For example, the computing device (100) can delete at least one name identified in the medical record text (410) or change it to any other name. As another example, the computing device (100) can delete personal information excluding names identified in the medical record text (410) or change it to any other text.

[0134] In one embodiment, the computing device (100) can identify a first text and a second text that are duplicated from each other and included in the medical record text (410). The two duplicate texts may refer to two texts that are similar beyond the matching texts and / or a preset text similarity. The computing device (100) can delete the second text from the medical record text (410) where the recording time is after the recording time of the first text. That is, since it is frequent in the medical field to refer to at least a portion of a patient's previous medical records as is during treatment, the computing device (100) can delete text identical to the past text from the medical record text (410). For example, the computing device (100) can delete duplicate text from the medical record text (410) using the Damerau-Levenshtein distance algorithm, the Longest common subsequence algorithm, etc.

[0135] In one embodiment, text (540) regarding biomarkers in medical record text (410) may not be determined to be duplicate text subject to removal. For example, the computing device (100) may not remove text (540) regarding biomarkers even if the same biomarker is recorded two or more times in the medical record text (410).

[0136] In one embodiment, the computing device (100) can extract text (540) about biomarkers from medical record text (410) using a pre-trained second large-scale language model (530). For example, the computing device (100) can extract text (540) about at least one biomarker related to a target disease from medical record text (410) using the second large-scale language model (530). Alternatively, the computing device (100) can extract text (540) about at least one biomarker recorded in the medical record text (410) from the medical record text (410) using the second large-scale language model (530).

[0137] In one embodiment, the computing device (100) can generate test results (440) for each of at least one biomarker associated with a target disease from text (540) about biomarkers using a first large-scale language model (430). Additionally, the computing device (100) can generate test results (440) for each of at least one biomarker recorded in text (540) about biomarkers from text (540) about biomarkers using a first large-scale language model (430).

[0138] In one embodiment, the second large language model (530) being pre-trained with a training dataset containing training duty record text may include fine-tuning the pre-trained large language model (520) using the training dataset containing training duty record text. That is, the second large language model (530) may correspond to the large language model that is fine-tuned to extract text for biomarkers in the pre-trained large language model (520).

[0139] In one embodiment, the computing device (100) may generate a third input prompt including medical record text (410), examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a third instruction text instructing the extraction of text regarding biomarkers related to the target disease. The computing device (100) may input the third input prompt into a second large-scale language model (530) to generate text (540) regarding at least one biomarker related to the target disease. The second large-scale language model (530) may, through inference on the medical record text (410), extract at least one disease expected to occur in the patient corresponding to the medical record text (410) and / or at least one disease that has already occurred as the target disease, based on a series of medical records included in the medical record text (410).

[0140] In one embodiment, the computing device (100) may generate a fourth input prompt including medical record text (410), an example of at least one biomarker associated with the target disease, and a third instruction text instructing text extraction for the biomarker associated with the target disease. The computing device (100) may input the fourth input prompt into a second large-scale language model (530) to generate text (540) for at least one biomarker associated with the target disease.

[0141] In one embodiment, the second large-scale language model (530) may correspond to a pre-trained large-scale language model that takes the medical record text (410) and the target disease as input and outputs text (540) for at least one biomarker related to the target disease, using a training dataset that includes the medical record text and biomarker information related to each of the plurality of diseases. Additionally, the second large-scale language model (530) may correspond to a pre-trained large-scale language model that takes the medical record text (410) as input and outputs text (540) for at least one biomarker recorded in the medical record text (410), using a training dataset that includes the medical record text and test results for the biomarker included in the medical record text.

[0142] Through this, the computing device (100) can achieve the technical effect of generating biomarker test results more efficiently and accurately by utilizing language models optimized for text extraction and NER respectively, by using a second large-scale language model (530) to extract only the text related to biomarkers from the medical record text (410) in which various medical records are recorded, and by using a first large-scale language model (430) to generate test results (440) for each of at least one biomarker from the extracted text (540) for biomarkers.

[0143] FIG. 6 is a flowchart illustrating the process of a computing device according to one embodiment of the present disclosure storing generated biomarker test results in a data structure of CDM.

[0144] In one embodiment, the computing device (100) may link CDM concepts (620) according to the type of biomarker and / or classified test results (610). In the present disclosure, a CDM concept (620) may mean a data subject classified to optimize data management in a database and / or table of a common data model. The computing device (100) may identify a CDM concept (620) corresponding to a test result (440) for each of at least one biomarker generated from a first large-scale language model, and store the test result (440) for each of at least one biomarker in a CDM table (450) corresponding to the identified CDM concept (620).

[0145] In one embodiment, the computing device (100) can classify the test results (440) for each of at least one biomarker based on a pre-set classification criterion for each of at least one biomarker. For example, the test results (440) for each of at least one biomarker generated from the first large-scale language model may be the test results for the biomarker recorded in the medical record text itself, and may not include the test results (610) classified according to the pre-set classification criterion. The computing device (100) can generate the classified test results (610) by classifying the test results (440) for each of at least one biomarker using a data classification algorithm. The computing device (100) can store the classified test results (610) in the data structure of a common data model. For example, the data classification algorithm is an algorithm for classifying multiple biomarker test results and storing them in a CDM table (450), and may include not only pre-set classification criteria but also algorithms such as K-Nearest Neighbors (K-NN), Decision Tree, Random Forest, Naive Bayes, and data classification artificial intelligence models.

[0146] FIG. 7 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0147] Although it has been stated that the present disclosure can generally be implemented by a computing device (100), a person skilled in the art will be well aware that the present disclosure can be implemented by combining computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0148] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0149] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0150] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0151] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0152] Referring to FIG. 7, an exemplary environment for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), a system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including the system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processors and other multiprocessor architectures may also be used as the processing unit (1104).

[0153] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0154] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0155] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0156] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0157] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0158] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0159] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0160] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0161] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0162] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0163] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0164] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0165] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0166] It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that, based on design priorities, the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0167] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0168] As described above, the relevant details have been described in the best mode for carrying out the invention.

[0169] It can be used in devices, systems, etc. that generate biomarker test results from medical records using large-scale language models.

Claims

1. A method for generating biomarker test results from medical records using a large-scale language model executed by a computing device, wherein Step of receiving the medical record text; A step of identifying a first text and a second text that overlap with each other and are included in the above-mentioned medical record text; In the above-mentioned mandatory record text, a step of deleting a second text whose recording time is after the recording time of the first text; A step of generating a test result for at least one biomarker associated with a target disease from the medical record text using a pre-trained first Large Language Model (LLM); and A step of storing the above test results in a data structure of a Common Data Model (CDM); including, method.

2. In Paragraph 1, Prior to the step of generating the above test results, A step of identifying at least one name from the medical record text using a pre-trained artificial intelligence-based name identification model; A step of identifying personal information excluding names from the medical record text using a regular expression; and A step of de-identifying personal information including at least one name identified using the name identification model and the regular expression in the above medical record text; Includes more of, and The above-mentioned name identification model corresponds to an artificial intelligence model pre-trained to identify at least one name included in an input text from an input text using a training dataset containing multiple names generated by combining N surnames and M given names. method.

3. In Paragraph 1, The step of generating the above test results is: A step of generating a first input prompt comprising the above medical record text, examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a second instruction text instructing the generation of test results for biomarkers related to the target disease; and A step of generating the inspection result by inputting the first input prompt into the first large-scale language model; including, method.

4. In Paragraph 1, The step of generating the above test results is: A step of generating a second input prompt comprising the above medical record text, an example of at least one biomarker associated with the target disease, and a second instruction text instructing the generation of a test result for the biomarker associated with the target disease; and A step of generating the inspection result by inputting the second input prompt into the first large-scale language model; including, method.

5. In Paragraph 4, Examples of at least one biomarker above include the name of at least one biomarker and classification criteria for each of the at least one biomarker, and The above test result includes a classification result for each of the at least one biomarker classified according to a classification criterion for each of the at least one biomarker. method.

6. In Paragraph 5, The classification criteria for each of the above-mentioned at least one biomarker are, It corresponds to a dichotomous classification method that classifies as positive or negative, and, including a preset threshold for each of the above at least one biomarker, method.

7. In Paragraph 6, The above preset threshold includes at least one preset threshold for each of at least one inspection methods, method.

8. In Paragraph 1, The first large-scale language model above corresponds to a large-scale language model that is pre-trained to output a test result for at least one biomarker associated with a target disease, using a training dataset containing training medical record text and biomarker information associated with each of a plurality of diseases, with the medical record text and the target disease as inputs. method.

9. In Paragraph 1, The step of generating the above test results is: A step of extracting text regarding biomarkers from the medical record text using a pre-trained second large-scale language model; and A step of generating the test result from the text regarding the biomarker using the first large-scale language model; including, method.

10. In Paragraph 9, The step of extracting text regarding biomarkers from the medical record text using a pre-trained second large-scale language model is: A step of generating a third input prompt comprising the above medical record text, examples of a plurality of biomarkers, a first instruction text instructing the extraction of a target disease, and a third instruction text instructing the extraction of text regarding biomarkers related to the target disease; and A step of generating text for at least one biomarker associated with the target disease by inputting the third input prompt into the second large-scale language model; including, method.

11. In Paragraph 9, The step of extracting text regarding biomarkers from the medical record text using a pre-trained second large-scale language model is: A step of generating a fourth input prompt comprising the above medical record text, an example of at least one biomarker associated with the target disease, and a third instruction text instructing text extraction for the biomarker associated with the target disease; and A step of generating text for at least one biomarker associated with the target disease by inputting the fourth input prompt into the second large-scale language model; including, method.

12. In Paragraph 9, The second large-scale language model above corresponds to a large-scale language model that is pre-trained to output text for at least one biomarker associated with a target disease, using a training dataset containing training medical record text and biomarker information associated with each of a plurality of diseases, with the medical record text and the target disease as inputs. method.

13. In Paragraph 1, The step of storing the above inspection results in the data structure of a common data model is: A step of classifying test results for each of the at least one biomarker based on pre-set classification criteria for each of the at least one biomarker; and A step of storing the classified inspection results in the data structure of the common data model; including, method.

14. In Paragraph 1, The step of generating the above test results is, A method comprising the step of generating a list of candidate diseases that may occur in a patient corresponding to the medical record text and the test results from the medical record text using the first large-scale language model, method.

15. In Paragraph 14, The above list of candidate diseases includes the name of at least one candidate disease and the probability that each of the at least one candidate disease may occur in a patient. method.

16. In Paragraph 14, The above-mentioned first large-scale language model corresponds to a large-scale language model that is pre-trained to output a list of candidate diseases and test results for at least one biomarker associated with the target disease, using a training dataset comprising training medical record text, biomarker information related to each of the multiple diseases, and the prevalence rate of each of the multiple diseases according to biomarker test results, with the medical record text and the target disease as input. method.

17. In Paragraph 14, The above target disease corresponds to the disease with the highest probability of occurring in the patient among at least one candidate disease included in the above candidate disease list, method.

18. A computing device that generates biomarker test results from medical records using a large-scale language model, A processor including at least one core; Memory; and Network Department; Includes, The above processor is, Receive medical record text, Identifying the first text and the second text that overlap with each other and are included in the above-mentioned medical record text, and In the above-mentioned medical record text, delete the second text, whose recording time is after the recording time of the first text, and Using a pre-trained first Large Language Model (LLM), a test result for at least one biomarker associated with a target disease is generated from the medical record text, and Storing the above test results in the data structure of the Common Data Model (CDM), Computing device.

19. A computer program stored on a computer-readable storage medium, wherein the computer program causes a processor of a computing device to perform a method for generating biomarker test results from a medical record using a large-scale language model, and said method: Step of receiving the medical record text; A step of identifying a first text and a second text that overlap with each other and are included in the above-mentioned medical record text; In the above-mentioned mandatory record text, a step of deleting a second text whose recording time is after the recording time of the first text; A step of generating a test result for at least one biomarker associated with a target disease from the medical record text using a pre-trained first Large Language Model (LLM); and A step of storing the above test results in a data structure of a Common Data Model (CDM); including, A computer program stored on a computer-readable storage medium.

Images