Method for structuring unstructured medical data using a language model

Abstract

According to one embodiment of the present disclosure, a method for structuring unstructured medical data based on a language model performed by a computing device is disclosed. The method comprises the steps of: acquiring unstructured medical data; acquiring structured medical data that matches the unstructured medical data; and further training a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data.

Description

Technology Field

[0001] The present invention relates to a method for structuring unstructured medical data, and more specifically, to a method for structuring unstructured medical data based on a language model. Background Technology

[0003] In the medical field, nursing records serve as critical data for monitoring patient conditions and ensuring the continuity of care. These records contain diverse clinical information, including nursing actions, patients' physiological and psychological responses, medication and treatment details, changes in patient condition, patient safety-related incidents, and consultations with physicians. However, most of this information is stored as unstructured data in the form of natural language, which limits its utilization compared to structured data when training and applying medical artificial intelligence (AI) models. Furthermore, medical staff must directly interpret and evaluate unstructured data, leading to reduced efficiency in accessing records and extracting information.

[0004] Recently, advancements in Natural Language Processing (NLP) technology and Large Language Models (LLM) have dramatically improved the level of medical text comprehension. However, for medical AI models requiring precise prediction or judgment, the structure of input data and the accuracy of labels have a significant impact on model performance and clinical reliability. The problem to be solved

[0006] The present disclosure aims to provide a method for converting unstructured medical data into structured medical data that can be utilized as input to an artificial intelligence model, based on a language model. means of solving the problem

[0008] According to one embodiment of the present disclosure for realizing the aforementioned task, a method for structuring unstructured medical data based on a language model is disclosed. The method may include the steps of acquiring unstructured medical data, acquiring structured medical data that matches the unstructured medical data, and further training a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data.

[0009] In one embodiment, the unstructured medical data and the structured medical data may be data pairs that are generated in time-synchronized and automatically matched.

[0010] In one embodiment, the pre-trained language model may be a model generated by additionally training a large-scale language model (LLM) using an autoregressive next-token prediction method that sequentially predicts the next token based on previous tokens based on the unstructured medical data.

[0011] In one embodiment, the step of further training the pre-trained language model may include the step of further training the pre-trained language model based on a first loss function for binary classification of whether a clinical event occurs.

[0012] In one embodiment, the step of further training the pre-trained language model may further include the step of further training the pre-trained language model based on a second loss function for predicting continuous clinical variables.

[0013] In one embodiment, the step of further training the pre-trained language model further includes the step of generating a first embedding vector based on the unstructured medical data, the step of generating a second embedding vector based on the structured medical data, and the step of further training the pre-trained language model based on a third loss function calculated based on the similarity between the first embedding vector and the second embedding vector, and the unstructured medical data and the structured medical data may have different modalities.

[0014] In one embodiment, the step of further training the pre-trained language model may further include the step of generating a plurality of first embedding vectors based on the unstructured medical data, the step of generating a plurality of second embedding vectors based on the structured medical data, and the step of further training the pre-trained language model based on a fourth loss function calculated based on a similarity matrix between the plurality of first embedding vectors and the plurality of second embedding vectors.

[0015] In one embodiment, the fourth loss function may be applied by calculating weights based on the difference in the time of creation between the unstructured medical data and the structured medical data pairs, thereby giving higher importance to data pairs that are temporally closer.

[0016] In one embodiment, the step of further training the pre-trained language model comprises: generating first embedding vectors based on the unstructured medical data; generating a plurality of second embedding vectors based on the structured medical data; generating a fusion embedding vector based on the generated plurality of second embedding vectors; and further training the pre-trained language model based on a fifth loss function calculated based on the generated fusion embedding vector and the first embedding vector, wherein the unstructured medical data and the structured medical data have different modalities, and the structured medical data may include medical data of different modalities.

[0017] In one embodiment, the step of acquiring the structured medical data that matches the unstructured medical data may include acquiring the structured medical data that matches the unstructured medical data based on a representative time of the times when the medical data of different modalities included in the structured medical data was acquired.

[0018] A computer program stored on a computer-readable storage medium according to one embodiment of the present disclosure for realizing the aforementioned tasks is disclosed. When the computer program is executed on one or more processors, the one or more processors are made to perform the following operations, the operations may include an operation of acquiring unstructured medical data, an operation of acquiring structured medical data that matches the unstructured medical data, and an operation of additionally training a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data.

[0019] A computing device according to one embodiment of the present disclosure for realizing the aforementioned task is disclosed. The device comprises at least one processor and a memory, and the at least one processor may be configured to acquire unstructured medical data, acquire structured medical data that matches the unstructured medical data, include the unstructured medical data as input data, and further train a pre-trained language model based on training data including the structured medical data as correct answer data. Effects of the invention

[0021] The present disclosure provides a method for converting unstructured medical data into structured medical data that can be used as input to an artificial intelligence model based on a language model, thereby having the effect of improving the prediction accuracy and clinical validity of the artificial intelligence model.

[0022] In addition, when structured medical data is used to train artificial intelligence models, the labeling accuracy of the training data set is improved, which has the effect of improving training stability and generalization performance.

[0023] In addition, by making various nursing information accessible to medical staff in a structured form, it has the effect of enabling medical staff to make more efficient and rapid clinical decisions. Brief explanation of the drawing

[0025] FIG. 1 is a block diagram of a computing device performing operations according to one embodiment of the present disclosure. FIG. 2 is a schematic diagram showing a neural network according to one embodiment of the present disclosure. FIG. 3 is a flowchart for explaining a method for structuring unstructured medical data based on a language model according to one embodiment of the present disclosure. FIG. 4 is a conceptual diagram illustrating a binary classification learning method according to one embodiment of the present disclosure. FIG. 5 is a conceptual diagram illustrating a regression learning method according to one embodiment of the present disclosure. FIG. 6 is a conceptual diagram illustrating a self-supervised representation learning method according to one embodiment of the present disclosure. FIG. 7 is a conceptual diagram illustrating a comparative learning method according to one embodiment of the present disclosure. FIG. 8 is a conceptual diagram illustrating a comparative learning method according to another embodiment of the present disclosure. FIG. 9 is a diagram illustrating the schematic flow of a framework according to one embodiment of the present disclosure. FIG. 10 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Specific details for implementing the invention

[0026] Various embodiments are now described with reference to the drawings. In this disclosure, various descriptions are provided to facilitate understanding of the disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0027] As used in this disclosure, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0028] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural inclusive substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases.

[0029] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this disclosure and claims should generally be interpreted to mean “one or more.”

[0030] And, the term “at least one of A or B” should be interpreted to mean “a case including only A,” “a case including only B,” or “a case combined with the composition of A and B.”

[0031] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as going beyond the scope of this disclosure.

[0032] Description of the presented embodiments is provided to enable those skilled in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein. The present disclosure should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0033] Meanwhile, the terms “image” or “image data” used throughout the detailed description and claims of the present invention refer to multidimensional data composed of discrete image elements (e.g., pixels in a two-dimensional image), and in other words, refer to a visible object (e.g., displayed on a video screen) or a digital representation of such object (e.g., a file corresponding to the pixel output of a CT, MRI detector, etc.).

[0034] For example, "image" or "video" may be a medical image of a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art of the present invention. The image does not necessarily have to be provided in a medical context and may be provided in a non-medical context, such as, for example, X-ray imaging for security screening.

[0035] Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a collective term for various standards used for digital image representation and communication in medical devices, and the DICOM standard is published by a joint committee formed by the American College of Radiological Engineers (ACR) and the National Electrical Manufacturers Association (NEMA).

[0036] In addition, throughout the detailed description and claims of the present invention, 'PACS (Picture Archiving and Communication System; medical image storage and transmission system)' is a term referring to a system that stores, processes, and transmits images in accordance with the DICOM standard, and medical image images acquired using digital medical imaging equipment such as X-ray, CT, and MRI can be stored in DICOM format and transmitted to terminals inside and outside the hospital via a network, and interpretation results and medical records may be added thereto.

[0038] FIG. 1 is a block diagram of a computing device performing operations according to one embodiment of the present disclosure.

[0039] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other components for configuring a computing environment, and only some of the disclosed components may constitute the computing device (100).

[0040] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0041] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform operations for training a neural network model. The processor (110) may perform calculations for training a neural network model, such as processing input data for training in deep learning (DL), extracting features from input data, calculating errors, and updating weights of the neural network model using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the training of the neural network model. For example, a CPU and a GPGPU can work together to process the training of a neural network model and the classification of data using the neural network model. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the training of a neural network model and the classification of data using the neural network model. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be in a form executable on a CPU, a GPGPU, or a TPU.

[0042] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0043] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0044] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as PSTN (Public Switched Telephone Network), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0045] In addition, the network unit (150) presented according to one embodiment of the present disclosure may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA), and other systems.

[0046] In one embodiment, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a Personal Area Network (PAN) or a Local Area Network (LAN). In addition, the network may be a known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. The technologies described in this disclosure may be used in other networks as well as the networks mentioned above.

[0048] FIG. 2 is a schematic diagram showing a neural network according to one embodiment of the present disclosure.

[0049] Throughout this disclosure, neural network, neural network model, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

[0050] In this case, within the neural network model, one or more nodes connected via links can form a relative relationship between an input node and an output node. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the relationship between an input node and an output node can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0051] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight (wherein, parameters and weights may be used interchangeably throughout this disclosure). The weight may be variable and may be varied by a user or an algorithm to perform desired functions of the neural network model (e.g., posture estimation, emotion estimation, and pain prediction). For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value may be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0052] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks may be recognized as different from each other.

[0053] A neural network can be composed of a set of one or more nodes. A subset of nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0054] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0055] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0056] A Deep Neural Network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. A computing device can use a Deep Neural Network to identify the latent structures of data. That is, a computing device can use a Deep Neural Network to identify the latent structures of photos, text, videos, voice, or music (e.g., what objects are in a photo, what the content and emotions of the text are, what the content and emotions of the voice are, etc.). Deep Neural Networks may include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Auto Encoders, Generative Adversarial Networks (GAN), Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Q Networks, U Networks, and Siamese Networks. The description of the deep neural network described above is merely an example and the present disclosure is not limited thereto.

[0057] In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to the input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be placed between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrically with respect to the input layer). The autoencoder may perform non-linear dimensionality reduction. The number of input and output layers may correspond to the dimension after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layers included in the encoder may have a structure in which the number of nodes decreases as it moves away from the input layer. Since the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and the decoder) may not transmit a sufficient amount of information if it is too small, it may be maintained at a certain number or more (e.g., more than half of the input layer).

[0058] A neural network model including a neural network can be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training of a neural network model may be a process of applying knowledge to the neural network model to perform a specific action.

[0059] Neural network models can be trained to minimize the error in their output. The training of a neural network model may involve a process of repeatedly inputting training data into the model, calculating the error between the model's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer to reduce the error. In the case of supervised learning, training data with correct answers labeled for each data point (i.e., labeled training data) is used, whereas in the case of unsupervised learning, the correct answers may not be labeled for each training data point. For example, in the case of supervised learning for data classification, the training data may be in the form where each data point is labeled with a category. Labeled training data is input into the neural network model, and the error can be calculated by comparing the model's output (category) with the labels of the training data.

[0060] As another example, in the case of unsupervised learning for data classification, errors can be calculated by comparing the input training data with the output of the neural network model. The calculated errors are backpropagated within the neural network model (i.e., from the output layer to the input layer), and this backpropagation can update the connection weights of each node in each layer of the neural network model. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network model's calculation of the input data and the backpropagation of errors can constitute an epoch. The learning rate can be applied differently depending on the number of iterations of the neural network model's epoch. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0061] In the training of neural network models, training data is generally a subset of real-world data (i.e., the data intended to be processed using the trained neural network model). Consequently, there may exist epochs where the error on the training data decreases, but the error on the real data increases. Overfitting refers to a phenomenon where the model fits excessively to the training data, leading to an increase in the error on real-world data. For example, a neural network model trained solely on yellow cats failing to recognize non-yellow cats as cats can be a form of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), or batch normalization can be applied.

[0063] The present disclosure relates to a technology for converting unstructured medical data into structured medical data using a Large Language Model (LLM), and in particular aims to provide an efficient learning framework and operation method for generating structured data from freely written text, such as nursing records.

[0065] FIG. 3 is a flowchart for explaining a method for structuring unstructured medical data based on a language model according to one embodiment of the present disclosure.

[0066] Referring to FIG. 3, the computing device (100) can acquire structured medical data (S110). Structured medical data is medical data that is periodically recorded in a structured format in a hospital electronic medical record (EMR) system, and consists of standardized field names and values, so it can be directly used as a target label for model learning without separate preprocessing.

[0067] A computing device (100) can acquire unstructured medical data that matches the structured medical data (S120). Unstructured medical data refers to medical information recorded in natural language or a free form without a standardized format, and may refer to data that is difficult for a computer to process or analyze directly. For example, unstructured medical data may be in the form of text, such as nursing records, diagnostic reports, consultation records, medical opinions, surgical records, or discharge summaries. However, it is not limited to this, and unstructured medical data may be in the form of voice, such as a doctor's oral records or conversations with a patient, or in the form of images, such as medical images without metadata.

[0068] Unstructured medical data can be generated in parallel with structured medical data at the same or similar timeframes. For example, at the time endotracheal intubation (ETI) is performed, documents such as nursing records or reports regarding the procedure are generated in the form of unstructured text, while information such as the procedure code and the time of execution is stored in the electronic medical record system in the form of structured fields. Therefore, such unstructured and structured medical data can be automatically matched data pairs as they are generated in temporally synchronized.

[0069] The computing device (100) can additionally train a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data (S130).

[0070] In the present disclosure, additional learning may refer to a process of additionally training a pre-trained language model (LLM) based on relatively small data or data directly related to a target task, after the pre-trained language model has been pre-trained based on large text data. For example, additional learning may include fine-tuning to additionally train the model with small data specialized for a specific task, and may include a process of updating the weights of the model.

[0071] In the present disclosure, fine-tuning may include various methodologies. For example, fine-tuning may include full fine-tuning, partial fine-tuning, LoRA (Low-Rank Adaptation), Adapter, and RLHF (Reinforcement Learning from Human Feedback).

[0072] The above additional learning can be performed using a supervised learning method by utilizing training data that includes correct answer data.

[0073] The above-mentioned pre-trained language model may be a model generated by additionally training a large-scale language model (LLM) using an autoregressive next-token prediction method that sequentially predicts the next token based on previous tokens based on the above-mentioned unstructured medical data.

[0074] Token prediction in autoregression methods can mean that a model predicts the next element (e.g., word, pixel, data point) based on the previous element of a sequence. In the context of Large Language Models (LLM), it can mean that a model predicts the next word in a sentence based on words that have already been generated.

[0075] According to one embodiment of the present disclosure, an additionally trained language model can receive unstructured medical data as input and output structured medical data. The structured medical data can be utilized as input to an artificial intelligence model to assist a physician in clinical decision-making, thereby improving the accuracy of the prediction. For example, the artificial intelligence model may include a lesion detection model, a disease diagnosis model, an anomaly detection model, and a medical event prediction model such as cardiac arrest.

[0077] FIG. 4 is a conceptual diagram illustrating a binary classification learning method according to one embodiment of the present disclosure.

[0078] Referring to FIG. 4, a computing device (100) can train a classification model to predict whether a clinical event occurs based on unstructured medical data. The clinical event may include whether tracheal intubation is performed, whether a high-flow nasal cannula (HFNC) is performed, whether continuous renal replacement therapy (CRRT) or hemodialysis (HD) is performed, whether cardiopulmonary resuscitation (CPR) is performed, whether a Do Not Resuscitate (DNR) order is issued, whether the patient is transferred to an Intensive Care Unit (ICU) or readmitted, etc. The occurrence of the clinical event may be structured medical data.

[0079] For example, a high-flow nasal cannula is a treatment that supplies a high flow of oxygen to patients with hypoxia, and continuous renal replacement therapy may be a 24-hour continuous dialysis treatment performed on patients with acute renal failure.

[0080] The computing device (100) can learn the classification model by additionally learning the pre-trained language model (230) based on a loss function for binary classification of whether a clinical event has occurred. For example, the language model (230) may include a large-scale language model (LLM).

[0081] Specifically, the computing device (100) can set a time window defined around a time point (T0) in which tracheal intubation is performed during a clinical event. That is, the time window may represent a time interval from a time point (T-1) before the time point (T0) to a time point (T+1) after the time point (T0). The computing device (100) can obtain structured medical data (220) regarding whether tracheal intubation was performed and unstructured medical data (210) related to the structured medical data (220) within the set time window (T-1 to T+1).

[0082] In order to obtain unstructured medical data (210) related to the above structured medical data (220), the computing device (100) can select data semantically associated with "tracheal intubation" from the unstructured medical data (210). For example, the computing device (100) can select unstructured medical data (210) related to clinical events from the unstructured medical data by utilizing keyword-based filtering, named entity recognition, embedding-based semantic similarity, or a neural network-based classifier.

[0083] The computing device (100) may provide the selected unstructured medical data (210) to a pre-trained language model (230). The language model (230) may process the selected unstructured medical data (210) to generate a representation vector. The computing device (100) may obtain a binary classification result through a classifier (240) based on the representation vector, which is the output of the language model (230). For example, the binary classification result may include information regarding whether a clinical event has occurred.

[0084] For example, the classifier (240) may be a neural network model based on a Multi-Layer Perceptron (MLP). The classifier (240) may output a binary classification result based on an input vector, and the binary classification result may indicate whether or not intubation is performed. For example, depending on the binary classification result, "1" may indicate that tracheal intubation is performed, and "0" may indicate that tracheal intubation is not performed.

[0085] The computing device (100) can calculate a first loss function between the binary classification result and the structured medical data (220). For example, the first loss function may be a cross-entropy loss function. The computing device (100) can update the language model (230) by performing backpropagation based on the calculated loss value. According to embodiments, the computing device (100) can update the language model (230) and the classifier (240) together by performing backpropagation based on the calculated loss value.

[0086] According to embodiments of the present disclosure, a computing device (100) can utilize structured medical data recorded in an electronic medical record (EMR) as ground truth for learning without manual labeling. Thus, a prediction model based on unstructured medical data can be effectively constructed.

[0088] FIG. 5 is a conceptual diagram illustrating a regression learning method according to one embodiment of the present disclosure.

[0089] Referring to FIG. 5, a computing device (100) can train a regression model to predict continuous clinical variables based on unstructured medical data. Here, continuous clinical variables are variables expressed as continuous numerical values ​​and may include vital signs, blood test results, and length of stay. The continuous clinical variables may be structured medical data. For example, vital signs may include indicators such as body temperature, blood pressure, pulse, respiratory rate, and blood pressure.

[0090] The computing device (100) can further train the pre-trained language model (330) based on a loss function for predicting continuous clinical variables. For example, the language model (330) may include a large-scale language model (LLM).

[0091] Specifically, the computing device (100) can set a time window (T-1 to T+1) defined around a time point (T0) at which blood pressure is measured among vital signs. For example, the measured blood pressure may be systolic blood pressure (SBP). The computing device (100) can obtain structured medical data (320) regarding the value of the systolic blood pressure (SBP) measured within the set time window (T-1 to T+1) (e.g., 200 mmHg) and unstructured medical data (310) related to the structured medical data (320).

[0092] The computing device (100) can provide the acquired unstructured medical data (310) to a pre-trained language model (330). The language model (330) can process the unstructured medical data (310) to generate a representation vector. The computing device (100) can input the representation vector, which is the output of the language model (330), into a regression model (340).

[0093] For example, the regression model (340) may be a multiple perceptron (MLP)-based neural network model. The computing device (100) may predict a continuous clinical variable through the regression model (340) based on the representation vector. The regression model (340) may be a neural network model for predicting a continuous clinical variable based on an input vector, and the continuous clinical variable may be a numerical value for systolic blood pressure.

[0094] The computing device (100) can calculate a second loss function between the continuous clinical variable and the structured medical data (320) which are the prediction results. For example, the second loss function may be the Mean Square Error (MSE). The computing device (100) can update the language model (330) by performing backpropagation based on the calculated loss value. According to embodiments, the computing device (100) can update the language model (330) and the regression model (340) together by performing backpropagation based on the calculated loss value.

[0096] FIG. 6 is a conceptual diagram illustrating a self-supervised representation learning method according to one embodiment of the present disclosure.

[0097] Referring to FIG. 6, a computing device (100) can learn a model for performing cross-modal representation learning based on unstructured medical data (410) and structured medical data (420). Here, cross-modal may refer to the interaction between different modalities, such as text and image, or text and video. Structured medical data may be time-series medical data that is measured repeatedly at specific time intervals or at consecutive points in time. For example, the time-series medical data may include an electrocardiogram (ECG), a chest X-ray (CXR), vital signs, and blood test results.

[0098] Meanwhile, unstructured medical data (410) may be text data, and structured medical data (420) may be image data such as a chest X-ray. That is, unstructured medical data (410) and structured medical data (420) have different modalities and may exist in a heterogeneous representation space. In order to overcome the differences between modalities and learn the interrelationships, it is necessary to embed the text data and image data with an encoder suitable for their respective characteristics and then align them into a shared embedding space.

[0099] The Bootstrap Your Own Latent (BYOL) method to address this is a self-supervised representation learning method that can learn representations without contrastive loss. BYOL can include an online network and a target network.

[0100] The online network is a network where learning takes place and may include a projector (440) and an encoder (450). The projector (440) may be a multi-perceptron (MLP)-based neural network structure that non-linearly transforms the output representation vector of the language model (430) and maps it to a more aligned latent representation space. Meanwhile, according to another embodiment of the present disclosure, the online network may include a projector of a different type other than the projector (440). For example, it may include a projector (not shown) that non-linearly transforms the output of the encoder (450), or an integrated projector (not shown) that receives the output of the language model (430) and the output of the encoder (450) in parallel and non-linearly transforms them.

[0101] The target network may be a network that follows an online network using a momentum update method, rather than learning parameters directly through backpropagation. The target network includes an encoder (450) and may additionally include a “projector (440) for the language model (430),” a “projector (not shown) for the encoder (450),” or an “integrated projector (not shown) that integrally post-processes the outputs of the language model (430) and the encoder (450)” for learning stability or generalization performance.

[0102] As illustrated in FIG. 6, the computing device (100) can generate a first embedding vector based on the unstructured medical data (410) and generate a second embedding vector based on the structured medical data (420). The computing device (100) can further train the pre-trained language model (430) based on a third loss function calculated based on the similarity between the first embedding vector and the second embedding vector. For example, the language model (430) may include a large-scale language model (LLM).

[0103] Specifically, the computing device (100) can acquire structured medical data (420) for a chest X-ray existing within a time window (T-1 to T+1) defined around the time point (T0) at which the time-series medical data was acquired, and unstructured medical data (410) associated with the structured medical data (420).

[0104] Here, unstructured medical data (410) and structured medical data (420) can be converted into vector representations through a modality-specific encoder, respectively. That is, the computing device (100) can provide the acquired unstructured medical data (410) to a pre-trained language model (430). The language model (430) can process the unstructured medical data (410) to generate a representation vector. The computing device (100) can convert the representation vector, which is the output of the language model (430), into a common embedding space or a representation space capable of similarity calculation through a post-processing conversion module such as a projector (440).

[0105] Additionally, the computing device (100) may provide the acquired structured medical data (420) to an encoder (450). For example, if the structured medical data (420) is image data such as a chest X-ray, the encoder (450) may include a structure such as a Residual Network (ResNet), a Densely Connected Convolutional Network (DenseNet), or a Vision Transformer (ViT) as an image encoder for processing the image data.

[0106] The encoder (450) can be configured based on CNN or Transformer and can process other time-series medical data besides images. The encoder (450) can process structured medical data (420) to generate a representation vector.

[0107] According to the present invention, a computing device (100) can calculate a third loss function based on the similarity between a first embedding vector, which is the output of a projector (440) connected to a language model (430), and a second embedding vector, which is the output of an encoder (450). The third loss function can be designed to minimize the distance between the two vectors, and the language model (430) can be trained based on this. Additionally, a self-supervised learning loss of the BYOL method can be applied as shown in [Equation 1]. For example, the third loss function may be a self-supervised learning loss function of the BYOL method.

[0108] [Mathematical Formula 1]

[0109]

[0110] Here, x is unstructured medical data, x' is structured medical data, f(·) encodes the input into a representation space, g(·) and q(·) are multiperceptron (MLP)-based projectors that project into a latent embedding space suitable for BYOL learning, and sg(·) blocks backpropagation to prevent the model from being trained. represents the L2 squared distance, which is a loss function used to measure similarity between two vectors.

[0111] According to embodiments, the loss function may be applied by calculating weights based on the difference in the time of creation between pairs of unstructured medical data (410) and structured medical data (420), thereby giving higher importance to data pairs that are temporally closer. The weighted BYOL loss function considering the time difference is as follows.

[0112] [Mathematical Formula 2]

[0113] ,

[0114]

[0115] Here, w(Δt) may represent a function that weights the similarity loss according to the time difference (Δt) between structured medical data and unstructured medical data. A shorter time interval assigns higher similarity, and τ is a scaling parameter that can be a hyperparameter for controlling time sensitivity.

[0116] The computing device (100) can update the language model (430) by backpropagating based on a loss function.

[0117] Meanwhile, according to another embodiment of the present disclosure, if the architecture in which the projector is placed is changed, the third loss function may also be changed. For example, if a projector (not shown) that non-linearly transforms the output of the encoder (450) is included instead of the projector (440), the third loss function may be calculated based on the similarity between the “output embedding vector of the language model (430)” and the “output embedding vector of the projector (not shown) connected to the encoder (450).” Additionally, if a projector (not shown) that non-linearly transforms the output of the encoder (450) is further included in addition to the projector (440), the third loss function may be calculated based on the similarity between the “output embedding vector of the projector (440) connected to the language model (430)” and the “output embedding vector of the projector (not shown) connected to the encoder (450).” Additionally, if an integrated projector (not shown) is included that receives the output of the language model (430) and the output of the encoder (450) in parallel and transforms them non-linearly, the third loss function may be calculated based on the similarity between the first output vector of the integrated projector and the second output vector of the integrated projector. Here, the first output vector of the integrated projector corresponds to an embedding vector that non-linearly transforms the output of the language model (430), and the second output vector corresponds to an embedding vector that non-linearly transforms the output of the encoder (450).

[0119] FIG. 7 is a conceptual diagram illustrating a contrastive loss method according to one embodiment of the present disclosure.

[0120] Referring to FIG. 7, a computing device (100) can perform cross-modal representation learning based on unstructured medical data (510) and structured medical data (520). The structured medical data (520) may be time-series medical data, and the time-series medical data may include electrocardiograms, chest X-rays, vital signs, blood test results, etc. Also, the structured medical data (420) may be unimodal medical data.

[0121] The computing device (100) may generate a plurality of first embedding vectors based on the unstructured medical data (510) and generate a plurality of second embedding vectors based on the structured medical data (520). The computing device (100) may further train the pre-trained language model (530) based on a loss function calculated based on a similarity matrix between the plurality of first embedding vectors and the plurality of second embedding vectors. For example, the language model (530) may include a large-scale language model (LLM).

[0122] Specifically, the computing device (100) can acquire structured medical data (520) for a chest X-ray existing within a time window (T-1 to T+1) defined around a time point (T0) at which time-series medical data is acquired, and unstructured medical data (510) associated with said structured medical data (520).

[0123] Here, unstructured medical data (510) and structured medical data (520) can be converted into vector representations through encoders specialized for each modality. That is, the computing device (100) can provide the acquired unstructured medical data (510) to a pre-trained language model (530). The language model (530) can process the unstructured medical data (510) to generate a representation vector.

[0124] Additionally, the computing device (100) may provide the acquired structured medical data (520) to an encoder (540). The encoder (540) may be an image encoder for processing image data. The encoder (540) may process the structured medical data (520) to generate a representation vector.

[0125] At this time, a fourth loss function that considers time differences may be applied so that the time interval between different modalities does not have a negative effect on learning. The fourth loss function may be a weighted contrastive loss function and may be calculated based on the similarity matrix and time information.

[0126] The corresponding loss function can be defined as follows.

[0127] [Mathematical Formula 3]

[0128] ,

[0129]

[0130] Here, y i is an indicator representing whether the input pair is a positive pair; it can be set to 1 if the data is from the same patient at the same time point, and 0 otherwise. D(·,·) is the distance function between two embedding vectors, and Euclidean distance, etc., can be used. w( ) is the difference in generation time between input pairs It is a weighting function determined according to, where m represents the contrastive margin.

[0131] This loss function can reflect data discrepancies in the actual clinical time flow and increase the stability of representation learning by applying differential weights based on the time difference even between positive pairs.

[0132] As illustrated in the drawing, unstructured medical data (510) is embedded through a language model (530), and structured medical data (520) having a different modality from the unstructured medical data (510) can be embedded through an encoder (540) specialized for a different modality from the language model (530).

[0133] The embedding vectors generated through the language model (530) and encoder (540), respectively, constitute a similarity matrix (550), and similarity can be calculated for all positive pairs and negative pairs. Based on this similarity matrix (550), a weighted contrast loss function can be applied to perform model training such that the similarity of positive pairs increases and the similarity of negative pairs decreases. Additionally, considering the time difference between different modalities, higher importance can be assigned to data pairs that are temporally closer.

[0134] The computing device (100) can update the language model (530) based on the loss function, and, depending on the embodiment, the encoder (540) can also be updated together.

[0136] FIG. 8 is a conceptual diagram illustrating a comparative learning method according to another embodiment of the present disclosure.

[0137] Referring to FIG. 8, a computing device (100) can generate a model that performs cross-modal representation learning based on unstructured medical data (610) and structured medical data (620-1, 620-2, 620-3). The structured medical data (620-1, 620-2, 620-3) may be time-series medical data, and the time-series medical data may include electrocardiogram signals, chest X-ray images, vital signs, and blood test results. For example, the first structured medical data (620-1) may correspond to a chest X-ray, the second structured medical data (620-2) may correspond to a blood test result, and the third structured medical data (620-3) may correspond to an electrocardiogram signal.

[0138] Meanwhile, unstructured medical data (610) and structured medical data (620-1, 620-2, 620-3) have different modalities and may exist in heterogeneous representation spaces. Also, each of the structured medical data (620-1, 620-2, 620-3) may include medical data of different modalities.

[0139] The computing device (100) can generate a first embedding vector through a language model (630) based on the unstructured medical data (610), and in this process, convert it into a form that can be aligned in a common representation space through a projector (640). A plurality of second embedding vectors can be generated based on structured medical data (620-1, 620-2, 620-3). For example, the first structured medical data (620-1) can be converted into a second-1 embedding vector through a first encoder (650-1), the second structured medical data (620-2) can be converted into a second-2 embedding vector through a second encoder (650-2), and the third structured medical data (620-3) can be converted into a second-3 embedding vector through a third encoder (650-3).

[0140] Specifically, the computing device (100) is at the time (T01, T02, when each of the structured medical data (620-1, 620-2, 620-3) is acquired. A time window defined around the point in time can be set based on T03). Each modality has a unique measurement period, and there may be a time difference between points in time.

[0141] The computing device (100) is at the time (T01, T02, when medical data of different modalities included in the structured medical data (620-1, 620-2, 620-3) is respectively acquired. Based on T03), a representative time window (Average T) can be set. For example, the computing device (100) may have a central point in time (T01, T02, of the above time windows. Based on T03), an average time point can be calculated, and a time window defined as an interval including the calculated average time point can be set as the representative time window (Average T).

[0142] And, the computing device (100) exists within a representative time window (Average T) and can acquire unstructured medical data (610) associated with each structured medical data (620-1, 620-2, 620-3). The structured medical data (620-1, 620-2, 620-3) is converted into a vector representation through an encoder (650-1, 650-2, 650-3) specialized for each modality, and the unstructured medical data (610) can be embedded through a pre-trained large-scale language model (630).

[0143] The computing device (100) can generate a fusion embedding vector based on a plurality of second embedding vectors corresponding to structured medical data (620-1, 620-2, 620-3). For example, the fusion model (660) can generate a fusion embedding vector by concatenating the embedding vectors of each modality and converting them into an integrated representation through a multilayer perceptron (MLP). As another example, the fusion model (660) can learn semantic interactions or associations between modalities by applying a cross-attention or self-attention mechanism and generate a fusion embedding vector therefrom.

[0144] At this time, the fused embedding vector can be utilized to enable representation vectors extracted from different modalities, such as text, images, and biosignals, to be aligned or compared within a shared representation space.

[0145] The computing device (100) can further train the pre-trained language model (630) based on a loss function calculated based on the generated fusion embedding vector and the first embedding vector.

[0146] According to the present invention, a computing device (100) can learn a language model (630) by a fifth loss function calculated based on the similarity between an embedding vector embedded through a pre-trained large-scale language model (LLM) and a fused embedding vector output through a modality-specific encoder, which can be designed to minimize the distance between the two vectors.

[0148] FIG. 9 is a diagram illustrating the schematic flow of a framework according to one embodiment of the present disclosure.

[0149] Referring to FIG. 9, since the expression method and writing style of unstructured medical data vary depending on the author, there may be differences in the difficulty of standardization even if the same clinical information is included. According to the present disclosure, the input data space can be divided based on the difficulty of standardization, and data processing and learning strategies can be applied differentially accordingly.

[0150] Specifically, the input space (710) is generated from unstructured medical data consisting of clinical text, and examples that are relatively easy to structure can be defined as the input space of easy samples (720). Since these examples can be structured without separate learning by applying a predefined rule-base, the learning efficiency can be improved by excluding the examples from supervised fine-tuning (SFT) based on a large-scale language model (LLM).

[0151] On the other hand, complex or diverse high-difficulty unstructured clinical data is classified into an outer region (730) of an easy example input space (720), and sophisticated learning using a large-scale language model (LLM; 750) can be performed on these. For example, a large-scale language model (LLM; 750) can be supervised fine-tuned (SFT) by selecting few-shot examples or additional training data (740) from the input space of the outer region (730). In this way, by dividing the input space based on difficulty and selectively excluding easy cases from the training target, the complexity of the data that the model must focus on learning can be reduced, and the waste of training resources resulting therefrom can be prevented.

[0152] Additionally, some of the exceptional cases in which a large-scale language model (LLM) repeatedly makes errors during the map fine-tuning (SFT) process can be simply resolved by defining certain rules. To automatically detect such cases, the present disclosure may include an active learning-based procedure that detects and collects cases where the difference between the generated structured data (760) of the large-scale language model (LLM) and the ground truth structured data (770) exceeds a specific threshold. Among the collected exceptional cases, if rule-based processing is possible, the cases may be processed using a rule-based method (780) instead of learning the large-scale language model (LLM).

[0153] Therefore, simple patterns in which the large-scale language model (LLM) continues to make errors can be solved based on rules rather than model learning, thereby reducing the input space that the large-scale language model (LLM) has to process. That is, by gradually expanding the easy example input space (720) (arrow in FIG. 9) to obtain the expanded easy example input space (790), unnecessary data ranges can be excluded, allowing the large-scale language model (LLM) to focus only on difficult cases.

[0154] According to embodiments of the present invention, the present invention provides a technical effect that can improve the efficiency and accuracy of large-scale language model (LLM) learning through the combination of difficulty-based input space partitioning of unstructured medical data, supervised fine-tuning (SFT) learning target selection strategy, active learning-based exception detection, and rule-based processing method.

[0156] According to one embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed.

[0157] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data for solving specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of the device's resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0158] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0159] Non-linear data structures can be structures where multiple data items are connected after a single data item. Non-linear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it can be a data structure that does not form loops in a graph data structure.

[0160] The data structure may include a neural network model. The data structure including the neural network model may be stored on a computer-readable medium. The data structure including the neural network model may include data preprocessed for processing by the neural network model, data input to the neural network model, weights of the neural network model, hyperparameters of the neural network model, data obtained from the neural network model, activation functions associated with each node or layer of the neural network model, loss functions for training the neural network model, etc. The data structure including the neural network model may include any of the components disclosed above. That is, the data structure including the neural network model may be composed of all or any combination thereof, such as data preprocessed for processing by the neural network model, data input to the neural network model, weights of the neural network model, hyperparameters of the neural network model, data obtained from the neural network model, activation functions associated with each node or layer of the neural network model, and loss functions for training the neural network model. In addition to the aforementioned components, the data structure including the neural network model may include any other information that determines the characteristics of the neural network model. Additionally, the data structure may include all forms of data used or generated during the computation process of the neural network model, and is not limited to the foregoing. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. The neural network model may be composed of a set of interconnected computational units that may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network model is composed of at least one node.

[0161] A data structure may include data input to a neural network model. A data structure including data input to a neural network model may be stored on a computer-readable medium. Data input to a neural network model may include training data input during the training process of the neural network model and / or input data input to a neural network model after training is complete. Data input to a neural network model may include pre-processed data and / or data subject to pre-processing. Pre-processing may include a data processing process for inputting data into the neural network model. Accordingly, a data structure may include data subject to pre-processing and data generated by pre-processing. The aforementioned data structure is merely an example, and the present disclosure is not limited thereto.

[0162] The data structure may include weights of a neural network model. (In this disclosure, weights and parameters may be used interchangeably.) The data structure including the weights of the neural network model may be stored on a computer-readable medium. The neural network model may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm to enable the neural network model to perform a desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node may determine the data value output from the output node based on values ​​input to the input nodes connected to the output node and weights set on the links corresponding to each input node. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0163] As an example rather than a limitation, weights may include weights that vary during the training process of the neural network model and / or weights for which the training of the neural network model is complete. Weights that vary during the training process of the neural network model may include weights at the start of an epoch and / or weights that vary during an epoch. Weights for which the training of the neural network model is complete may include weights for which an epoch has been completed. Accordingly, a data structure containing the weights of the neural network model may include a data structure containing weights that vary during the training process of the neural network model and / or weights for which the training of the neural network model is complete. Therefore, the weights and / or combinations of each weight described above are included in the data structure containing the weights of the neural network model. The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0164] Data structures containing the weights of a neural network model may be stored on a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization may be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed for use. A computing device may serialize the data structure to transmit and receive data over a network. A serialized data structure containing the weights of a neural network model may be reconstructed on the same or different computing devices through deserialization. Data structures containing the weights of a neural network model are not limited to serialization. Furthermore, data structures containing the weights of a neural network model may include data structures designed to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree in non-linear data structures). The foregoing is merely an example and the present disclosure is not limited thereto.

[0165] The data structure may include hyperparameters of a neural network model. The data structure including the hyperparameters of the neural network model may be stored on a computer-readable medium. Hyperparameters may be variables that are varied by the user. Hyperparameters may include, for example, a learning rate, a cost function, a loss function, the number of epochs, weight initialization (e.g., setting the range of weight values ​​to be initialized), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structure is merely an example and the present disclosure is not limited thereto.

[0167] FIG. 10 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0168] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0169] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, a person skilled in the art will be well aware that the method of the present disclosure can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0170] The embodiments described in the present disclosure may be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0171] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, but not limiting, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0172] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0173] An exemplary environment (1100) for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit (1104).

[0174] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may include high-speed RAM, such as static RAM, for caching data.

[0175] The computer (1102) includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0176] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, a person skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0177] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module, and / or data may be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0178] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0179] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0180] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0181] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) includes a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means of establishing communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0182] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0183] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0184] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0185] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of this disclosure.

[0186] The various embodiments presented herein may be implemented as methods, devices, or articles manufactured using standard programming and / or engineering techniques. The term "article manufactured" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0187] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0188] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

Claims

Claim 1 A method for structuring unstructured medical data based on a language model, performed by a computing device, comprising: a step of acquiring structured medical data; a step of acquiring unstructured medical data that is time-synchronized and matched with the structured medical data; and a step of further training a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data. Claim 2 delete Claim 3 In claim 1, the method wherein the pre-trained language model is a model generated by additionally training a large-scale language model using an autoregressive next-token prediction method. Claim 4 A method according to claim 1, wherein the step of further training the pre-trained language model comprises: a step of generating a binary classification result regarding whether a clinical event occurs based on the unstructured medical data; a step of calculating a first loss function between the binary classification result and the structured medical data; and a step of further training the pre-trained language model by performing backpropagation based on the first loss function. Claim 5 A method according to claim 1, wherein the step of further training the pre-trained language model comprises: a step of predicting a continuous clinical variable based on the unstructured medical data; a step of calculating a second loss function between the continuous clinical variable and the structured medical data; and a step of further training the pre-trained language model by performing backpropagation based on the second loss function. Claim 6 A method according to claim 1, wherein the step of further training the pre-trained language model comprises: a step of generating a first embedding vector based on the unstructured medical data; a step of generating a second embedding vector based on the structured medical data; and a step of further training the pre-trained language model based on a third loss function calculated based on the similarity between the first embedding vector and the second embedding vector, wherein the unstructured medical data and the structured medical data have different modalities. Claim 7 A method according to claim 1, wherein the step of further training the pre-trained language model comprises: a step of generating a plurality of first embedding vectors based on the unstructured medical data; a step of generating a plurality of second embedding vectors based on the structured medical data; and a step of further training the pre-trained language model based on a fourth loss function calculated based on a similarity matrix between the plurality of first embedding vectors and the plurality of second embedding vectors. Claim 8 In claim 7, the fourth loss function is applied in a manner that calculates weights based on the difference in the time of creation between the pairs of unstructured medical data and structured medical data, thereby assigning higher importance to data pairs that are temporally closer. Claim 9 A method according to claim 1, wherein the step of further training the pre-trained language model comprises: a step of generating first embedding vectors based on the unstructured medical data; a step of generating a plurality of second embedding vectors based on the structured medical data; a step of generating a fusion embedding vector based on the generated plurality of second embedding vectors; and a step of further training the pre-trained language model based on a fifth loss function calculated based on the generated fusion embedding vector and the first embedding vector, wherein the unstructured medical data and the structured medical data have different modalities, and the structured medical data includes medical data of different modalities. Claim 10 In claim 9, the step of acquiring the unstructured medical data that matches the structured medical data comprises the step of acquiring the unstructured medical data that matches the structured medical data based on a representative time of the times when the medical data of the different modalities included in the structured medical data was acquired. Claim 11 A computer program stored on a computer-readable storage medium, wherein when the computer program is executed by at least one processor, the at least one processor is made to perform the following operations, the operations comprising: an operation of acquiring structured medical data; an operation of acquiring unstructured medical data that is time-synchronized and matched with the structured medical data; and an operation of additionally training a pre-trained language model based on training data that includes the unstructured medical data as input data and the structured medical data as correct answer data. Claim 12 A computing device comprising at least one processor; and memory, wherein the at least one processor is configured to acquire structured medical data, acquire unstructured medical data that is time-synchronized and matched with the structured medical data, and further train a pre-trained language model based on training data that includes the structured medical data as input data and the unstructured medical data as correct answer data.

Images