Medical information processing apparatus, method, and program
The medical information processing apparatus and method address factual inaccuracies in radiology report generation by using a classification process and modified loss function to refine the text generation model, improving the accuracy of generated reports.
Patent Information
- Application Number
- JP2025132360
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-08-14
- Filing Date
- 2025-08-07
- Publication Date
- 2026-02-27
Smart Images
Figure 2026034392000001_ABST
Abstract
Description
[Technical Field]
[0001] The embodiments disclosed in the present specification and drawings relate to a medical information processing device, a method, and a program. [Background technology]
[0002] It is known to use machine learning models to generate text. Specifically, it is known to use machine learning to automatically generate radiology reports. Current text generator AI / ML language models tend to have good stylistic and grammatical accuracy, but not factual accuracy. As a result, the generated text may contain clinical content errors, for example, in the form of hallucinations (added, untrue facts), omissions (omitted facts), or misinformation (altered facts). Hallucinations, omissions, and misinformation are clearly undesirable. These issues can arise from using a standard pre-training objective—next token prediction—with a cross-entropy loss function based on linguistic similarity (i.e., a perfect score is achieved when the wording matches perfectly) rather than semantic similarity to the ground truth text (i.e., rewarding similar meaning even when different wording is used).
[0003] To illustrate how generated reports can appear reasonable but contain different types of clinical content errors, the following example is provided. This example of a generated chest radiology report is taken from Dalla Serra, F., Clackett, W., MacKinnon, H., Wang, C., Deligianni, F., Dalton, J., and O'Neil, AQ, 2022, November, "Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations," in AACL-IJCNLP proceedings. The original report states, "There is a large opacity in the left lower lobe. There does not appear to be significant mediastinal shift. No pneumothorax. The cardiac image is not clearly identified or adequately evaluated. The mediastinal contour is very plain." The generated report states, "The PA and lateral views of the chest were reviewed and compared to the previous study. The right pleural effusion has increased in size since the previous study. The left lung is clear. No pneumothorax." Comparing the original report with the generated report, there are obvious clinical errors, such as omission of a shadow in the left lower lung lobe and hallucination of pleural effusion.
[0004] Lovelace and Mortazavi ("Learning to generate clinically coherent chest X-ray reports," in Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1235-1243), 2020, November) proposed supervised training of text generation models using a loss function that models clinical accuracy, called clinical coherence loss. For the generated text, a neural network classifier ("differentiable CheXpert") predicts clinical labels for the presence or absence of 14 pathologies, and then backpropagates the prediction error to a generative model. A differentiable neural network is used as the classifier so that the prediction error can be backpropagated through this model and then through the target generative model during training.
[0005] CheXpert is a large public dataset for chest radiograph interpretation, containing 224,316 chest radiographs from 65,240 patients with associated radiology reports. Each radiology report is labeled using an automatic labeler to extract observations from the free-text radiology report that are used as structured labels. The labels correspond to the status of each of 14 types of observations: positive, negative, unknown, or not mentioned. The labels can be used as ground truth labels for the radiographs, along with forming training data for training supervised or at least semi-supervised models. [Prior art documents] [Non-patent literature]
[0006] [Non-Patent Document 1] Dalla Serra, F., Clackett, W., MacKinnon, H., Wang, C., Deligianni, F., Dalton, J. and O'Neil, AQ, 2022, November, “Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations”, in AACL-IJCNLP proceedings(https: / / aclanthology.org / 2022.aacl-main.47 / ) [Non-patent document 2] Lovelace and Mortazavi(“Learning to generate clinically coherent chest X-ray reports”, in Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1235-1243), 2020, November)(https: / / aclanthology.org / 2020.findings) Summary of the Invention [Problem to be solved by the invention]
[0007] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve fact accuracy in text generation using a machine learning model. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]
[0008] A medical information processing apparatus according to an embodiment includes a processing circuit that performs the following data processing tasks: obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; and using the predicted ground truth label information and the further label information.
[0009] Embodiments will now be described by way of example and not limitation and are illustrated in the following figures: [Brief explanation of the drawings]
[0010] [Figure 1] FIG. 1 is a schematic diagram showing an apparatus according to an embodiment. [Figure 2] FIG. 2 is a flowchart outlining an example of a method according to an embodiment. [Figure 3] FIG. 3 is a flowchart outlining an example of a method according to an embodiment. [Figure 4] FIG. 4 shows the results obtained using the method of FIG. DETAILED DESCRIPTION OF THE INVENTION
[0011] One embodiment provides an apparatus, such as a medical information processing apparatus, comprising: a processing circuit for obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; and performing a data processing task using the predicted ground truth label information and the further label information.
[0012] An embodiment relates to a method including: obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; and performing a data processing task using the predicted ground truth label information and the further label information.
[0013] One embodiment relates to a program that causes a computer to: obtain initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtain predicted ground truth label information by performing a classification process on the ground truth medical text data; obtain further label information including annotated ground truth label information for the ground truth medical text data; and perform data processing tasks using the predicted ground truth label information and the further label information.
[0014] An apparatus 10 according to an embodiment is shown schematically in Figure 1. The apparatus 10 may be referred to as a medical information processing apparatus. For example, medical information may be provided in the form of medical data. The apparatus 10 comprises a computing device 12, which in this example is a personal computer (PC) or workstation. The computing device 12 is connected to a display screen 14 or other display device and one or more input devices 16, such as a computer keyboard and a mouse.
[0015] Computing device 12 receives the initial medical data from data store 20. In alternative embodiments, computing device 12 receives the medical data from one or more additional data stores (not shown) instead of or in addition to data store 20. For example, computing device 12 may receive the medical data from one or more remote data stores (not shown) that may form part of a Picture Archiving and Communication System (PACS) or other information system.
[0016] The computing device 12 provides processing resources for automatically or semi-automatically processing medical text data. The computing device 12 includes a processing unit 30. The processing unit 30 includes a training circuit 32 for training a text generation model and / or a classification model, a text generation circuit 34 for generating text data using the trained text generation model, and a classification circuit 36 for performing classification using the trained classification model.
[0017] The processing unit 30 also includes further data processing circuitry 38 for performing at least one further data processing task. In some embodiments, the data processing circuitry performs fine-tuning of the text generation model and / or the classification model. Such model fine-tuning may also be referred to as further training or refining the model. In some embodiments, the at least one further data processing task includes evaluating the output of the text generation model and / or the classification model. In some embodiments, the at least one further data processing task includes calculating a loss function, for example, as part of the training and / or refinement process. In some embodiments, the training circuitry includes at least a first training circuit for training the text generation model and a second training circuit for training the classification model.
[0018] In this embodiment, the circuits 32, 34, 36, 38 are each implemented in the processing unit 30 by a computer program having computer-readable instructions that are executable to perform the method of the embodiment. However, in other embodiments, the various circuits may be implemented as one or more Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs).
[0019] Computing device 12 also has a hard drive and other components of a PC, including RAM, ROM, a data bus, an operating system including various device drivers, and hardware devices including a graphics card, although such components are not shown in FIG.
[0020] The apparatus of FIG. 1 performs the method of the embodiment shown in FIG.
[0021] Prior to method step 102, a synthetic text generator is obtained. The synthetic text generator corresponds to a procedure based on a text generation model. Thus, obtaining the synthetic text generator included performing a training process to train the text generation model using training circuitry 32. Training the synthetic text generation model may include a general training phase to train the text generation model without specific reference to medical text data. The synthetic text generation model may be a large-scale language model and / or other deep learning derived from language models. A deep learning model may be understood as a model based on a multi-layer artificial neural network. In some embodiments, a pre-trained or at least partially trained synthetic text generator model is obtained. In some embodiments, the synthetic text generator is untrained.
[0022] As a non-limiting example, a large amount of training data is obtained to train a text generation model. It will be understood that the training data may include clinical text training data as well as text data that is not considered medical text data. The text data is used to train embeddings using a standard pre-training task, such as word2vec. The standard pre-training task includes training embeddings using a large text corpus. In an embedding-based representation, text is viewed as a set of word tokens. A word token may be, for example, a single word, a portion of a word, or a group of words. An embedding vector is assigned to each word token. The standard pre-training task trains the embeddings. The standard pre-training task may be performed multiple times with the embeddings updated at each iteration. The embedding vector captures the semantic similarity between word tokens in a multidimensional embedding space. The embedding may be a dense (vector) representation of the semantic space of words. The output of the training process is a trained embedding that includes a vector representation for each of multiple words from the training corpus.
[0023] During initial training, the synthetic text generator may be initially trained on a large amount of data, including non-medical data. The generated synthetic text may be stylistically and grammatically correct, but the output may contain clinical errors. Clinical errors include, but are not limited to, additional fabricated information (referred to as hallucinations), omissions, or inaccurate information. As a result, while the generated output text is likely to be stylistically and grammatically correct (and such text will receive a high score), it is clearly undesirable to output text containing clinical errors.
[0024] In some embodiments, a synthetic text generation model may be initially trained using a first training objective and then further trained or refined using a second training objective, as described below. For example, the first training objective may be based on achieving a level of grammatical similarity between input and output, and training may use a suitable loss function (e.g., a cross-entropy-based loss function). Further training and refinement of the model is based on the second training objective, as described below.
[0025] The text generation model may be, for example, a transformer model. However, any suitable text generation model that generates free text or at least partially unstructured text may be used. The text generator may be a BART model, as described, for example, in Lewis et al. "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension," ACL 2020.
[0026] Additionally, a classifier model is trained, for example, using the training circuitry 32. The classifier may include any differentiable classifier. As a non-limiting example, training the classifier includes receiving an annotated clinical text corpus, for example, including a plurality of annotated radiology reports. The radiology report annotations may include, for example, classification labels. The annotated clinical text corpus is used as training input for training a deep learning model, which in this example is a classifier. The deep learning model is trained to provide an output for each of a plurality of classes. The output includes, for each class, a score indicating the probability or likelihood of that class. The score may be a document-level or sentence-level score. The model output may include, for each class, a word-level attention weight.
[0027] In this embodiment, a differentiable classifier is described. Further embodiments may use a non-differentiable classifier with a modified loss function.
[0028] In one example, a classifier is trained to obtain predictions for a given set of labels for each sentence in a radiology report and / or for the entire radiology report. Each label relates to a corresponding medical term, such as a finding or impression. For example, labels may include hemorrhage and tumor. A classifier may be trained to classify each sentence or report and say whether hemorrhage and tumor are present, respectively.
[0029] For each sentence, each label is classified into one of several certainty classes. The certainty classes include positive, uncertain, and negative. A positive certainty class classification is made when the model determines from the sentence that the finding or impression represented by the label is present in the image. A negative certainty class classification is made when the model determines from the sentence that the finding or impression represented by the label is not present in the image. A uncertain certainty class classification is made when the model determines from the sentence that it is unclear whether the finding or impression is present in the image. For example, a sentence may suggest that a finding or impression may be present, even though there is not a strong enough indication to classify it as positive. In the embodiment below, the labels are from a set of 14 observations.
[0030] In some embodiments, the classifier is substantially as described in Schrempf et al. "Templated text synthesis for expert-guided multi-label extraction from radiology reports" MAKE 2021, 3(2), the contents of which are incorporated herein by reference.
[0031] Returning to the method of FIG. 2 , initial medical data is obtained at stage 102. The initial medical data is obtained from initial medical image data provided together with at least partially unstructured medical text data. In the following embodiment, the initial medical data is obtained from the initial medical images in the form of one or more X-ray images. The X-ray images are provided together with initial ground truth data in the form of a human-generated radiology report, referred to as a ground truth radiology report. The ground truth radiology report is provided together with annotated labels, referred to as ground truth label information. It will be understood that the ground truth radiology report comprises a portion of unstructured medical text data.
[0032] As will be described with reference to Figure 3, in some embodiments, the initial medical data is structured data obtained from an initial medical image. The initial medical text data includes as input an unordered sequence of triplets, which are given in the format of entity->relation->attribute, and are represented as " <h>entity <r>relationship <t>Attributes." Further details regarding the generation of structured data are described with reference to FIG.
[0033] Other suitable initial medical data may be used. In this embodiment, the initial medical data includes medical image data, and the associated text data is, for example, a radiology report corresponding to the image. In an alternative embodiment, the structured medical text data is derived from unstructured medical text data. Thus, the initial medical data is associated with ground truth data.
[0034] At stage 104, synthetic medical text data is generated using a synthetic text generator. The synthetic text generator receives initial medical text data, such as the structured initial medical data described above, and generates free-form synthetic medical text in the form of a synthetic radiology report. The generated synthetic text may be partially unstructured, as it includes one or more unstructured portions. In other embodiments, the synthetic text generator receives medical image data and generates synthetic text that is at least partially unstructured.
[0035] At stage 106, a classification process is performed on the initial ground truth data to obtain predicted ground truth label information. In this embodiment, a classifier is applied to the ground truth radiology reports to obtain predicted ground truth label information.
[0036] At stage 108, further label information is obtained. The further label information includes synthetic text label information obtained by performing further classification processing on the generated synthetic text from step 104. Additionally or alternatively, the further label information includes ground truth label information initially provided with the initial medical data and stored. As described above, the ground truth label information includes annotations for the initial radiology report. In some embodiments, the further label information includes both stored ground truth label information and synthetic label information.
[0037] At stage 110, a further data processing task is performed using at least the obtained predicted ground truth label information and the further label information. In some embodiments, the further data processing task is a model training, refinement, or fine-tuning task. For example, the further data processing task may be a model fine-tuning process that fine-tunes a text generator based on at least the predicted ground truth and the further label information. In some embodiments, the further data processing task may include an error backpropagation process that identifies clinical errors and feeds them back into the classifier and / or text generation model.
[0038] In some embodiments, a further data processing task may be an evaluation task that evaluates the generated synthetic text by comparing the predicted ground truth labels with the initial ground truth labels to obtain a score or other evaluation measure, which may indicate the clinical accuracy of the generated text.
[0039] Figure 3 shows a flow chart outlining a method for training and / or fine-tuning a synthetic text generator according to an embodiment. Figure 3 shows a text generator 202 and a classifier 204 used in the method. The text generator substantially corresponds to the text generator described with reference to Figure 2. The classifier corresponds to the classifier described with reference to Figure 2.
[0040] As explained with reference to FIG. 2, the initial stage is to obtain a trained or at least partially trained text generator 202.
[0041] Initial medical data 206 is obtained. In this embodiment, the initial medical text data is structured medical data. In this embodiment, the initial structured medical data is obtained from X-ray images or other medical image data 201. A triplet extractor is used to obtain the initial structured medical image data in step 203 to obtain the initial medical text data. The medical image data 201 corresponds to the initial ground truth data in the form of a ground truth report 220. In an embodiment, the medical image data 201 corresponding to the ground truth report 220 is obtained from a database.
[0042] The initial medical text data includes as input a sequence of triplets, which may be ordered or unordered. These triplets are given in the format of entity → relation → attribute, and are described as follows: <h>entity <r>relationship <t>The structured data is represented in text as "attributes." Structured data is derived from the initial image using a known image classification model. An example of a known classification model is illustrated in Figure 1 of Dalla Serra, F., Clackett, W., MacKinnon, H., Wang, C., Deligianni, F., Dalton, J. and O'Neil, AQ, 2022, November, "Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations," in AACL-IJCNLP proceedings.
[0043] In this example, the initial medical data is <h>Abnormal <r>The status you have is <t>denial, <h>Atrophy <r>The status you have is <t>affim, <h>In abnormal <r>Progress <t>profound, <h>In bleeding <r>Anatomical structure <t>Located inside the skull, <h>Atrophy <r>As time <t>It is chronic and <h>Bleeding <r>The status you have is <t>"Negative."
[0044] In this embodiment, the ground truth radiology report 220 states, "Against a background of macroscopic generalized atrophy, there is no intracranial hemorrhage or other significant abnormalities."
[0045] In stage 208, the structured initial medical data is provided to a synthetic text generator 202. The synthetic text generator 202 processes the initial medical data and generates a generated medical report 210 as output. The generated medical report 210 includes partially unstructured text data. The medical report 210 is an example of synthetic text data generated by the synthetic text generator 202. In this example, the generated synthetic text includes a portion including unstructured text data stating, "No significant range changes. Specifically, no intracranial hemorrhage or marked background atrophy." Obtaining the generated report from the initial medical image data may be referred to as a two-step pipeline, as described above.
[0046] In stage 212, a first classification process is performed using a differentiable classifier 204. The first classification process includes applying the classifier 204 to the synthetic text 210 to generate first label information 216. The label information obtained by applying the classifier to the synthetic text is called synthetic label information to distinguish it from the label information obtained in stage 218 and the initial ground truth label information 226. In this embodiment, the synthetic label information is expressed by the following mathematical formula (1):
[0047]
number
[0048] At stage 218, a second classification process is performed. The second classification process involves applying the classifier 204 to the initial ground truth data in the form of a ground truth report 220 to generate second label information 222. The second label information 222 is referred to as predicted ground truth label information to distinguish it from ground truth label information that is initially available with the ground truth radiology report, i.e., annotation labels obtained following a human annotation process of the radiology report. In the illustrated example, the predicted ground truth label information is given by the following equation (2):
[0049]
number
[0050] In this embodiment, the first classification process generates first label information 216 including synthetic label information for the generated synthetic medical text 210, and the second classification process generates second label information 222 in the form of ground truth label information 222.
[0051] At stage 224, a comparison process is performed using the predicted ground truth label information 222. In this embodiment, the comparison process uses the predicted ground truth label information 222 to provide clinical content error information. The ground truth label information may be used according to embodiments to obtain clinical error information in several different ways. In this embodiment, the clinical error information is obtained by comparing the predicted ground truth label information 222 obtained by applying a classifier to the initial ground truth report 220 with the initial ground truth label information 226, i.e., human-annotated labels. In this stage, the classifications in the ground truth report 220 are compared with the human-annotated labels.
[0052] In this embodiment, the ground truth label information is L̂ = [bleeding = 0, atrophy = 1, ...]. In this example, the label information for label hemorrhage does not match based on a comparison between the ground truth label information and the predicted ground truth label information. Thus, it is clear that a classifier that predicted there was hemorrhage given the ground truth report as input failed to extract the correct fact. Since the classifier is deemed unreliable, errors based on the classifier prediction should not be fed back to the generator model based on the classifier prediction.
[0053] In some embodiments, clinical content error information for the generated synthetic text may be identified in the generated synthetic text by comparing predicted ground truth labels with true ground truth labels. In some embodiments, predicted labels that do not match the annotated labels indicate clinical content errors. The mismatched labels may represent discrepancies in the clinical content of the generated text. The mismatched labels may represent hallucinations, omissions, or incorrect clinical information. The mismatched labels may represent clinical inaccuracies in the generated synthetic text.
[0054] At stage 228, one or more further data processing tasks are performed. In this embodiment, a model fine-tuning task is performed to modify one or more weights or other model parameters of the text generation model and / or classifier based on the obtained predicted ground truth label information, the ground truth label information, and the synthetic label information. In the example of Figure 3, the error information for the bleeding label indicates that feedback for this label should be discarded and not provided. Further details regarding fine-tuning the text generation model are described with reference to the following embodiments.
[0055] In further detail, step 228 includes an error propagation process in which a loss function for the text generator uses the identified clinical errors to modify and / or refine one or more weights of the text generation model. In the described embodiment, backpropagation may refer to the calculation of one or more suitable terms in the loss function.
[0056] As described below, fine-tuning the text generation model involved calculating a loss function that penalizes possible errors in the generated text. For example, as described below, if a predicted ground truth label does not match an initial ground truth label from the annotated ground truth, then the contribution to the loss function for that label is discarded and / or suppressed in fine-tuning the text generation model.
[0057] As a further example, the clinical coherence loss function may be extended and / or modified to provide feedback from soft labels. Soft label output from a classifier is distinguished from hard labels in that it includes a score representing the probability that the label is correct, as opposed to hard label information representing a predicted label (e.g., following a thresholding process). As an example, rather than applying the consistency loss function to the hard-predicted ground truth labels and the initial annotated ground truth labels, the loss function is applied to the classifier's output logits, as described above. In such an embodiment, a loss function is calculated that has a contribution based on the soft error information.
[0058] As described below, fine-tuning the text generation model involved calculating a loss function that penalizes possible errors in the generated text. For example, as described below, if a predicted ground truth label does not match an initial ground truth label from the annotated ground truth, then the contribution to the loss function for that label is discarded and / or suppressed in fine-tuning the text generation model.
[0059] Further embodiments are described in more detail below. The embodiments below describe a clinical coherence loss function that uses synthetic label information and ground truth label information. In embodiments, the label information includes one or more labels. Each label corresponds to data representing a number of classes. In some embodiments, the label information also includes a corresponding score that represents the probability or likelihood that the data is represented by the label. In some embodiments, the score represents the probability or likelihood that the data belongs to the class of the label. In some embodiments, the label information includes a logit corresponding to each label.
[0060] The following embodiments describe comparisons between label information. In some embodiments, the comparisons include determining common labels, one or more mismatched labels, and / or labels that are present in one label set but not another. Mismatched or missing labels may indicate clinical errors, such as omission, hallucination, or factually incorrect information. In some embodiments, the label information may also include an associated score representing, for example, a probability or likelihood. This label information may be referred to as soft label information. Comparing the soft label information may include calculating one or more comparison functions using the scores, such as a cross-entropy function.
[0061] The text generator g (corresponding to the text generator 202) can be thought of as a function that maps data to output logits. The text labeler f (corresponding to the classifier 204) can be thought of as a function that maps token embeddings to output labels. The output logits can be thought of as the output or intermediate value of the text generator model. The logits can be thought of as the unscaled output of the intermediate layer of the model.
[0062] g: Input text / image / data → Output logit
[0063] f: Embed tokens → Output labels
[0064] During training of a text generator, given input text / image / data (x), the result of the text generator g consists of output logits, which are given by equation (3) below: In equation (3), K is the vocabulary length and m is the length of the target output sequence.
[0065]
number
[0066] Then, the output logit π is input to a Gumbel Softmax function to obtain a set of predicted tokens represented by the following equations (4) and (5).
[0067]
number
[0068]
number
[0069] The Gumbel softmax function is a differentiable function that returns the predicted token, expressed as equation (6) below, represented as an m-number of one-hot vectors with the most likely predicted token marked as 1 and the others marked as 0.
[0070]
number
[0071] These are multiplied by the word embedding matrix (7) below (where d is the dimension of each embedded token) to obtain the predicted token embeddings. The following equation (8) is the encoded representation of the generated text.
[0072]
number
[0073]
number
[0074] Finally, the text labeler f takes as input the predicted token embeddings, expressed in equation (9) below, and returns the predicted probabilities associated with the generated text, expressed in equations (10) and (11) below.
[0075]
number
[0076]
number
[0077]
number
[0078] In this example, the consistency loss function expressed by the following Equation (12) corresponds to the cross entropy loss function (Cross Entropy: CE) expressed by the following Equation (15) between the predicted label expressed by the following Equation (13) and the ground truth label expressed by the following Equation (14).
[0079]
number
[0080]
number
[0081]
number
[0082]
number
[0083] As such, the loss function represents a measure of similarity or agreement between ground truth label information and predicted (synthetic) label information. When the ground truth labels match the synthetic labels, the cross-entropy loss function increases; when the ground truth labels do not match the synthetic labels, the cross-entropy loss function decreases. In this way, the cross-entropy loss function allows for comparison of the synthetic labels with the ground truth labels. The loss function allows for clinical context information to be used to train and / or refine the text generation model.
[0084] In addition to the above method, the present embodiment includes a modified loss function term. In some embodiments, the modified loss function corresponds to the loss function described above (the cross-entropy function using the predicted synthetic label information and the ground truth labels), but is constrained using the predicted ground truth label information.
[0085] As a first example, a classifier is applied to extract ground truth predicted label information, as described with reference to Figures 2 and 3. Using the ground truth report in tokenized form, as shown in Equation (16) below, the predicted ground truth labels can be obtained as shown in Equations (17) and (18) below.
[0086]
number
[0087]
number
[0088]
number
[0089] The predicted label probability is binarized using a threshold (t) to obtain the predicted ground truth label expressed by the following equation (19). Typically, a value of 0.5 is used.
[0090]
number
[0091] Using the predicted ground truth label information, the clinical coherence loss function described above can be modified or extended as follows:
[0092] As a first example, possible errors in the generated text can be discarded. For example, if the predicted ground truth label, expressed in Equation (20) below, disagrees with the initial ground truth label (also called the annotated ground truth label), expressed in Equation (21) below, any contribution to the consistency loss function for that label can be discarded. The loss function can be expressed as Equation (22) below.
[0093]
number
[0094]
number
[0095]
number
[0096] Then, the piecewise function, expressed as Equation (23) below, which is constrained based on the predicted ground truth labels, has a non-zero contribution if and only if the values of the predicted ground truth labels match the values of the ground truth labels, as expressed as Equation (24) below.
[0097]
number
[0098]
number
[0099] In an alternative example, the clinical coherence loss function may be extended and / or modified to provide feedback from soft labels. As an example, rather than applying the consistency loss function with the predicted ground truth labels and the initial annotated ground truth labels, the loss function is applied to the output logits of the text labeler f, as described above.
[0100] In such an embodiment, the predicted logit, expressed as equation (25) below, is obtained from the tokenized synthesized text report (i.e., predicted tokens) expressed as equation (26) below, as shown in equation (27) below.
[0101]
number
[0102]
number
[0103]
number
[0104] Here, Equation (28) below is a text labeler network without a final classification layer. Similarly, the predicted ground truth logits, Equation (29) below, are obtained from the tokenized ground truth report, Equation (30) below.
[0105]
number
[0106]
number
[0107]
number
[0108]
number
[0109] Then, the consistency loss is calculated as follows:
[0110]
number
[0111] In this example, we combine consistency loss with discarding errors, as described above.
[0112] In the above example, the predicted ground truth label information in the form of predicted labels, expressed as Equation (33) below, and the association score, which in the above example is a logit, expressed as Equation (34) below, are used as inputs to the loss function for training / refinement and / or evaluation. We note that some errors may be false positives, where the classifier incorrectly extracts or does not extract a label from the generated text. In addition, there may be a disadvantage in using hard feedback from the correct label class in the form of a one-hot label vector. Such feedback provides limited information about the relationship with other classes.
[0113]
number
[0114]
number
[0115] As presented above, the coherence loss function can be modified / extended. As a first example, possible errors can be discarded. As a second example, the matching tool may provide more granular feedback, for example, by associating labels at the report or sentence level with sentences and words that should contain label information. This provides heterogeneous feedback across reports. As a further example, a more mature clinical fact source may be employed than that used to train the dedicated labeler, for example, using a general entity / attribute extraction model (e.g., a classifier trained to extract UMLS concepts). As a further example, the classifier's predictions on ground truth reports may be treated as true labels for the generated report.
[0116] In the above embodiments, if the initial ground truth information does not match the predicted ground truth label information sufficiently, the cost function is zero or has a low contribution. Matching may be based on hard labels or soft labels. In some embodiments, a matching process is performed to determine which labels do not match. In some embodiments, for soft label information, a matching process is performed to determine a distance measure between different sets of label information.
[0117] Figure 4 shows a table of results from the method described above. The report is generated sentence-wise, and the report is evaluated on 463 sentences. The first row of the table shows the results of the original training task with cross-entropy loss (CE) for next-token prediction. The second row shows the results with the addition of clinical coherence loss (CCL). The third row shows the results of a method according to an embodiment. The loss function of the method includes a modified clinical coherence loss CCL0, where ground truth predictions are discarded or output as zero in the example. The fourth row shows the upper bound.
[0118] The BLEU score is reported (Papineni et al., 2002). This metric indicates the linguistic similarity between the generated report and the ground truth report. The table includes the average micro- and macro-precision, recall, and F1-score. As can be observed from the table, the proposed loss, CE+CCL0, improves these metrics.
[0119] The above metrics are obtained by comparing the detected labels with the ground truth labels. For example, if all ground truth labels are predicted correctly, the recall will be 1. If some extra instances are predicted incorrectly (false positives), it will result in a decrease in precision, which will be lower than 1.
[0120] In the above embodiment, the prediction of the predicted ground truth label information can discard possible errors in the generated text. For example, if the classifier makes an error on the original text, we can assume that the generated text is likely to make an error and ignore the feedback from these labels. In addition, soft label information can be used to provide more useful error information as feedback. This can be based on a distillation technique.
[0121] In addition to, or instead of, fine-tuning a text generation model, further data processing tasks may be performed based on the predicted ground truth label information. As a first non-limiting example, the predicted ground truth label information may be used as part of a matching tool to provide more granular feedback by associating labels (report-level, sentence-level) of the generated synthetic text with sentences / words that should contain the label information, i.e., heterogeneous feedback across reports. In such an embodiment, the same loss function may be used with stronger supervision. In such an embodiment, the error signal is more closely associated with the portion of the text that caused the error.
[0122] As a second non-limiting example, predicted ground truth label information may supplement sources of clinical information for training a classifier, for example, with a general entity / attribute extraction model (e.g., a classifier trained to extract UMLS concepts). In some embodiments, a general classifier (e.g., a dual encoder such as described in "Fast and Effective Biomedical Entity Linking Using a Dual Encoder" by Bhowmik et al.) may be used as a conduit for feeding back error signals from a broader set of concepts than those used during training.
[0123] In a further example, clinical content information may be identified in the synthesized text and the identified portions of the text containing errors may be displayed to the user. In some embodiments, the output of the matching process may discard the identified portions from the generated synthesized text. In some embodiments, the identified portions may be penalized.
[0124] In some embodiments, classification may be performed only on portions of generated text to obtain label information for the portions of text. This may generate label information for more than one portion of text, which may be at the sentence level, for example. The method may include matching portions of text based on their predicted labels and labels of ground truth text.
[0125] Additionally, the predicted ground truth label information can be stored as ground truth data for the generated report. In this manner, the generated report and the predicted ground truth labels can be stored and then used as training data for further unsupervised or at least semi-supervised learning procedures to train further models. In such training processes, the predicted ground truth labels are treated as the correct labels for the generated report.
[0126] One embodiment provides a computer-implemented method comprising a deep learning model for generating outputs, a differentiable labeling tool for extracting structured data from unstructured outputs, a dataset of (un)structured inputs and (un)structured ground truth outputs, and a set of labels describing the fact content of the target outputs. During training, the differentiable labeling tool is applied to both the generated outputs and the ground truth outputs to generate label probabilities. Clinical content errors may be calculated from the label probabilities and fed back to the generator model via backpropagation.
[0127] The clinical content errors may be fed back only for labels whose label predictions on the ground truth report match a set of ground truth labels annotated by human experts.
[0128] A soft clinical content error may be fed back based on the difference between the generated probabilities and the ground truth label probabilities. A soft clinical content error may be fed back without using hard ground truth labels.
[0129] Unsupervised training may be performed based on using automatically generated ground truth labels assigned by a labeling tool.
[0130] Although particular circuits are described herein, in alternative embodiments, the functionality of one or more of these circuits may be provided by a single processing resource or other component, or the functionality provided by a single circuit may be provided by a combination of two or more processing resources or other components. A reference to a single circuit encompasses multiple components that provide the functionality of that circuit, whether or not such components are separate from one another. A reference to multiple circuits encompasses a single component that provides the functionality of those circuits.
[0131] While certain embodiments have been described, these embodiments are presented for illustrative purposes only and are not intended to limit the scope of the invention. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions, and changes in the form of the methods and systems described herein may be made without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms and modifications as fall within the scope of the invention.
[0132] One embodiment relates to a medical information processing device including a processing circuit that performs the following operations: obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; and performing a data processing task using the predicted ground truth label information and the further label information.
[0133] The processing circuit may generate synthetic medical text data from the initial medical data using a synthetic text generator, the generated synthetic medical text data comprising at least partially unstructured data, and the further label information may comprise synthetic label information for the generated synthetic medical text data.
[0134] The classification process may use a differentiable model.
[0135] At least one further said data processing task comprises: a) refining and / or training said synthetic text generator and / or classification process; b) evaluating the output of the synthetic text generator based on at least the predicted ground truth label information; c) identifying clinical content error information for at least a portion of the generated synthetic text; and d) calculating a loss function that depends on at least the predicted ground truth label information; It may include at least one of a), b), c), and d).
[0136] The processing circuitry may use at least the predicted ground truth label information and the further label information to determine a loss function, such as cross-entropy or another function.
[0137] The loss function may have a zero value and / or at least contribute less if at least a portion of the initial ground truth label information does not sufficiently match at least a portion of the predicted ground truth label information.
[0138] The classification process may include applying a classifier, optionally a differentiable classifier, to extract structured information, including label information, from the at least partially unstructured text data.
[0139] The further label information may include ground truth label information for the ground truth data, the processing circuitry may perform a comparison process between the initial ground truth label information and the predicted ground truth label information, and at least one further data processing task is based on at least the comparison process.
[0140] The predicted ground truth label information may include label information generated by applying the classifier to the ground truth medical text data, and the ground truth label information may include human-annotated or other annotated labels.
[0141] The comparison process on the ground truth medical text data may use a pre-trained classifier, such as a differentiable classifier, and the processing circuitry performs a classification process on the composite label information using the same classifier to obtain the composite label information.
[0142] The processing circuitry may use at least the initial ground truth label information to obtain clinical content error information in the generated synthetic text, the clinical content error information including or representing one or more of uncertain information, inaccurate information, or omitted information.
[0143] The ground truth medical text data may include real data, such as a medical report and / or a clinical text corpus including multiple medical reports.
[0144] The label information may include a classification label, and optionally, the label information includes a score associated with the classification label, the score comprising at least one of a logit, a probability, and / or a likelihood.
[0145] At least one further data processing task may include performing an error backpropagation process to feed back obtained clinical content error information to the synthetic text generator based on the predicted ground truth label information.
[0146] The error backpropagation process may be based at least on label classification scores, which represent the probability and / or likelihood of the labels.
[0147] The synthetic text generator may include a deep learning model, such as a transducer model or a neural network model.
[0148] The processing circuitry may store the predicted ground truth label information along with the generated synthetic medical text data as ground truth label information for the generated synthetic medical text data, and may use the predicted ground truth label information and the generated synthetic medical text data as training data for further, at least partially unsupervised, training processes for further models.
[0149] At least one further data processing task may include identifying and discarding one or more portions of the generated synthetic medical text data that contain errors and / or otherwise penalizing the one or more portions of the generated synthetic medical text data.
[0150] The processing circuitry may further match and / or determine one or more relationships between the predicted ground truth label information and portions of the generated synthetic medical text data.
[0151] An embodiment relates to a method including: obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; and performing a data processing task using the predicted ground truth label information and the further label information.
[0152] One embodiment relates to a program that causes a computer to: obtain initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; obtain predicted ground truth label information by performing a classification process on the ground truth medical text data; obtain further label information including annotated ground truth label information for the ground truth medical text data; and perform data processing tasks using the predicted ground truth label information and the further label information.
[0153] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]
[0154] 10 equipment 12 Computing Devices 14 display screens 16 Input Devices 20 Data storage unit 30 Processing equipment 32 Training circuit 34 Text Generation Circuit 36 Classification circuit 38 Data Processing Circuit< / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h> < / t> < / r> < / h>
Claims
1. obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; Obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; performing a data processing task using the predicted ground truth label information and the further label information; A medical information processing device equipped with a processing circuit that performs the above.
2. The processing circuitry generating synthetic medical text data from the initial medical data using a synthetic text generator, wherein the generated synthetic medical text data includes at least partially unstructured data, and the further label information includes synthetic label information for the generated synthetic medical text data. The medical information processing apparatus according to claim 1 , wherein the medical information processing apparatus performs the following steps.
3. the classification process uses a differentiable model; The medical information processing device according to claim 1 .
4. At least one further said data processing task comprises: a) refining and / or training said synthetic text generator and / or classification process; b) evaluating the output of the synthetic text generator based on at least the predicted ground truth label information; c) identifying clinical content error information for at least a portion of the generated synthetic text; and d) calculating a loss function that depends on at least the predicted ground truth label information; The present invention includes at least one of a), b), c), and d), The medical information processing device according to claim 2 .
5. the processing circuitry uses at least the predicted ground truth label information and the further label information to determine a loss function, such as cross-entropy or other function. The medical information processing device according to claim 1 .
6. the loss function has a zero value and / or at least contributes less if at least a portion of the initial ground truth label information does not sufficiently match at least a portion of the predicted ground truth label information; The medical information processing device according to claim 5 .
7. the classification process includes applying a classifier, optionally a differentiable classifier, to extract structured information, including label information, from the at least partially unstructured text data; The medical information processing device according to claim 2 .
8. the further label information includes ground truth label information for ground truth data; the processing circuitry performs a comparison process between initial ground truth label information and the predicted ground truth label information; at least one further data processing task is based at least on the comparison process; The medical information processing device according to claim 7 .
9. the predicted ground truth label information includes label information generated by applying the classifier to the ground truth medical text data; the ground truth label information includes human-annotated or other annotated labels; The medical information processing device according to claim 8 .
10. the comparison to the ground truth medical text data uses a pre-trained classifier, such as a differentiable classifier; the processing circuit performs a classification process on the composite label information using the same classifier to obtain the composite label information. The medical information processing device according to claim 8 .
11. the processing circuitry uses at least initial ground truth label information to obtain clinical content error information in the generated synthetic text; the clinical content error information includes or represents one or more of uncertain information, inaccurate information, or omitted information; The medical information processing device according to claim 2 .
12. the ground truth medical text data comprises actual data, such as a medical report and / or a clinical text corpus comprising a plurality of medical reports; The medical information processing device according to claim 1 .
13. the label information includes classification labels, and optionally, the label information includes scores associated with the classification labels, the scores including at least one of logits, probabilities, and / or likelihoods; The medical information processing device according to claim 1 .
14. At least one further data processing task includes performing an error backpropagation process to feed back obtained clinical content error information to the synthetic text generator based on the predicted ground truth label information. The medical information processing device according to claim 2 .
15. The error backpropagation process is performed based at least on label classification scores representing the probability and / or likelihood of the labels. The medical information processing device according to claim 14.
16. the synthetic text generator comprises a deep learning model, such as a transducer model or a neural network model; The medical information processing device according to claim 2 .
17. the processing circuitry stores the predicted ground truth label information together with the generated synthetic medical text data as ground truth label information for the generated synthetic medical text data, and optionally uses the predicted ground truth label information and the generated synthetic medical text data as training data for a further, at least partially unsupervised, training process for a further model. The medical information processing device according to claim 2 .
18. At least one further data processing task includes identifying and discarding one or more portions of the generated synthetic medical text data that contain errors and / or otherwise penalizing the one or more portions of the generated synthetic medical text data. The medical information processing device according to claim 2 .
19. the processing circuitry further matches and / or determines one or more relationships between the predicted ground truth label information and portions of the generated synthetic medical text data. The medical information processing device according to claim 2 .
20. obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; Obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; performing a data processing task using the predicted ground truth label information and the further label information; A method comprising:
21. On the computer, obtaining initial medical data associated with ground truth medical text data, the ground truth medical text data including at least partially unstructured medical text data, and the initial medical data including structured medical text data generated from at least one of the ground truth medical text data and medical image data associated with the ground truth medical text data; Obtaining predicted ground truth label information by performing a classification process on the ground truth medical text data; obtaining further label information including annotated ground truth label information for the ground truth medical text data; performing a data processing task using the predicted ground truth label information and the further label information; A program that executes the following.