LLM-based image report modification type automatic evaluation method and system
The automatic evaluation method and system using LLM and NLP for image report modifications addresses the inefficiency of manual judgment, providing accurate and cost-effective quality assessment of junior doctors' reports.
Patent Information
- Application Number
- CN202411177890.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-17
AI Technical Summary
Existing methods for evaluating image report modifications require manual judgment, increasing labor costs and inefficiency.
An automatic evaluation method and system using Large Language Model (LLM) to predict and analyze image report modifications, comparing results with Natural Language Processing (NLP) for consistency, reducing the need for manual verification.
Reduces labor costs by using LLM and NLP to automatically evaluate image report modifications, ensuring accurate and efficient quality assessment of junior doctors' reports without manual checking.
Smart Images
Figure CN119153014A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical information, and more specifically, to an automatic evaluation method and system for image report modification types based on LLM. Background Art
[0002] In the process of writing imaging reports, the same report will be initially written by junior doctors and reviewed and modified by senior doctors. In order to evaluate the quality of junior doctors' report writing and to check for gaps in junior doctors' knowledge, it is necessary to first classify the modifications to the diagnostic impression in the medical imaging report. In the prior art, many literatures have been consulted on how to allocate the modification points of the preliminary report by the review report doctor, but there has been no clear method. There are also many classification methods for the types of imaging report modifications. The main technologies are as follows: Rule-based text classification method, which classifies text according to some manually formulated rules. Due to different scenarios, the rules are also different. Statistical text classification method, such as using machine learning and deep learning, to classify text through supervised learning with a large amount of labeled data. Text classification method based on large language model prompt words. It mainly refers to directly inputting the prompt words into the large language model to obtain the classification results without fine-tuning or training, etc. However, whether these classification results are correct still requires manual judgment by the reporting doctor, which increases the labor cost. Summary of the invention
[0003] In view of this, the main purpose of the present invention is to provide an automatic evaluation method and system for image report modification types based on LLM. The method can use LLM to perform model prediction according to pre-set questions, and compare the image report modification type predicted by LLM with the image report modification type analyzed by NLP for consistency, thereby solving the problem in the prior art that manual judgment is required as to whether the report modification type is correct, thereby increasing labor costs.
[0004] To achieve the above object, the technical solution of the present invention is achieved as follows:
[0005] On the one hand, the present invention provides an automatic evaluation method for image report modification types based on LLM, comprising: predicting a medical image report before modification and a medical image report after modification based on LLM, and outputting the modification type of the image report, which is defined as the LLM modification type; wherein the modification type of the image report is missed diagnosis, misdiagnosis, positioning error, unrecognized feature, feature recognition error, diagnostic logic error, text grammar and format error, inconsistent diagnostic confidence, inconsistent language habits, degree judgment error, and clinical communication problem; analyzing the medical image report before modification and the medical image report after modification through NLP, and outputting the modification type of the image report, which is defined as the NLP modification type; judging the consistency of the LLM modification type and the NLP modification type, and outputting the judgment result.
[0006] Preferably, the method of predicting the medical imaging report before and after modification based on LLM and outputting the modification type of the imaging report comprises the following steps: obtaining prompt words, wherein the prompt words are: imaging report before and after modification; normal and abnormal judgment before and after modification; type of disease before and after modification; imaging features before and after modification; definition of modification type of imaging report; obtaining examples, wherein the examples are a preset number of examples with the highest similarity match in a library of historical modification type examples; based on the prompt words and the examples, performing the LLM modification type prediction and outputting the LLM modification type.
[0007] Preferably, the method further comprises: performing multiple predictions of the LLM modification type based on the prompt word and the example, and taking the LLM modification type with consistent multiple results as the final output result.
[0008] Preferably, the medical imaging report before modification and the medical imaging report after modification are analyzed by NLP, and the modification type of the imaging report is output, and the steps include: preprocessing the medical imaging report before modification and the medical imaging report after modification, and presenting the preprocessed data in the form of sentence pairs; identifying the named entities and named entity types in the sentence pairs; pairing the named entities in each sentence pair, and outputting the paired named entities and named entity types; wherein the named entities that fail to match are inserted into the sequence in the original order; taking a comma or period separator as a unit, and judging the modification type of each unit according to preset logical rules based on whether the paired named entities are the same and the named entity types corresponding to the named entities.
[0009] Preferably, the method further comprises: the medical imaging report before modification and the medical imaging report after modification are preprocessed, and the steps include: identifying sentence boundaries: using a period or a colon as a sentence boundary, breaking the medical imaging report text into short sentences that express complete meanings; adjusting the word order: comparing each short sentence before and after modification, if when the short sentence is moved to a corresponding position, the increase in the longest common subsequence length before and after diagnosis compared to the number of sentences not moved accounts for a proportion of the moved short sentence that reaches a preset value, then it is determined that the short sentence needs to be moved to the corresponding position; sentence pairing: based on the longest common subsequence, using a packaged algorithm to obtain the corresponding positions and corresponding relationships of the short sentences before and after modification, and generating four moving pointers according to the corresponding positions and corresponding relationships, which respectively correspond to the starting position and end position of the short sentence before modification, and the starting position and end position of the short sentence after modification, and combining the short sentence boundary positions, continuously updating the four pointers, recording the matched short sentences in pairs, and forming the sentence pair.
[0010] Preferably, the method also includes: when there are multiple modification types for the imaging report, the modification type of the imaging report with the highest priority is used as the modification type of the imaging report; wherein the priority levels are arranged as follows: misdiagnosis>missed diagnosis>feature recognition error>feature not recognized>positioning error>diagnostic logic error>clinical communication problem>degree judgment error>text grammar and format errors>inconsistent diagnostic confidence>inconsistent language habits.
[0011] On the other hand, the present invention also provides an automatic evaluation system for image report modification types based on LLM, the system comprising: an LLM prediction module, an NLP analysis module and a result comparison module, wherein the LLM prediction module is connected to the result comparison module, and is used to predict the medical image report before modification and the medical image report after modification based on LLM, and output the modification type of the image report, which is defined as the LLM modification type; wherein the modification type of the image report is missed diagnosis, misdiagnosis, positioning error, feature non-recognition, feature recognition error, diagnostic logic error, text grammar and format error, inconsistent diagnostic confidence, inconsistent language habits, degree judgment error, and clinical communication problem; the NLP analysis module is connected to the result comparison module, and is used to analyze the medical image report before modification and the medical image report after modification through NLP, and output the modification type of the image report, which is defined as the NLP modification type; the result comparison module is respectively connected to the LLM prediction module and the NLP analysis module, and is used to judge the consistency of the LLM modification type and the NLP modification type, and output the judgment result.
[0012] Preferably, the LLM prediction module includes: an acquisition unit and a prediction unit, wherein the acquisition unit is connected to the prediction unit, and is used to acquire prompt words and examples; wherein the prompt words are: imaging reports before and after modification; normal and abnormal judgments before and after modification; types of diseases before and after modification; imaging features before and after modification; definition of modification types of imaging reports; the examples are a preset number of examples with the highest similarity matches in a library of historical modification type examples; the prediction unit is connected to the acquisition unit, and is used to predict the LLM modification type based on the prompt words and the examples, and output the LLM modification type.
[0013] Preferably, the NLP analysis module includes: a data preprocessing unit, a data identification unit, a named entity pairing unit and a modification type judgment unit, wherein the data preprocessing unit is connected to the data identification unit, and is used to preprocess the medical imaging report before modification and the medical imaging report after modification, and the preprocessed data presents the report data before modification and after modification in the form of sentence pairs; the data identification unit is respectively connected to the data preprocessing unit and the named entity pairing unit, and is used to identify the named entities and named entity types in the sentence pairs; the named entity pairing unit is respectively connected to the data identification unit and the modification type judgment unit, and is used to pair the named entities in each sentence pair, and output the paired named entities and named entity types; wherein the named entities that fail to match are inserted into the sequence in the original order; the modification type judgment unit is connected to the named entity pairing unit, and is used to take a comma or period separator as a unit, and based on whether the paired named entities are the same and the named entity types corresponding to the named entities, judge the modification type of each unit according to preset logical rules.
[0014] Preferably, when the modification types of the imaging report include multiple ones, the system further includes an LLM modification type determination module and an NLP modification type determination module, which are respectively connected to the LLM prediction module and the NLP analysis module, and are both used to use the modification type with the highest priority as the modification type of the imaging report; wherein the priority levels are ranked as follows: misdiagnosis> missed diagnosis> feature recognition error> feature not recognized> positioning error> diagnostic logic error> clinical communication problem> degree judgment error> text grammar and format error> inconsistent diagnostic confidence> inconsistent language habits.
[0015] Technical effects of the present invention:
[0016] The method of the present invention predicts a medical image report before modification and a medical image report after modification based on LLM, outputs a modification type of the image report, and defines it as an LLM modification type; analyzes the medical image report before modification and the medical image report after modification through NLP, outputs a modification type of the image report, and defines it as an NLP modification type; judges the consistency of the LLM modification type and the NLP modification type, and outputs a judgment result. If the LLM modification type and the NLP modification type are the same, they are used as the gold standard result for sampling inspection, and there is no need to manually check whether the modification type is correct one by one; if the LLM modification type and the NLP modification type are the same or different, the results of the LLM modification type and the NLP modification type are analyzed one by one, and the consistency of the two results is improved by improving the NLP rule or the LLM prompt content, thereby reducing labor costs, saving doctors' time, and each doctor can be responsible for more tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0018] Figure 1 A flow chart of an automatic evaluation method for image report modification type based on LLM according to Embodiment 1 of the present invention is shown;
[0019] Figure 2 A schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 2 of the present invention is shown;
[0020] Figure 3 A schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 3 of the present invention is shown;
[0021] Figure 4 A schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to a fourth embodiment of the present invention is shown;
[0022] Figure 5 A schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 5 of the present invention is shown. DETAILED DESCRIPTION
[0023] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.
[0024] Embodiment 1
[0025] Figure 1 FIG. 4 shows a flow chart of an automatic evaluation method for image report modification type based on LLM according to Embodiment 1 of the present invention; Figure 1 As shown, the method comprises the following steps:
[0026] Step S101, predicting the medical imaging report before and after modification based on LLM (large language model), and outputting the modification type of the imaging report, which is defined as the LLM modification type;
[0027] Among them, before LLM predicts the medical imaging report before and after the modification, it is also necessary to extract the corresponding text content of the split image reports before and after the modification according to the image diagnosis and image performance, so as to facilitate better prediction.
[0028] Among them, the types of modification of the imaging report are missed diagnosis, misdiagnosis, positioning error, unrecognized features, incorrect feature recognition, diagnostic logic errors, text grammar and format errors, inconsistent diagnostic confidence, inconsistent language habits, incorrect degree judgment, and clinical communication problems;
[0029] -Missed diagnosis: Generally, there is no description of the disease type before the modification, but there is a description of the disease type after the modification.
[0030] - Misdiagnosis: The disease types at the same location are different before and after modification, or there are disease types before modification but they are normal after modification.
[0031] - Positioning error: The disease type or characteristics are consistent before and after modification, but the anatomical position or spatial relationship is inconsistent.
[0032] -Feature not recognized: There is no image feature before the corresponding position is modified, but there is an image feature after the modification.
[0033] - Feature recognition error: There is an image feature before the corresponding position is modified but there is no image feature after the modification, or there are affected features before and after the modification but the features are inconsistent.
[0034] -Diagnostic logic error: the importance of disease types is not ranked correctly, and the order of sentences has been adjusted.
[0035] -Text grammar and format errors: Text and format errors, such as typos and punctuation.
[0036] -Inconsistent diagnostic confidence: The words "possible, highly likely, not excluded" or the expression of possibility before and after the modification are inconsistent.
[0037] - Inconsistent language habits: The writing habits before and after the modification are inconsistent, and synonyms are used.
[0038] - Error in judgment of severity: The description of the severity of the disease before and after the modification is inconsistent. The relevant severity descriptions include: mild, slight, slightly, slightly, severe, serious, significant, etc.
[0039] -Clinical communication issues: For example, clinical questions are not answered, the text is normal before modification but denied to be normal or the disease after modification; or there are hints for clinical work, such as the words "contrast", "comparison", "suggestion", etc.
[0040] The steps of predicting the medical imaging report before and after modification based on LLM and outputting the modification type of the imaging report include:
[0041] Get the prompt word, the prompt word is:
[0042] Imaging reports before and after modification;
[0043] Normal and abnormal judgment before and after modification;
[0044] Types of diseases before and after modification;
[0045] Image features before and after modification;
[0046] Definition of modification types for imaging reports;
[0047] Examples are obtained, where the examples are the preset number of examples with the highest similarity matches in the historical modification type example library; wherein the preset number may be 3, without any limitation here, and may be determined according to actual needs.
[0048] Based on the prompt word and the example, the LLM modification type is predicted and the LLM modification type is output.
[0049] The method further includes: performing multiple predictions of the LLM modification type based on the prompt word and the example, and taking the LLM modification type with consistent multiple results as the final output result.
[0050] Among them, the method also includes the judgment of normal and abnormal imaging reports based on LLM: using prompt words to perform multiple predictions on the imaging diagnostic reports before and after modification through the LLM model, and in the prediction process, the three examples with the highest similarity matching in the normal and abnormal examples library of historical reports can be added (the number of examples is determined according to actual needs), and the data with consistent results from multiple times are calculated as the prediction results of the imaging report for normal and abnormal.
[0051] Judgment of disease types in imaging reports based on LLM: Use prompt words to predict the abnormal imaging diagnosis text content of imaging reports before and after modification multiple times through the LLM model. During the prediction process, the three most similar examples of disease types in historical reports and the ten most similar disease types in the disease type knowledge base can be added. After the prediction is completed, the data with consistent results from multiple times will be used as the prediction result of the disease type of the imaging report through calculation according to the disease type.
[0052] Judgment of the disease type description in the imaging report based on LLM: Use prompt words to predict the abnormal imaging diagnosis text content of the imaging report before and after modification multiple times through the LLM model. During the prediction process, the three examples with the highest similarity match in the historical report disease type description example library can be added. Through calculation, the data with consistent results from multiple times are used as the prediction results of the disease type description of the imaging report.
[0053] Use LLM to judge the imaging features of the disease type in the imaging report: Use prompt words to predict the abnormal imaging diagnosis text content of the imaging report before and after modification multiple times through the LLM model. During the prediction process, the three examples with the highest similarity matching in the library of historical reported disease type and image feature examples can be added. Through calculation, the data with consistent results from multiple times are used as the image features of the disease type in the imaging report.
[0054] Step S102, analyzing the medical imaging report before modification and the medical imaging report after modification by NLP (Natural Language Processing), and outputting the modification type of the imaging report, which is defined as the NLP modification type;
[0055] The medical imaging report before modification and the medical imaging report after modification are analyzed by NLP, and the modification type of the imaging report is output, and the steps include: preprocessing the medical imaging report before modification and the medical imaging report after modification, and presenting the preprocessed data in the form of sentence pairs; identifying the named entities and named entity types in the sentence pairs; pairing the named entities in each sentence pair, and outputting the paired named entities and named entity types; wherein the named entities that fail to match are inserted into the sequence in the original order; taking a comma or period separator as a unit, and judging the modification type of each unit according to preset logical rules based on whether the paired named entities are the same and the named entity types corresponding to the named entities.
[0056] The method further comprises: the medical imaging report before modification and the medical imaging report after modification are preprocessed, and the steps include: identifying sentence boundaries: using a period or a colon as a sentence boundary, breaking the medical imaging report text into short sentences that express complete meanings; adjusting word order: comparing each short sentence before and after modification, if when the short sentence is moved to a corresponding position, the increase in the longest common subsequence length before and after diagnosis compared to the number of short sentences that are not moved accounts for a proportion of the moved short sentence that reaches a preset value, then it is determined that the short sentence needs to be moved to the corresponding position; sentence pairing: based on the longest common subsequence, using a packaged algorithm to obtain the corresponding positions and corresponding relationships of the short sentences before and after modification, and generating four moving pointers according to the corresponding positions and corresponding relationships, which respectively correspond to the starting position and end position of the short sentence before modification, and the starting position and end position of the short sentence after modification, and combining the short sentence boundary positions, continuously updating the four pointers, recording the matched short sentences in pairs, and forming the sentence pair.
[0057] Wherein, the method also includes: when the modification type of the imaging report includes multiple types, the modification type of the imaging report with the highest priority is used as the modification type of the imaging report; wherein the priority levels are arranged as follows: misdiagnosis> missed diagnosis> feature recognition error> feature not recognized> positioning error> diagnostic logic error> clinical communication problem> degree judgment error> text grammar and format error> inconsistent diagnostic confidence> inconsistent language habits.
[0058] The following takes the modification type determination of a lumbar vertebra report as an example to illustrate the steps implemented by the method of the present invention:
[0059] -Original Report
[0060]
[0061]
[0062] - Sentence boundary recognition
[0063] In this example, the period and colon are used as sentence boundaries for splitting. After splitting, each line can basically express a relatively complete meaning.
[0064]
[0065] - Word order adjustment
[0066] According to the results of the previous step, it was found that "L4-5 horizontal spinal canal stenosis" should have been cut to the back, and the lower position was simply modified to "L3-5". This is indeed the case through program calculation. If the sentence is moved to the corresponding position, the length of the longest common subsequence before and after the diagnosis increases by 8 compared to the original. The length of this sentence is 10, accounting for more than 40%, so it is determined that the sentence should be moved; as for moving other sentences, there is no gain in the length of the longest common subsequence, so it is not moved, and the final result is as follows.
[0067]
[0068]
[0069] - Sentence pairing
[0070]
[0071] -Named Entity Recognition
[0072] From the aligned report above, we take a typical sentence for the following analysis.
[0073] In the dictionary of named entity recognition, "L5", "S1", and "right side" are in the positioning dictionary, "intervertebral disc herniation" is in the disease type dictionary, and "nerve root compression" is in the image feature dictionary, so the corresponding named entities can be identified from the text.
[0074]
[0075]
[0076] - Named Entity Pairing
[0077] By calculating the defined algorithm, we can get the following named entity matching results:
[0078]
[0079] -Modify type judgment
[0080] According to the named entity matching results, before and after the modification, there are only two additional entities after the modification, namely positioning and image features. Therefore, the error types are positioning error and feature unrecognition. Since the priority of feature unrecognition is higher than positioning error, the final modification type is feature unrecognition.
[0081]
[0082] Step S103, judging the consistency of the LLM modification type and the NLP modification type, and outputting the judgment result;
[0083] Use a computer program to compare the judgment results of LLM and NLP modification types. If the LLM modification type and the NLP modification type are the same, then use them as the gold standard result for sampling inspection without manual confirmation one by one. If the LLM modification type and the NLP modification type are different, analyze the LLM modification type and NLP modification type results one by one, and improve the NLP rules or LLM prompt content to improve the consistency of the two results.
[0084] The embodiment of the present invention predicts the medical image report before modification and the medical image report after modification based on LLM, outputs the modification type of the image report, which is defined as the LLM modification type; analyzes the medical image report before modification and the medical image report after modification through NLP, outputs the modification type of the image report, which is defined as the NLP modification type; judges the consistency of the LLM modification type and the NLP modification type, and outputs the judgment result. If the LLM modification type and the NLP modification type are the same, they are used as the gold standard result for sampling inspection without manual inspection of whether the modification type is correct one by one; if the LLM modification type and the NLP modification type are the same or different, the results of the LLM modification type and the NLP modification type are analyzed one by one, and the consistency of the two results is improved by improving the NLP rules or the LLM prompt content, thereby reducing labor costs, saving doctors' time, and each doctor can be responsible for more tasks.
[0085] Embodiment 2
[0086] Figure 2 FIG. 4 shows a schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 2 of the present invention; Figure 2 As shown, the system includes: an LLM prediction module 10, an NLP analysis module 20 and a result comparison module 30, wherein:
[0087] The LLM prediction module 10 is connected to the result comparison module 30, and is used to predict the medical image report before modification and the medical image report after modification based on LLM, and output the modification type of the image report, which is defined as the LLM modification type;
[0088] Among them, before LLM predicts the medical imaging report before and after the modification, it is also necessary to extract the corresponding text content of the split image reports before and after the modification according to the image diagnosis and image performance, so as to facilitate better prediction.
[0089] Among them, the types of modification of the imaging report are missed diagnosis, misdiagnosis, positioning error, unrecognized features, incorrect feature recognition, diagnostic logic errors, text grammar and format errors, inconsistent diagnostic confidence, inconsistent language habits, incorrect degree judgment, and clinical communication problems;
[0090] -Missed diagnosis: Generally, there is no description of the disease type before the modification, but there is a description of the disease type after the modification.
[0091] - Misdiagnosis: The disease types at the same location are different before and after modification, or there are disease types before modification but they are normal after modification.
[0092] - Positioning error: The disease type or characteristics are consistent before and after modification, but the anatomical position or spatial relationship is inconsistent.
[0093] -Feature not recognized: There is no image feature before the corresponding position is modified, but there is an image feature after the modification.
[0094] - Feature recognition error: There is an image feature before the corresponding position is modified but there is no image feature after the modification, or there are affected features before and after the modification but the features are inconsistent.
[0095] -Diagnostic logic error: the importance of disease types is not ranked correctly, and the order of sentences has been adjusted.
[0096] -Text grammar and format errors: Text and format errors, such as typos and punctuation.
[0097] -Inconsistent diagnostic confidence: The words "possible, highly likely, not excluded" or the expression of possibility before and after the modification are inconsistent.
[0098] - Inconsistent language habits: The writing habits before and after the modification are inconsistent, and synonyms are used.
[0099] - Error in judgment of severity: The description of the severity of the disease before and after the modification is inconsistent. The relevant severity descriptions include: mild, slight, slightly, slightly, severe, serious, significant, etc.
[0100] -Clinical communication issues: For example, clinical questions are not answered, the text is normal before modification but denied to be normal or the disease after modification; or there are hints for clinical work, such as the words "contrast", "comparison", "suggestion", etc.
[0101] The NLP analysis module 20 is connected to the result comparison module 30, and is used to analyze the medical image report before modification and the medical image report after modification through NLP, and output the modification type of the image report, which is defined as the NLP modification type;
[0102] The result comparison module 30 is connected to the LLM prediction module 10 and the NLP analysis module 20 respectively, and is used to judge the consistency of the LLM modification type and the NLP modification type, and output the judgment result.
[0103] Use a computer program to compare the judgment results of LLM and NLP modification types. If the LLM modification type and the NLP modification type are the same, then use them as the gold standard result for sampling inspection without manual confirmation one by one. If the LLM modification type and the NLP modification type are different, analyze the LLM modification type and NLP modification type results one by one, and improve the NLP rules or LLM prompt content to improve the consistency of the two results.
[0104] Embodiment 3
[0105] Figure 3 FIG. 4 shows a schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 3 of the present invention; Figure 3 As shown, the LLM prediction module 10 includes: an acquisition unit 102 and a prediction unit 104, wherein:
[0106] The acquisition unit 102 is connected to the prediction unit 104 and is used to acquire prompt words and examples;
[0107] Among them, the prompt words are: imaging reports before and after modification; normal and abnormal judgments before and after modification; disease types before and after modification; imaging features before and after modification; definition of modification types of imaging reports; the examples are the preset number of examples with the highest similarity matches in the historical modification type example library; among them, the preset number can be 3, without any limitation here, and can be determined according to actual needs.
[0108] The prediction unit 104 is connected to the acquisition unit 102, and is used to predict the LLM modification type based on the prompt word and the example, and output the LLM modification type.
[0109] The system further includes: performing multiple predictions of the LLM modification type based on the prompt words and the examples, and taking the LLM modification type with the consistent multiple results as the final output result.
[0110] Embodiment 4
[0111] Figure 4 FIG. 4 shows a schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 4 of the present invention; Figure 4 As shown, the NLP analysis module 20 includes: a data preprocessing unit 202, a data identification unit 204, a named entity pairing unit 206 and a modification type judgment unit 208, wherein:
[0112] The data preprocessing unit 202 is connected to the data identification unit 204 and is used to preprocess the medical imaging report before modification and the medical imaging report after modification, and the preprocessed data presents the report data before modification and the report data after modification in the form of sentence pairs;
[0113] The data identification unit 204 is connected to the data preprocessing unit 202 and the named entity pairing unit 206, respectively, and is used to identify the named entities and named entity types in the sentence pairs;
[0114] The named entity pairing unit 206 is connected to the data identification unit 204 and the modification type determination unit 208, respectively, and is used to pair the named entities in each sentence pair and output the paired named entities and named entity types; wherein the unmatched named entities are inserted into the sequence according to the original order;
[0115] The modification type judgment unit 208 is connected to the named entity pairing unit 206, and is used to use a comma or period separator as a unit, and judge the modification type of each unit according to preset logical rules based on whether the paired named entities are the same and the named entity types corresponding to the named entities.
[0116] Embodiment 5
[0117] Figure 5 FIG. 4 shows a schematic diagram of the structure of an automatic evaluation system for image report modification types based on LLM according to Embodiment 5 of the present invention. Figure 5 As shown, when the modification type of the imaging report includes multiple ones, the system also includes an LLM modification type determination module 40 and an NLP modification type determination module 50, which are respectively connected to the LLM prediction module 10 and the NLP analysis module 20, and are both used to use the modification type with the highest priority as the modification type of the imaging report; wherein the priority levels are arranged as follows: misdiagnosis> missed diagnosis> feature recognition error> feature not recognized> positioning error> diagnostic logic error> clinical communication problem> degree judgment error> text grammar and format error> inconsistent diagnostic confidence> inconsistent language habits.
[0118] The following takes the modification type judgment of a lumbar spine report as an example to illustrate the implementation of this system:
[0119] -Original Report
[0120]
[0121]
[0122] - Sentence boundary recognition
[0123] In this example, the period and colon are used as sentence boundaries for splitting. After splitting, each line can basically express a relatively complete meaning.
[0124]
[0125] - Word order adjustment
[0126] According to the results of the previous step, it was found that "L4-5 horizontal spinal canal stenosis" should have been cut to the back, and the lower position was simply modified to "L3-5". This is indeed the case through program calculation. If the sentence is moved to the corresponding position, the length of the longest common subsequence before and after the diagnosis increases by 8 compared to the original. The length of this sentence is 10, accounting for more than 40%, so it is determined that the sentence should be moved; as for moving other sentences, there is no gain in the length of the longest common subsequence, so it is not moved, and the final result is as follows.
[0127]
[0128]
[0129] - Sentence pairing
[0130]
[0131] -Named Entity Recognition
[0132] From the aligned report above, we take a typical sentence for the following analysis.
[0133] In the dictionary of named entity recognition, "L5", "S1", and "right side" are in the positioning dictionary, "intervertebral disc herniation" is in the disease type dictionary, and "nerve root compression" is in the image feature dictionary, so the corresponding named entities can be identified from the text.
[0134]
[0135]
[0136] - Named Entity Pairing
[0137] By calculating the defined algorithm, we can get the following named entity matching results:
[0138]
[0139] -Modify type judgment
[0140] According to the named entity matching results, before and after the modification, there are only two additional entities after the modification, namely positioning and image features. Therefore, the error types are positioning error and feature unrecognition. Since the priority of feature unrecognition is higher than positioning error, the final modification type is feature unrecognition.
[0141]
[0142] The embodiment of the present invention predicts the medical image report before modification and the medical image report after modification based on LLM, outputs the modification type of the image report, which is defined as the LLM modification type; analyzes the medical image report before modification and the medical image report after modification through NLP, outputs the modification type of the image report, which is defined as the NLP modification type; judges the consistency of the LLM modification type and the NLP modification type, and outputs the judgment result. If the LLM modification type and the NLP modification type are the same, they are used as the gold standard result for sampling inspection without manual inspection of whether the modification type is correct one by one; if the LLM modification type and the NLP modification type are the same or different, the results of the LLM modification type and the NLP modification type are analyzed one by one, and the consistency of the two results is improved by improving the NLP rules or the LLM prompt content, thereby reducing labor costs, saving doctors' time, and each doctor can be responsible for more tasks.
[0143] From the above description, it can be seen that the above-mentioned embodiments of the present invention achieve the following technical effects: the embodiments of the present invention predict the medical imaging report before modification and the medical imaging report after modification based on LLM, and output the modification type of the imaging report, which is defined as the LLM modification type; analyze the medical imaging report before modification and the medical imaging report after modification through NLP, and output the modification type of the imaging report, which is defined as the NLP modification type; judge the consistency of the LLM modification type and the NLP modification type, and output the judgment result. If the LLM modification type and the NLP modification type are the same, they are used as the gold standard result for sampling inspection, and there is no need to manually check whether the modification type is correct one by one; if the LLM modification type and the NLP modification type are the same or different, the results of the LLM modification type and the NLP modification type are analyzed one by one, and the NLP rules or LLM prompt content are improved to improve the consistency of the two results, thereby reducing labor costs, saving doctors' time, and each doctor can be responsible for more tasks.
[0144] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An automatic evaluation method for image report modification type based on LLM, characterized in that: include: Based on LLM, the medical imaging report before and after modification is predicted, and the modification type of the imaging report is output, which is defined as the LLM modification type; wherein the modification type of the imaging report is missed diagnosis, misdiagnosis, positioning error, feature failure, feature recognition error, diagnostic logic error, text grammar and format error, inconsistent diagnostic confidence, inconsistent language habits, degree judgment error, and clinical communication problem; Analyzing the medical imaging report before modification and the medical imaging report after modification by NLP, and outputting a modification type of the imaging report, which is defined as an NLP modification type; The consistency of the LLM modification type and the NLP modification type is judged, and the judgment result is output.
2. The method for automatically evaluating the modification type of image report based on LLM according to claim 1, characterized in that: The steps of predicting the medical imaging report before and after modification based on LLM and outputting the modification type of the imaging report include: Obtain prompt words, the prompt words are: imaging report before and after modification; normal and abnormal judgment before and after modification; disease type before and after modification; imaging features before and after modification; definition of modification type of imaging report; Obtaining examples, wherein the examples are a preset number of examples with the highest similarity matches in a historical modification type example library; Based on the prompt word and the example, the LLM modification type is predicted and the LLM modification type is output.
3. The method for automatically evaluating the modification type of image report based on LLM according to claim 2, characterized in that: The method also includes: performing multiple predictions of the LLM modification type based on the prompt word and the example, and taking the LLM modification type with consistent multiple results as the final output result.
4. The method for automatically evaluating the modification type of image report based on LLM according to claim 1, characterized in that: The steps of analyzing the medical image report before modification and the medical image report after modification by NLP and outputting the modification type of the image report include: Preprocessing the medical imaging report before and after the modification, and presenting the report data before and after the modification in the form of sentence pairs; identifying named entities and named entity types in the sentence pairs; Pair the named entities in each sentence pair and output the paired named entities and named entity types; among them, the unmatched named entities are inserted into the sequence in the original order; A comma or period separator is used as a unit. Based on whether the paired named entities are the same and the named entity types corresponding to the named entities, the modification type of each unit is determined according to preset logical rules.
5. The method for automatically evaluating the modification type of image report based on LLM according to claim 4, characterized in that: The steps of preprocessing the medical imaging report before and after modification include: Identify sentence boundaries: Use periods or colons as sentence boundaries to break the medical imaging report text into short sentences that express complete meanings. Adjust word order: Compare each short sentence before and after the modification. If, when the short sentence is moved to the corresponding position, the increase in the longest common subsequence length before and after the diagnosis reaches a preset value compared with the short sentence before the move, it is determined that the short sentence needs to be moved to the corresponding position; Sentence pairing: Based on the longest common subsequence, the corresponding positions and corresponding relationships of the short sentences before and after the modification are obtained by using the packaged algorithm. According to the corresponding positions and corresponding relationships, four moving pointers are generated, corresponding to the starting position and the ending position of the short sentence before the modification, and the starting position and the ending position of the short sentence after the modification, respectively. Combined with the boundary positions of the short sentences, the four pointers are continuously updated to record the matched paired short sentences to form the sentence pairs.
6. The method for automatically evaluating the modification type of image report based on LLM according to claim 1, characterized in that: The method also includes: when the modification type of the image report includes multiple types, the modification type of the image report with the highest priority is used as the modification type of the image report; wherein the priority levels are arranged as follows: misdiagnosis> missed diagnosis> feature recognition error> feature not recognized> positioning error> diagnostic logic error> clinical communication problem> degree judgment error> text grammar and format errors> inconsistent diagnostic confidence> inconsistent language habits.
7. An automatic evaluation system for image report modification types based on LLM, characterized in that: The system includes: LLM prediction module, NLP analysis module and result comparison module, among which, The LLM prediction module is connected to the result comparison module, and is used to predict the medical imaging report before and after modification based on LLM, and output the modification type of the imaging report, which is defined as the LLM modification type; wherein the modification type of the imaging report is missed diagnosis, misdiagnosis, positioning error, feature failure, feature recognition error, diagnostic logic error, text grammar and format error, inconsistent diagnostic confidence, inconsistent language habits, degree judgment error, and clinical communication problem; The NLP analysis module is connected to the result comparison module, and is used to analyze the medical image report before modification and the medical image report after modification through NLP, and output the modification type of the image report, which is defined as the NLP modification type; The result comparison module is connected to the LLM prediction module and the NLP analysis module respectively, and is used to judge the consistency of the LLM modification type and the NLP modification type, and output the judgment result.
8. The automatic evaluation system for image report modification type based on LLM according to claim 7 is characterized in that: The LLM prediction module includes: an acquisition unit and a prediction unit, wherein: The acquisition unit is connected to the prediction unit and is used to acquire prompt words and examples; wherein the prompt words are: image reports before and after modification; normal and abnormal judgments before and after modification; disease types before and after modification; image features before and after modification; definition of modification types of image reports; the examples are the preset number of examples with the highest similarity matching in the historical modification type example library; The prediction unit is connected to the acquisition unit, and is used to predict the LLM modification type based on the prompt word and the example, and output the LLM modification type.
9. The automatic evaluation system for image report modification type based on LLM according to claim 7, characterized in that: The NLP analysis module includes: a data preprocessing unit, a data identification unit, a named entity pairing unit and a modification type judgment unit, wherein: The data preprocessing unit is connected to the data identification unit and is used to preprocess the medical imaging report before modification and the medical imaging report after modification, and the preprocessed data presents the report data before modification and the report data after modification in the form of sentence pairs; The data identification unit is connected to the data preprocessing unit and the named entity pairing unit respectively, and is used to identify the named entities and named entity types in the sentence pairs; The named entity pairing unit is connected to the data identification unit and the modification type judgment unit respectively, and is used to pair the named entities in each sentence pair and output the paired named entities and named entity types; wherein the unmatched named entities are inserted into the sequence according to the original order; The modification type judgment unit is connected to the named entity pairing unit, and is used to use a comma or period separator as a unit, and judge the modification type of each unit according to preset logical rules based on whether the paired named entities are the same and the named entity types corresponding to the named entities.
10. The automatic evaluation system for image report modification type based on LLM according to claim 7, characterized in that: When the modification types of the imaging report include multiple ones, the system also includes an LLM modification type determination module and an NLP modification type determination module, which are respectively connected to the LLM prediction module and the NLP analysis module, and are both used to use the modification type with the highest priority as the modification type of the imaging report; wherein the priority levels are arranged as follows: misdiagnosis> missed diagnosis> feature recognition error> feature not recognized> positioning error> diagnostic logic error> clinical communication problem> degree judgment error> text grammar and format error> inconsistent diagnostic confidence> inconsistent language habits.