Kidney CT image report structured conversion method and system based on large language model, electronic equipment and storage medium
By combining a large language model with multiple prompt words, the problem of structured conversion of kidney CT image reports was solved, achieving efficient and accurate information extraction and reliability for clinical application, and providing a multi-dimensional performance evaluation method.
Patent Information
- Application Number
- CN202511087724.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-05
- Publication Date
- 2025-11-11
AI Technical Summary
Existing technologies struggle to efficiently and accurately convert free-text reports of kidney CT images into structured reports, resulting in low information extraction efficiency and poor reliability in clinical applications.
By employing a large language model combined with word cloud generation and keyword visualization techniques, we designed various prompt words and uncertainty quantification methods. Through multiple large language models, we performed structured transformation on the free text report of kidney CT images, and selected the model with the lowest entropy value as the final result.
It improves the efficiency of information extraction from renal CT image reports and the quality of clinical decision-making, ensures the reliability and accuracy of the model, and provides a multi-dimensional performance evaluation framework.
Smart Images
Figure CN120932801A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a method, system, electronic device, and storage medium for structured conversion of kidney CT image reports based on a large language model. Background Technology
[0002] In clinical practice, radiology reports are a crucial means for clinicians to obtain information about patients' conditions. However, traditional radiology reports are mostly presented in free text format, which is time-consuming to write and suffers from poor readability and clinical applicability. Furthermore, this non-standardized and unstructured nature poses significant challenges to information extraction, data integration, and subsequent analysis. Structured reports, with their clear logic, orderly organization, and standardized format, present information in a way that significantly improves the decision-making efficiency of medical professionals compared to traditional report formats.
[0003] Early research on extracting key information from free text reports primarily fell into two categories: rule-based methods and machine learning-based methods. Rule-based methods generally rely on medical expertise, manually constructing a series of precise rules and patterns, such as specific combinations of medical terminology, symptom description formats, and disease diagnostic criteria. These pre-defined rules are used to scan and match content in free text reports; when a text fragment conforms to the rules, it is identified as key information to be extracted. However, this approach requires extensive medical expertise and is time-consuming and labor-intensive, making it difficult to cover all possible variations in expression. Machine learning-based methods utilize algorithms to allow computers to autonomously learn the features and patterns of key information from large amounts of labeled medical text data. First, key information in diagnostic reports (such as disease names, symptoms, and examination results) needs to be manually labeled to form a training dataset. Then, a suitable machine learning model (such as support vector machines, decision trees, and neural networks) is selected and trained to master the representation and patterns of this key information in the text. The trained model is then used to extract key information from new, unlabeled free text reports. However, the main difficulty with this approach lies in the difficulty of data acquisition and the extremely high cost of labeling, leading to a scarcity of relevant medical data and thus making model training challenging.
[0004] The technical means to convert free text reports into structured reports still need to be improved, especially how to achieve this conversion process efficiently and accurately, which has become an urgent problem to be solved in the medical field. Summary of the Invention
[0005] This invention proposes a structured conversion method for kidney CT image reports based on a large language model to solve the above-mentioned problems.
[0006] The proposed method for structured conversion of renal CT image reports based on a large language model includes the following specific steps:
[0007] Step 1: Collect free text reports of kidney CT images and construct a dataset.
[0008] Step 2: Word cloud generation and keyword visualization techniques were used to perform text analysis on all free-text reports of renal CT images. Combined with clinical experience, 15 key features were identified. Based on these key imaging features, a structured report template for renal CT images was designed.
[0009] Step 3: Based on the structured report template, design three different prompt words for the large language model, and determine the final prompt words to be used by quantitatively evaluating the performance of the three prompt words on the overall accuracy index.
[0010] Step 4: Using the free text report of kidney CT images as input data, and with the designed prompts, call the API interfaces of five major language models, namely DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o and Gemini, respectively to perform inference, obtain the output results of each model, and each model independently generates a structured report containing 15 predefined key features.
[0011] Step 5: Design an uncertainty quantification method based on Temperature sampling. The Temperature parameter is a key parameter in the large language model sampling process, used to control the randomness of the output results (Temperature=1 represents the original probability distribution, Temperature>1 enhances randomness, and Temperature<1 enhances determinism). While maintaining a consistent Temperature parameter, DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o, and Gemini models are used to independently repeat the free-text report of the same original kidney CT image five times, calculating the entropy value to perform a deterministic evaluation of the structured report output by each model.
[0012] Step 6: Compare the entropy values of the output results of the five large language models, and determine the structured report corresponding to the model with the lowest entropy value as the final high-confidence structured report.
[0013] Compared with the prior art, the beneficial effects of the method of the present invention are as follows:
[0014] (1) This invention utilizes a large language model to perform structured transformation of free text reports of kidney CT images. Compared with traditional natural language processing algorithms, the large language model can extract key information more efficiently and accurately, thereby improving information extraction efficiency and the quality of clinical decision-making.
[0015] (2) The uncertainty quantification method based on temperature sampling proposed in this invention provides an effective means to evaluate the certainty of results from different large language models, and provides a basis and calculation method for selecting the most suitable model from many large language models on the market. By evaluating the predictive certainty of the model, a quantitative basis is provided for the reliability of the model's clinical application, which helps to ensure the safety and accuracy of clinical application.
[0016] (3) The multi-dimensional evaluation framework constructed in this invention evaluates the model from multiple dimensions such as key feature extraction accuracy, text similarity and prediction certainty, comprehensively and objectively evaluates the performance of the model, and provides a scientific basis for selecting a suitable model for clinical application. Attached Figure Description
[0017] Figure 1 This is a framework diagram of a structured conversion method for kidney CT image reports based on a large language model, according to the present invention.
[0018] Figure 2 This is a statistical chart of high-frequency words in the image report before and after the large language model structuring process in this invention;
[0019] Figure 3 These are word cloud images of the image reports before and after structuring by a large language model in this invention;
[0020] Figure 4 This is a bar chart showing the accurate quantity statistics of the structured reports generated by this invention;
[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation
[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The description based on the drawings is exemplary and is only used to explain the present invention in detail, and should not be construed as limiting the present invention.
[0023] This invention proposes a structured conversion method for renal CT image reports based on a large language model, such as... Figure 1As shown. The invention mainly comprises four parts: structured template design, prompt word engineering, structured report generation, and model uncertainty quantification. In the structured template design stage, word cloud generation and keyword visualization technologies are used to analyze the text of all free-text reports of renal CT images. Simultaneously, combined with clinical experience, 15 key features are identified to design a structured report template for renal CT images. In the prompt word engineering stage, three different prompt words for a large language model are designed. The quality of the prompt words is quantitatively evaluated on the GPT4 model, and the prompt word with the best structured report generation effect is selected as the prompt word used in the structured report generation stage. In the structured report generation stage, using the free-text report of renal CT images as input data, and in conjunction with the designed prompt words, the API interfaces of five large language models—DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o, and Gemini—are called for inference. The output results of each model are obtained, and the effectiveness of different large models is evaluated. In the model uncertainty quantification stage, while keeping the temperature parameter consistent, the five major models independently and repeatedly reasoned five times on the same original kidney CT image free text report, and calculated the information entropy value H, thereby performing uncertainty quantification analysis on the model. The structured report corresponding to the model with the lowest entropy value was determined as the final high-confidence structured report.
[0024] The proposed method for structured conversion of renal CT image reports based on a large language model mainly includes four parts: structured template design, prompt word engineering, structured report generation, and model uncertainty quantification. Specifically, it includes the following steps:
[0025] Step 1: Collect free text reports of kidney CT images and construct a dataset.
[0026] Step 2: In this invention, word cloud generation and keyword visualization technologies are used to perform text analysis on all free-text reports of renal CT images. Combined with clinical experience, 15 key medical features were ultimately identified, including: location, size, boundary, growth pattern, calyx invasion, renal sinus fat invasion, perirenal fat invasion, breach of renal fascia, ipsilateral adrenal gland invasion, intratumoral fat, intratumoral hemorrhage, intratumoral calcification, intratumoral necrosis, enhancement on contrast-enhanced scans, and lesion blood supply source. Based on these key imaging features, a structured report template for renal CT images was designed.
[0027] Step 3: In this invention, prompt words are designed based on a structured report template. Three different prompt words are designed for the large language model: zero_shot_prompt, few_shot_prompt, and COT-prompt. In zero_shot_prompt, the large model is required to extract 15 key features and output a structured report. few_shot_prompt adds examples of structured reports to zero_shot_prompt, making the output format and content requirements clearer and improving the consistency of the generated results. COT-prompt introduces a Chain of Thought (COT) reasoning mechanism, which guides the large model to think step by step by prompting it to output a structured report. Finally, the performance of the three prompt words on the overall accuracy index is quantitatively evaluated to determine the final prompt words to be used.
[0028] Step 4: Using the prompts determined in Step 2, and the original free-text report of the kidney CT image as input data, along with the designed prompts, call the API interfaces of five major language models—DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o, and Gemini—for inference, and obtain the output results of each model. Each model independently generates a structured report containing 15 predefined key features. Finally, evaluate the performance of the five major models on key feature extraction accuracy, ROUGE, and BERTScore metrics.
[0029] Step 5: Design an uncertainty quantification method based on Temperature sampling to evaluate the determinism of the structured reports generated by the large model. First, input the original kidney CT image report and the prompt words into the same model, and repeat the reasoning independently five times under the constraint of the temperature parameter Temperature to obtain five structured reports. According to the preset three-level quality assessment categories C = {c1: "Completely correct", c2: "Partially incorrect", c3: "Completely incorrect"}, classify each structured report, count the frequency of each category in the five results, and then obtain the category probability distribution P(C). The calculation formula of P is shown in formula (1).
[0030] (1)
[0031] Based on the probability distribution, the information entropy H is calculated according to formula (2). According to the magnitude of the information entropy value H, the confidence of the model prediction is divided into two levels: when 0 ≤ H < 0.3, it is a high confidence level, indicating that the model has a high degree of certainty about the current prediction result; when H ≥ 0.3, it is a low confidence level, indicating that the uncertainty of the model prediction result is high.
[0032] (2)
[0033] Step 6: Compare the entropy values of the output results of the five large language models, and determine the structured report corresponding to the model with the lowest entropy value as the final high-confidence structured report.
[0034] Experimental setup:
[0035] A) Dataset: This invention collected 646 free-text image reports of kidney patients from the Cancer Prevention and Control Center between January 1, 2019 and November 1, 2022. All patients were newly diagnosed with kidney cancer and had not undergone surgery or other treatments. The reports were all in free-text format, and each report described the location, size, and growth pattern of the kidney tumor. After excluding two blank reports and reports with unclear descriptions, 644 reports were finally used for the generation experiment of structured reports.
[0036] B) Evaluation metrics: This invention uses ROUGE, BERTScore, and accuracy to evaluate the quality of the structured reports generated by the model, and also evaluates the overall accuracy of the structured reports.
[0037] Experimental results:
[0038] This invention compares the differences in the quality of structured reports converted based on different prompting strategies (zero-shot, few-shot, and COT). The performance of the three methods on the overall accuracy index is quantitatively evaluated, as shown in Table 1, where the best experimental results are marked in bold.
[0039] Table 1. Accuracy of Structured Reports Generated by Different Prompts
[0040] Prompt Accuracy (%) zero_shot_prompt 73 few_shot_prompt 82 COT-prompt 85
[0041] This invention evaluated the extraction accuracy of five large models for 15 key image features. The specific results are shown in Table 2, where the best experimental results are marked in bold.
[0042] Table 2. Accuracy of Key Feature Extraction by Different Models
[0043] feature Deepseek-R1 GPT-4 GPT-4o QWen-max Gemini Location 1 1 1 1 1 size 1 1 1 1 1 boundary 0.9907 0.9752 0.8959 0.9970 0.9099 Growth mode 0.9193 0.9938 0.9782 0.9845 0.8944 Renal calyx invasion 0.9736 0.9115 0.9907 0.9503 0.9565 Renal sinus fat invasion 0.9922 0.9798 0.9953 0.9938 0.9891 Perirenal fat invasion 0.9845 0.9798 0.9953 0.9798 0.9953 Breakthrough of renal fascia 0.9907 0.9860 0.9953 1 0.9922 ipsilateral adrenal gland invasion 0.9907 0.9813 0.9906 0.9860 0.9984 Intratumoral fat 0.9984 0.8509 0.8509 0.9814 0.9798 Intratumoral hemorrhage 0.9984 0.8431 0.9705 0.8509 0.9876 Intratumoral calcification 0.9969 0.8385 97.049 0.8431 1 Intratumoral necrosis 0.9860 0.8447 0.9814 0.9844 0.9705 Enhanced scan 0.9968 0.8261 0.9752 0.9472 0.9006 Blood supply to the lesion 0.9456 0.7826 0.9752 0.9938 0.8369 Average accuracy 0.9843 0.9196 0.9709 0.9661 0.9607
[0044] Meanwhile, this invention also evaluated the structured reports generated by the large model. It comprehensively evaluated the three text generation metrics ROUGE-1, ROUGE-2 and ROUGE-L, and combined them with the precision (P), recall (R) and F1 score (F1) of BERTScore. The specific results are shown in Table 3, where the best experimental results are marked in bold.
[0045] Table 3. Structured reports ROUGE and BERTScore generated by different models
[0046] Model ROUGE-1 ROUGE-2 ROUGE-L BERTScore_P BERTScore_R BERTScore_F1 Deepseek-R1 0.5956 0.4301 0.5932 0.7026 0.6460 0.6727 GPT-4 0.5993 0.4395 0.5958 0.7008 0.6541 0.6761 GPT-4o 0.6192 0.4559 0.6157 0.7038 0.6532 0.6771 Qwen-max 0.3356 0.1737 0.3343 0.6969 0.6433 0.6686 Gemini 0.3583 0.1897 0.3571 0.6922 0.6392 0.6642
[0047] This invention performed high-frequency word statistics and analysis on medical reports before and after structured processing, extracting the top 20 most frequent words from each version of the report (e.g., Figure 2 (as shown) and word cloud (as shown) Figure 3 (As shown).
[0048] This invention quantitatively assesses the accuracy of structured reports for five major model transformations, such as... Figure 4 As shown.
[0049] This invention performs uncertainty quantification assessment based on temperature sampling on five large models, selects the large model with the lowest entropy value, and determines the structured report generated by it as the final high-confidence structured report.
[0050] As can be seen from Table 1:
[0051] The structured reports generated using the COT-prompt prompting strategy significantly outperformed the control group, demonstrating the guiding role of contextual examples in the model's semantic understanding. Based on this experimental result, subsequent experiments uniformly adopted the COT-prompt prompting strategy to construct the structured report generation framework, ensuring methodological consistency in evaluating the performance of different models.
[0052] As can be seen from Table 2:
[0053] All the large language models achieved 100% accuracy in recognizing the features of "position" and "size". In comparison, the five large models also achieved an average key feature extraction accuracy of over 90%.
[0054] As can be seen from Table 3:
[0055] The GPT-4o model exhibits significant performance advantages: it significantly outperforms all comparison models in ROUGE series metrics (including single-word granularity ROUGE-1, double-word granularity ROUGE-2, and longest common subsequence ROUGE-L) as well as in BERTScore precision (P) and overall F1 score.
[0056] from Figure 2 and Figure 3 It can be seen that:
[0057] The original report contained many non-specific descriptions (such as "not seen" and "size") in its high-frequency words, while the structured version showed a more concentrated cluster of lesion feature words. This result indicates that large-scale model structuring can effectively extract the core information from medical texts, significantly improving the information density and clinical relevance of the report.
[0058] from Figure 4 It can be seen that:
[0059] The Deepseek-R1 model performed best in terms of the number of reports generated that fully met the standards (n=505), while the GPT-4 model generated significantly fewer accurate reports than other comparative models (n=263).
[0060] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the device includes a processor 60, a memory 61, an input device 62, and an output device 63; the number of processors 60 in the device can be one or more. Figure 5 Taking a processor 60 as an example; the processor 60, memory 61, input device 62, and output device 63 in the device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.
[0061] The memory 61, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the kidney CT image report structure conversion method in this embodiment of the invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 61, thereby realizing the aforementioned kidney CT image report structure conversion method.
[0062] The memory 61 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include memory remotely located relative to the processor 60, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0063] Input device 62 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 63 may include display devices such as a display screen.
[0064] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the kidney CT image report structure conversion method of any embodiment.
[0065] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for structured conversion of renal CT image reports based on a large language model, characterized in that, Includes the following steps: Step 1: Collect free text reports of kidney CT images and construct a dataset. Step 2: Word cloud generation and keyword visualization techniques were used to perform text analysis on all free-text reports of renal CT images. Combined with clinical experience, 15 key features were identified. Based on these key imaging features, a structured report template for renal CT images was designed. Step 3: Based on the structured report template, design three different prompt words for the large language model, and determine the final prompt words to be used by quantitatively evaluating the performance of the three prompt words on the overall accuracy index. Step 4: Using the free text report of kidney CT images as input data, and with the designed prompts, call the API interfaces of five major language models, namely DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o and Gemini, respectively to perform inference, obtain the output results of each model, and each model independently generates a structured report containing 15 predefined key features. Step 5: Design an uncertainty quantification method based on temperature sampling. While keeping the temperature parameter consistent, use DeepSeek-R1, Qwen-Max, GPT-4, GPT-4o and Gemini models to independently repeat the free text report of the same kidney CT image five times, calculate the entropy value, and evaluate the stability of the structured report output by each model. Step 6: Compare the entropy values of the output results of the five large language models, and determine the structured image report corresponding to the model with the lowest entropy value as the final high-confidence structured image report.
2. The method according to claim 1, characterized in that: Key features in step 2 include: location, size, boundaries, growth pattern, renal calyx invasion, renal sinus fat invasion, perirenal fat invasion, breach of renal fascia, ipsilateral adrenal gland invasion, intratumoral fat, intratumoral hemorrhage, intratumoral calcification, intratumoral necrosis, enhancement on contrast-enhanced CT scan, and lesion blood supply source.
3. The method according to claim 1, characterized in that: The three prompts designed in step 3 are: zero_shot_prompt, few_shot_prompt, and COT-prompt. Zero_shot_prompt requires the large model to extract 15 key features and output a structured report. Few_shot_prompt, building upon zero_shot_prompt, adds examples of structured reports, making the output format and content requirements clearer for the model and improving the consistency of the generated results. COT-prompt introduces a Chain of Thought (COT) reasoning mechanism, which prompts the large model to think step by step and output a structured report, guiding the large model to think progressively.
4. The method according to claim 1, characterized in that: The uncertainty quantification method in step 5 is as follows: First, the original kidney CT image free text report and prompt words are input into the same model. The temperature parameter Temperature is a key parameter in the large language model sampling process, which is used to control the randomness of the output results (Temperature=1 is the original probability distribution, Temperature>1 enhances randomness, and Temperature<1 enhances determinism). Under the constraint of the temperature parameter Temperature, the reasoning is repeated five times independently to obtain five structured reports. According to the preset three-level quality assessment category C={c1: "completely correct", c2: "partially incorrect", c3: "completely incorrect"}, each structured report is classified, and the frequency of each category in the five results is counted to obtain the category probability distribution P(C). The calculation formula of P is shown in formula (1): （1） Based on the probability distribution, the information entropy H is calculated according to formula (2): （2） Based on the information entropy value H, the confidence level of the model prediction is divided into two levels: when 0 ≤ H < 0.3, it is considered high confidence, indicating that the model has a high degree of certainty about the current prediction result; when H ≥ 0.3, it is considered low confidence, indicating that the model's prediction result has a high degree of uncertainty.
5. A structured conversion system for renal CT image reports based on a large language model, characterized in that, It includes the following modules: a data storage module for storing free-text report datasets of kidney CT images; and a structured template module for building structured report templates. The prompt word engineering module is used to guide the generation of structured reports from large language models using prompt words. The uncertainty quantification module calculates the predicted entropy value based on Temperature sampling and outputs the confidence level results to evaluate the confidence level of the structured report generated by the model. The structured report output module outputs optimal structured reports.
6. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the kidney CT image report structure conversion method according to any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the kidney CT image report structure conversion method according to any one of claims 1-4.