Method and system for automatically generating radiation report based on radiation priori knowledge

Abstract

The invention relates to the field of radiology department medicine and computers, and discloses a radiology report automatic generation method and system based on radiology priori knowledge, which comprises the following steps: constructing a radiology priori knowledge performing document dicing on the priori knowledge of a radiology report through a sliding window so as to construct the radiology priori knowledge base; data set reconstruction and fine tuning: constructing text data of a radiation report, and performing instruction fine tuning on the text data of the radiation report by using a large language model; construction of a comprehensive retrieval vectorization module: constructing the comprehensive retrieval vectorization module in combination with semantic similarity retrieval and full-text retrieval technologies; and automatically generating a radiation report by combining the matched radiation priori knowledge after passing through the comprehensive retrieval vectorization module. According to the system, the radiation prior knowledge base, the comprehensive retrieval vectorization module and the report generation module are integrated in advance, so that a doctor can generate a high-quality radiation image report according with a standard format only by inputting radiation diagnosis information.

Description

Technical Field

[0001] The invention relates to the fields of radiology medicine and computers, and in particular to a method and system for automatically generating a radiology report based on radiology priori knowledge. Background Art

[0002] Radiology plays a pivotal role in medical diagnosis and is an indispensable examination method in clinical practice and medical research. The traditional process of writing radiology examination reports consumes a lot of time, especially for less experienced radiologists, the timeliness, professionalism and accuracy of report writing are more difficult to guarantee. In view of this, with the continuous development of large language model technology, the demand for automatic generation of radiology reports is becoming increasingly urgent. This technology aims to reduce the workload of radiologists while improving the speed and accuracy of writing radiology reports.

[0003] While large language models have developed rapidly in general fields, scholars have gradually applied them to specialized fields.

[0004] For example, prior art 1: CN117037994A uses large language model technology to guide students to update diagnostic reports based on standard diagnostic information. This application provides assistance for students to write diagnostic reports but does not fully automate the generation of reports. In addition, this application does not embed additional prior knowledge to enhance the professionalism of medical terminology.

[0005] Prior art 2: CN114860899A provides a small sample question answering method that introduces prior knowledge, which proves that the quality of text generation can be effectively improved by introducing a small amount of prior knowledge.

[0006] Prior art 3: CN117828050 provides a TCM question-answering method based on long document retrieval and enhanced generation, in which document segmentation is performed only according to the document length threshold and the overlap between the previous and next documents, but does not involve the sorting or integration of summary information of each document block. Summary of the invention

[0007] The present invention provides a method and system for automatically generating radiological reports based on radiological prior knowledge. By pre-integrating a radiological prior knowledge base, a comprehensive retrieval vectorization module and a report generation module, doctors only need to input radiological diagnostic information to generate high-quality radiological image reports that conform to the standard format.

[0008] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0009] A method for automatically generating a radiology report based on radiology prior knowledge, the method comprising:

[0010] The construction of radiology prior knowledge base, through the sliding window to slice the prior knowledge of radiology reports, so as to construct the radiology prior knowledge base;

[0011] Dataset reconstruction and fine-tuning, by constructing the text data of radiology reports and using a large language model to fine-tune the text data of radiology reports;

[0012] The construction of comprehensive search vectorization module is carried out by combining semantic similarity search and full-text search technology;

[0013] The generation of radiology reports is carried out automatically by combining radiology prior knowledge matched after comprehensive retrieval and vectorization modules.

[0014] As a preferred embodiment, the construction of the radiological prior knowledge base includes:

[0015] Collection of radiology report data, including the anatomical structure of each organ system, radiology report templates, abnormal diagnosis and treatment knowledge, and radiology technology related knowledge;

[0016] Cleaning of radiology report prior knowledge: converting non-text radiology report prior knowledge in radiology report data into pure text radiology report data through data processing technology; cleaning of radiology report data by removing punctuation marks and stop words;

[0017] Document segmentation of radiology reports: segmenting radiology report prior knowledge documents through sliding windows, thus completing the construction of radiology report prior knowledge base;

[0018] Updating of radiology report prior knowledge base, updating and storing the constructed radiology prior knowledge base.

[0019] As a preferred embodiment: the document segmentation of the radiology report forms a radiology prior knowledge base P by dynamically adjusting the window size w and the context summary function f;

[0020] P = h{C1, C2, ...C i , …, C n};

[0021] Among them, h is the performance evaluation function, {C1, C2, ...C i , …, C n}To evaluate performance;

[0022] C i =RELU(W1*d i +W2*A i + b);

[0023] Among them, W1 is the first weight matrix, W1 is the second weight matrix, and b is the bias term;

[0024] d i is the i-th document of n documents into which the original document D is split;

[0025] Split the original radiology report document D into n text blocks, denoted as {d1, d2, ..., d n}; The variable w represents a window size, that is, d i The number of blocks to consider on either side of the text block;

[0026] For each text block d i , generate a summary S i , Abstract i Contains i Information about the text block;

[0027] Calculate each text block d i Additive contextual summary A i , A i Include text block d i and context S i The correlation between i =f(d i ,S i ), f is the attention mechanism function, which focuses on S according to the importance of the context block i The information in is weighted averaged;

[0028] Each block d i With its additive contextual summary A i Combined to form the final input block C i ; C i =g(d i ,A i ), g is a function that combines the original block and its additive context summary;

[0029] C i =RELU(W1·d i +W2·A i +b), where W1 and W2 are weight matrices, b is the bias term, and RELU is the activation function;

[0030] The sliding window document segmentation method dynamically adjusts the window size w and the context summary function f.

[0031] According to the block set {C1,C2,...,C n}Evaluate system performance.

[0032] As a preferred method: construct a comprehensive search vectorization module, and obtain the comprehensive search score S through the final weighted averagefinal Construct a comprehensive retrieval vectorization module;

[0033] S final =αS iv (q,d iv )+βS f (q,d iv ),in,

[0034] Comprehensive search combines semantic similarity search and full-text search to build a comprehensive search vectorization module, including:

[0035] q v The vector for the i-th document block is: d iv is the query text vector; the vector retrieval similarity S between the i-th document blocks iv (q,d iv ),

[0036]

[0037] where q·d iv represents the dot product of the text vector and the document block vector, ||q||·||d iv || represents the Euclidean distance between two vectors;

[0038] Thus, we can get the full-text retrieval score S of the two vectors f (q,d iv ):

[0039]

[0040] Where IDF(t) is the inverse document frequency of term t, f t,d is the probability of term t appearing in document d, k1 is the adjustment parameter, b is the adjustment parameter that controls the degree of normalization of the document, N d is the length of the document block, avg(N) is the average length of the documents in the entire document set;

[0041]

[0042] Where N is the total number of documents, n t is the number of documents containing term t,

[0043] The final weighted average comprehensive retrieval score S final for:

[0044] S final =αS iv (q,d iv )+βS f (q,d iv ),in,

[0045] Preferably, the generation of the radiology report includes:

[0046] Data acquisition, which is used to capture the user's text input in real time and receive radiological examination information for integration. Encryption technology and access control mechanisms are used in the data acquisition process to ensure data security and privacy protection;

[0047] Extract radiological prior knowledge, combine the acquired content, and quantize and match radiological prior knowledge through comprehensive retrieval;

[0048] Report generation, which uses the extracted radiology prior knowledge combined with a fine-tuned large language model to generate accurate radiology reports;

[0049] Corrective feedback: Provide corrective feedback on generated radiology reports, systematically learn doctors' usage habits and continuously optimize the quality of report generation.

[0050] In order to solve the above technical problems, the present invention also provides a radiation report automatic generation system based on radiation prior knowledge, which includes:

[0051] The module for constructing the radiology prior knowledge base is to segment the prior knowledge of radiology reports into documents through a sliding window, thereby constructing the radiology prior knowledge base.

[0052] The dataset reconstruction and fine-tuning module constructs the text data of radiology reports and uses a large language model to fine-tune the text data of radiology reports;

[0053] The construction module of the comprehensive search vectorization module combines semantic similarity search and full-text search technology to construct the comprehensive search vectorization module;

[0054] The radiology report generation module automatically generates radiology reports by combining radiology prior knowledge matched after comprehensive retrieval and vectorization modules.

[0055] The present invention has significant technical effects due to the adoption of the above technical solution:

[0056] The present invention proposes a sliding window cutting method to segment the text to complete the construction of a radiology prior knowledge base, and proposes a comprehensive retrieval vectorization technology to match the radiology prior knowledge base, while combining large language model related technologies to achieve automatic generation of radiology reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of the present invention;

[0058] Figure 2 is a flow chart of constructing a radiological prior knowledge base according to the present invention;

[0059] Figure 3 It is a recall rate evaluation graph of the fine-tuned large language model of the present invention;

[0060] Figure 4 It is a schematic diagram of the comprehensive retrieval vectorization module constructed in the present invention;

[0061] Figure 5 It is a schematic diagram of a unit of a report automatic generation system based on radiological prior knowledge of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0063] Example 1

[0064] The present application embodiment provides a method for constructing a radiological prior knowledge base, such as Figure 2 , obtain prior knowledge in the field of radiology, introduce the sliding window method to optimize the document segmentation method, and thus construct a radiology prior knowledge base.

[0065] Constructing a radiological prior knowledge base specifically involves the collection of data, cleaning of radiological knowledge and document segmentation, persistent storage of knowledge and updating of the knowledge base.

[0066] Prior knowledge in the field of radiology comes from radiation-related knowledge in various formats such as e-books, PDF reports, academic papers, case data, etc., including the anatomical structure of various organ systems, radiology report templates, abnormal diagnosis and treatment knowledge, and knowledge related to radiology technology.

[0067] Collect prior knowledge in the field of radiology and process and clean it up. As a feasible implementation method, non-textual prior radiology data is converted into plain text data with the help of data processing technology, such as using OCR technology for image recognition and key text information extraction; using the table unstructured method to automatically analyze the table data in the data, construct a table index based on the table title, convert the extracted table data into a summary, and finally combine the table index and summary to generate the complete information of the table.

[0068] The extracted text data is integrated and preprocessed, such as removing punctuation marks and stop words, to obtain the summarized radiological prior knowledge documents. An optimized document segmentation method, i.e., a sliding window-based method, is used to segment the radiological prior knowledge documents, thereby completing the construction of the radiological prior knowledge base.

[0069] Specifically, the original document D is split into n text blocks, denoted as {d1, d2, ..., d n}. Define the variable w to represent a window size, that is, d iThe number of blocks that need to be considered on both sides of the text block. In this embodiment, the window size w is dynamically adjusted according to the indicator, thereby determining the optimal context range of the text block.

[0070] For each text block d i , generate a summary S i , the abstract contains d i The information of the text block. Then calculate each block d i Additive contextual summary A i , A i Include text block d i and context S i The correlation between i =f(d i ,S i As a feasible implementation method, in this embodiment, f is an attention mechanism function, which is used to evaluate S according to the importance of the context block. i The information in is weighted averaged.

[0071] Finally, each block d i With its additive contextual summary A i Combined to form the final input block C i :C i =g(d i ,A i ), where g is a function that combines the original block and its additive context summary. C i =RELU(W1·d i +W2·A i +b), where W1 and W2 are weight matrices, b is the bias term, and RELU is the activation function.

[0072] The sliding window document segmentation method can optimize the performance P of the splitting method by dynamically adjusting the window size w and the context summary function f, which is expressed as: P = h({C1, C2, ..., C n}), where h is a performance evaluation function, according to the final block set {C1,C2,...,C n}Evaluate system performance.

[0073] Example 2

[0074] Based on Example 1, this example specifically involves constructing a radiology report text dataset and fine-tuning the instructions of a large language model.

[0075] The radiology report text data including examination type, imaging findings and diagnosis conclusions are collected. The data is desensitized and cleaned text and does not involve privacy and sensitive information.

[0076] A high-quality fine-tuning instruction dataset is constructed through manual evaluation and scoring by models such as GPT-4. The constructed data includes user instructions, positive outputs, and negative outputs. The implementation method is as follows:

[0077]

[0078]

[0079] According to the above template, the specific input information, imaging findings, and diagnostic conclusions are extracted from the radiology report text after desensitization and cleaning, so as to complete the construction of a radiology report text generation data set with a number of N. The ORPO method, a reinforcement feedback learning method, is selected to fine-tune the large language model.

[0080] With ORPO, LLM simultaneously learns to answer commands and satisfy human preferences.

[0081] The loss function is calculated as follows during fine-tuning:

[0082]

[0083] L ORPO =Ε (x,yw,yl) [L SFT +λ·L OR ]

[0084] Among them, L OR is a contrast loss function, odds θ (y w |x) is the correct category y when the model parameter is θ and the input is x w Odds θ (y l |x) is the error category y when the model parameter is θ and the input is x w The odds of, σ is the sigmoid function, L OR Strengthens the model's ability to distinguish between correct and incorrect categories. SFT is the standard cross entropy loss, and the hyperparameter λ is used to adjust L OR and L SFT Previous weight.

[0085] The fine-tuned large language model can learn to answer instructions and give data formats that meet human preferences. The recall hit rate evaluation index values ​​of the fine-tuned model are as follows: Figure 3 As shown, by analyzing the ROUGE and BLEU evaluation scores, it can be concluded that the fine-tuned large language model in this embodiment can already complete the generation task well.

[0086] Example 3

[0087] Based on Example 1, Figure 4 The comprehensive search vector in this embodiment includes:

[0088] Comprehensive search combines semantic similarity search and full-text search technology. Semantic similarity search is based on segmented text vector indexing, and vectorization is used to calculate the similarity between the input text and the knowledge base document fragments. The vectorization module combines the keyword matching technology of full-text search, and achieves accurate comprehensive search by weighted averaging the vectorized similarity and keyword matching results;

[0089] The query text vector is: q v , the i-th document block vector is: d iv , then the vector retrieval similarity between the query text vector and the i-th document block is expressed as: where q·d iv represents the dot product of the text vector and the document block vector, ||q||·||d iv || represents the Euclidean distance between two vectors. From this, we can get the full-text retrieval scores of the two vectors:

[0090]

[0091] Among them, IDF(t) represents the inverse document frequency of term t, which is calculated as follows: Where N is the total number of documents, n t is the number of documents containing term t. f in the above formula t,d represents the probability of term t appearing in document d, k1 represents the adjustment parameter, b represents the adjustment parameter, and controls the degree of normalization of the document. d Represents the length of the document block, and avg(N) represents the average length of the documents in the entire document set.

[0092] The final weighted average comprehensive search score is:

[0093] S final =αS iv (q,d iv )+βS f (q,d iv ), which needs to meet

[0094] Then, the comprehensive scores of the query vector and each document block vector are obtained for comprehensive ranking, and the top N document blocks are selected as the final best matching N document blocks.

[0095] Example 4

[0096] Based on the above embodiments, this embodiment Figure 5To automate the reporting system, the main components of the system include: data acquisition unit, which is used to capture the user's text input in real time and receive radiological examination information for integration. The unit adopts encryption technology and access control mechanism in the data acquisition process to ensure data security and privacy protection;

[0097] The radiological prior knowledge extraction unit, in combination with the content acquired by the data acquisition unit, performs comprehensive retrieval and vectorized matching of radiological prior knowledge through the third embodiment.

[0098] First, the user input text and radiological examination information integrated by the data acquisition unit are received, and the input text is efficiently searched and matched in the radiological prior knowledge base using the comprehensive search module described in Example 3, and the Top N similar text blocks are retrieved as the most critical radiological prior knowledge of the current input text.

[0099] The report generation unit generates an accurate radiology report by using the radiology prior knowledge extracted by the radiology prior knowledge extraction unit and the large language model fine-tuned in the second embodiment.

[0100] The system uses the radiological prior knowledge extracted by the radiological prior knowledge extraction unit as the contextual supplementary information of the fine-tuned large language model. At this time, the prompt content input into the large model is:

[0101] Find the answer to the user """{{input content}}""" from the retrieved document block content """{{knowledge}}""" without mentioning the source of the knowledge. Just output the complete radiology report directly in JSON format.

[0102] The model strictly generates a complete radiology report containing imaging findings and diagnostic conclusions based on the current contextual knowledge. In addition, the JSON format report generated by the report generation unit is easy to parse the report content and is convenient for transmission and interface display.

[0103] The correction feedback unit provides correction feedback on the radiology reports generated by the report generation unit, systematically learns doctors' usage habits and continuously optimizes the quality of report generation.

[0104] The correction feedback unit provides correction feedback methods such as text editing, error marking, and quality scoring. Text correction means that doctors can directly edit the generated report text to correct inaccurate or erroneous content; error marking means that doctors can point out description errors, omission errors, etc. in the report through the marking tools provided by the system; quality scoring means that doctors can score the quality of the report, and the system evaluates the satisfaction with the report generation based on the score.

[0105] The system integrates the collected feedback information and extracts knowledge to regularly update the radiology prior knowledge base. The feedback data such as quality scores accumulated by the feedback unit are regularly integrated and corrected for retraining the large language model, thus forming a closed-loop feedback loop system, which enables the system to continuously optimize the quality of report generation.

[0106] Example 5

[0107] Based on the above embodiment, this embodiment is a radiology report automatic generation system based on radiology prior knowledge, which includes: a radiology prior knowledge base construction module, which performs document segmentation on the prior knowledge of the radiology report through a sliding window, thereby constructing a radiology prior knowledge base;

[0108] The dataset reconstruction and fine-tuning module constructs the text data of radiology reports and uses a large language model to fine-tune the text data of radiology reports;

[0109] The construction module of the comprehensive search vectorization module combines semantic similarity search and full-text search technology to construct the comprehensive search vectorization module;

[0110] The radiology report generation module automatically generates radiology reports by combining radiology prior knowledge matched after comprehensive retrieval and vectorization modules.

Claims

1. A method for automatically generating a radiology report based on radiology prior knowledge, the method comprising: The construction of radiology prior knowledge base, through the sliding window to slice the prior knowledge of radiology reports, so as to construct the radiology prior knowledge base; Dataset reconstruction and fine-tuning, by constructing the text data of radiology reports and using a large language model to fine-tune the text data of radiology reports; The construction of comprehensive search vectorization module is carried out by combining semantic similarity search and full-text search technology; The generation of radiology reports is carried out automatically by combining radiology prior knowledge matched after comprehensive retrieval and vectorization modules.

2. The method for automatically generating a radiation report based on radiation prior knowledge according to claim 1, characterized in that: The construction of radiological prior knowledge base includes: Collection of radiology report data, including the anatomical structure of each organ system, radiology report templates, abnormal diagnosis and treatment knowledge, and radiology technology related knowledge; Cleaning of radiology report prior knowledge: converting non-text radiology report prior knowledge in radiology report data into pure text radiology report data through data processing technology; cleaning of radiology report data by removing punctuation marks and stop words; Document segmentation of radiology reports: segmenting radiology report prior knowledge documents through sliding windows, thus completing the construction of radiology report prior knowledge base; Updating of radiology report prior knowledge base, updating and storing the constructed radiology prior knowledge base.

3. The method for automatically generating a radiation report based on radiation prior knowledge according to claim 2, characterized in that: The document segments of radiology reports are segmented to form a radiology prior knowledge base P by dynamically adjusting the window size w and the context summary function f; P=h{C1,C2,…C i ,…,C n }; Among them, h is the performance evaluation function, {C1, C2, ...C i , …, C n }To evaluate performance; C i =RELU(W1*d i +W2*A i +b); Among them, W1 is the first weight matrix, W1 is the second weight matrix, and b is the bias term; d i is the i-th document of n documents into which the original document D is split; Split the original radiology report document D into n text blocks, denoted as {d1, d2, ..., d n }; The variable w represents a window size, that is, d i The number of blocks to consider on either side of the text block; For each text block d i , generate a summary S i , Abstract i Contains i Information about the text block; Calculate each text block d i Additive contextual summary A i , A i Include text block d i and context S i The correlation between i =f(d i ,S i ), f is the attention mechanism function, which focuses on S according to the importance of the context block i The information in is weighted averaged; Each block d i With its additive contextual summary A i Combined to form the final input block C i ; C i =g(d i ,A i ), g is a function that combines the original block and its additive context summary; C i =RELU(W1·d i +W2·A i +b), where W1 and W2 are weight matrices, b is the bias term, and RELU is the activation function; The sliding window document segmentation method dynamically adjusts the window size w and the context summary function f. According to the block set {C1,C2,...,C n }Evaluate system performance.

4. The method for automatically generating a radiation report based on radiation prior knowledge according to claim 1, characterized in that: The construction of the comprehensive search vectorization module, and the comprehensive search score S obtained by the final weighted average final Construct a comprehensive retrieval vectorization module; S final =αS iv (q, d iv )+βS f (q, d iv ), among them, Comprehensive search combines semantic similarity search and full-text search to build a comprehensive search vectorization module, including: q v The vector for the i-th document block is: d iv is the query text vector; the vector retrieval similarity S between the i-th document blocks iv (q,d iv ), where q·d iv represents the dot product of the text vector and the document block vector, ||q||·||d iv || represents the Euclidean distance between two vectors; Thus, we can get the full-text retrieval score S of the two vectors f (q,d iv ): Where IDF(t) is the inverse document frequency of term t, f t,d is the probability of term t appearing in document d, k1 is the adjustment parameter, b is the adjustment parameter that controls the degree of normalization of the document, N d is the length of the document block, avg(N) is the average length of the documents in the entire document set; Where N is the total number of documents, n t is the number of documents containing term t, The final weighted average comprehensive retrieval score S final for: S final =αS iv (q,d iv )+βS f (q,d iv ), among them, 5. The method for automatically generating a radiation report based on radiation prior knowledge according to claim 1, characterized in that: Radiology report generation includes: Data acquisition, which is used to capture the user's text input in real time and receive radiological examination information for integration. Encryption technology and access control mechanisms are used in the data acquisition process to ensure data security and privacy protection; Extract radiological prior knowledge, combine the acquired content, and quantize and match radiological prior knowledge through comprehensive retrieval; Report generation, which uses the extracted radiology prior knowledge combined with a fine-tuned large language model to generate accurate radiology reports; Corrective feedback: Provide corrective feedback on generated radiology reports, systematically learn doctors' usage habits and continuously optimize the quality of report generation.

6. A system for automatically generating radiological reports based on radiological prior knowledge, characterized in that: include The module for constructing the radiology prior knowledge base is to segment the prior knowledge of radiology reports into documents through a sliding window, thereby constructing the radiology prior knowledge base. The dataset reconstruction and fine-tuning module constructs the text data of radiology reports and uses a large language model to fine-tune the text data of radiology reports; The construction module of the comprehensive search vectorization module combines semantic similarity search and full-text search technology to construct the comprehensive search vectorization module; The radiology report generation module automatically generates radiology reports by combining radiology prior knowledge matched after comprehensive retrieval and vectorization modules.

Images