Medical image report generation method and system based on large language model

Abstract

The invention discloses a medical image report generation method and system based on a large language model, and relates to the field of image report generation, and the method comprises the steps: firstly obtaining original image data and a clinical background text of a patient, and respectively extracting an image embedding vector and a background embedding vector; then, case retrieval based on priori knowledge is carried out by utilizing the embedded vectors, and K highly related historical case reports are screened out from massive historical data; and inputting the image embedding vector and the historical case report into an observation large language model, and outputting the image embedding vector and the historical case report in a structured JSON (JavaScript Object Notation) format. And finally, a large language model is written to integrate the visual evidence JSON, the clinical background text and the historical case report, and a final medical image report is generated. According to the mode, through a staged and multi-modal fusion mode, the problems of incoherent report logic, inaccurate information and the like are effectively solved, and the report quality and the generation efficiency are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image report generation, and more specifically, to a method and system for generating medical image reports based on a large language model. Background Art

[0002] With the rapid advancement of medical technology, medical imaging plays an increasingly important role in disease diagnosis and treatment. However, the writing of medical imaging reports is often done manually by radiologists, which is not only time-consuming and labor-intensive, but also subject to variations in report quality and consistency based on physician experience and subjective judgment. Doctors often face tremendous workload pressure when faced with massive amounts of imaging data, making the need for automated, intelligent generation of high-quality medical imaging reports increasingly urgent.

[0003] Although some studies have attempted to utilize artificial intelligence technologies to assist in the generation of medical imaging reports, such as methods based on template matching or rule engines, these methods often lack flexibility and struggle to handle complex clinical situations and diverse imaging manifestations. In recent years, large language models (LLMs) have achieved breakthroughs in the field of natural language processing, demonstrating powerful text generation and comprehension capabilities, providing new approaches for the automated generation of medical imaging reports. However, the direct application of large language models to medical imaging report generation remains challenging. Traditional LLMs often struggle to effectively integrate visual information when processing medical imaging data, and lack a deep understanding of medical expertise. This can lead to reports that are logically incoherent, contain inaccurate information, or lack clinical relevance. Furthermore, existing methods often fail to fully leverage the valuable experience of historical cases when generating reports, leaving room for improvement in the accuracy and reliability of the reports.

[0004] Therefore, there is an urgent need for a medical imaging report generation method and system that can effectively integrate medical imaging data, clinical background information and historical case experience. Summary of the Invention

[0005] In view of the above limitations of the existing methods, according to one aspect of the present application, there is provided a method for generating a medical image report based on a large language model, which includes: obtaining the original image data and clinical background text of a target patient object; extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; performing prior knowledge-based case retrieval based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain a visual evidence JSON; and inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

[0006] According to another aspect of the present application, there is provided a system for generating a medical image report based on a large language model, which includes: an image data background text acquisition module for obtaining the original image data and clinical background text of a target patient object; an image data background text acquisition module for extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; an image data background text acquisition module for performing prior knowledge-based case retrieval based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; an image data background text acquisition module for inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain a visual evidence JSON; and an image data background text acquisition module for inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

[0007] Compared with the prior art, a method and system for generating medical image reports based on large language models provided by the present application first obtain the original image data and clinical background text of a patient, and respectively extract image embedding vectors and background embedding vectors to achieve a preliminary digital representation of multi-modal information, solving the problem that traditional LLMs are difficult to effectively integrate visual information. Then, these embedding vectors are used for case retrieval based on prior knowledge, and K highly relevant historical case reports are screened out from a large amount of historical data to provide rich professional references for subsequent report generation, making up for the deficiency of existing methods in lacking the utilization of historical experience. Subsequently, the image embedding vectors and historical case reports are input into the observation large language model, which focuses on extracting objective visual evidence from the images and outputs it in a structured JSON format, ensuring the objectivity and accuracy of the report. Finally, the writing large language model integrates the visual evidence JSON, clinical background text, and historical case reports to generate the final medical image report. Through a phased and multi-modal fusion method, problems such as incoherent report logic and inaccurate information are effectively solved, significantly improving the quality and generation efficiency of the report. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features, and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail with reference to the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation to the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 It is a flowchart of a method for generating a medical image report based on a large language model according to an embodiment of the present application.

[0010] Figure 2 It is a schematic diagram of data flow of a method for generating a medical image report based on a large language model according to an embodiment of the present application.

[0011] Figure 3 It is a flowchart of step S2 in a method for generating a medical image report based on a large language model according to an embodiment of the present application.

[0012] Figure 4 It is a flowchart of step S3 in a method for generating a medical image report based on a large language model according to an embodiment of the present application.

[0013] Figure 5 It is a block diagram of a system for generating a medical image report based on a large language model according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0015] In view of the problems in the above background art, the present application proposes a method for generating medical image reports based on large language models. Figure 1 FIG. is a flowchart of a method for generating medical image reports based on large language models according to an embodiment of the present application. Figure 2 FIG. is a schematic diagram of data flow of a method for generating medical image reports based on large language models according to an embodiment of the present application. As Figure 1 and Figure 2 shown, the method for generating medical image reports based on large language models according to an embodiment of the present application includes: S1, obtaining the original image data and clinical background text of a target patient object; S2, extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; S3, performing case retrieval based on prior knowledge based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; S4, inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain a visual evidence JSON; S5, inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

[0016] In step S1, the original image data and clinical background text of a target patient object are obtained. It should be understood that the generation of medical image reports is essentially a process of multi-modal information fusion and understanding. The original image data carries the visual features of the patient's lesions and is the direct basis for diagnosis; while the clinical background text provides non-image information such as the patient's medical history, symptoms, and examination results, which plays an indispensable role in comprehensively understanding the condition and assisting diagnosis. Traditional methods for generating medical image reports often have difficulty effectively integrating these heterogeneous data, resulting in one-sided or inaccurate reports. Therefore, obtaining these two key pieces of information is the cornerstone for subsequent steps of feature extraction, case retrieval, and final report generation, ensuring that the model can obtain the complete information of the patient from multiple dimensions, thereby generating a more comprehensive and accurate medical image report.

[0017] For step S1, one possible implementation is as follows: First, the original image data of the target patient object can be obtained through various channels. For example, it can be directly retrieved from the Picture Archiving and Communication System (PACS) of the hospital. These data are stored in DICOM, the Digital Imaging and Communications in Medicine format, and contain the original scan images of the patient's CT, MRI, X-ray, etc. In actual operation, when a doctor or relevant personnel need to generate a medical image report for a specific patient, they will select or input the patient ID through the user interface, and the system will then retrieve and download the corresponding original image data from the PACS server. For example, for a patient undergoing a lung CT examination, the system will obtain a series of DICOM images in multiple axial, coronal, and sagittal planes.

[0018] For the clinical background text of the target patient object, the acquisition method includes extracting it from the hospital's Electronic Health Record system (EHR). These text data may include the patient's chief complaint, current medical history, past medical history, family history, allergy history, physical examination results, laboratory test results, medication conditions, and other relevant clinical records. During implementation, the system will query and extract the clinical background text related to the current imaging examination from the EHR database according to the patient ID or visit record. For example, for the above-mentioned lung CT patient, the system will obtain the chief complaint of coughing and expectorating for 3 weeks accompanied by fever, as well as the clinical information such as a past history of hypertension. These text data exist in an unstructured or semi-structured text format, such as plain text, XML, or JSON format.

[0019] In step S2, visual features are extracted from the original image data to obtain an image embedding vector, and text semantic features are extracted from the clinical background text to obtain a background embedding vector. It can be understood that the original medical image data (such as DICOM images) and clinical background text (such as medical history records) are heterogeneous and high-dimensional original information, which cannot be effectively processed by large language models directly. In order to enable these multimodal information to be understood and utilized by the subsequent case retrieval and report generation modules, it is necessary to convert them into a unified, low-dimensional, and numerically represented form with semantic information, that is, an embedding vector. Therefore, in the technical solution of this application, by extracting visual features and text semantic features, complex original data are mapped into a continuous vector space, so as to capture the lesion information in the image and the clinical semantics in the text, providing a standardized input for subsequent similarity calculation and model inference. This solves the problems mentioned in the background technology that traditional LLMs are difficult to effectively fuse visual information and lack a deep understanding of medical expertise, and is a key link in realizing multimodal information fusion and intelligent report generation.

[0020] Specifically, in an exemplary embodiment of this application, Figure 3It is a flowchart of step S2 in the medical image report generation method based on a large language model according to an embodiment of the present application. As Figure 3 shown, in step S2, visual features are extracted from the original image data to obtain an image embedding vector, and text semantic features are extracted from the clinical background text to obtain a background embedding vector, including: S21, preprocessing the original image data to obtain preprocessed original image data; S22, inputting the preprocessed original image data into a pre-trained visual encoder to obtain the image embedding vector, where the pre-trained visual encoder is a MedNeXt model; S23, inputting the clinical background text into a text editor to obtain the background embedding vector, and the text editor is a BERT model.

[0021] It is worth mentioning that the MedNeXt model is used as a visual encoder because this model is specifically designed for medical image analysis, especially three-dimensional volumetric data (such as CT, MRI). Different from general two-dimensional image models, MedNeXt inherits the excellent performance of the modern convolutional network ConvNeXt and is optimized for the characteristics of medical images, and can effectively capture the morphology, structure, and adjacency relationships of lesions in three-dimensional space. This ensures that the image embedding vector extracted from the original image data can highly condense and accurately represent complex pathological features, providing a high-quality visual evidence basis for subsequent similar case retrieval and report generation.

[0022] The reason for choosing the BERT model to process the clinical background text is that BERT, as a Transformer-based bidirectional encoder, has excellent capabilities in understanding the deep context semantics of natural language. The clinical background text contains a large number of key medical terms and patient descriptions with complex logical relationships. Through its unique masked language model and next sentence prediction tasks, BERT can capture the subtle meaning differences of words in different contexts, thereby transforming the clinical background text into a background embedding vector rich in semantic information that can accurately reflect the patient's medical history and current condition. This high-quality text representation is crucial for the subsequent large language model to associate image findings with the clinical context and generate clinically relevant diagnostic opinions.

[0023] For step S2, one possible implementation is as follows: First, perform S21. After obtaining the original image data of the target patient object (e.g., a CT image sequence in DICOM format), a series of preprocessing operations need to be performed on it to eliminate noise, standardize the data format, and optimize the image quality to make it more suitable for the input of the deep learning model. Specifically, in an exemplary embodiment of the present application, preprocessing the original image data to obtain preprocessed original image data includes: performing window width and window level adjustment, normalization, and size adjustment on the original image data to obtain the preprocessed original image data. In detail, window width and window level adjustment means adjusting the gray value range of the image for different medical imaging modalities and observation purposes to highlight the organizational structures or lesions of interest. For example, for lung CT images, the window width is set to 1500 HU, Hounsfield unit, and the window level is set to -500 HU to clearly display the lung parenchyma and tracheobronchial structures; while for bone windows, it may be set to a window width of 2000 HU and a window level of 300 HU to observe bone details. These window width and window level values are preset according to clinical experience and imaging standards. Normalization means scaling the pixel values of the image to a standard range, such as [0,1] or [-1,1], to eliminate the gray level differences caused by different scanning devices or scanning parameters and ensure the stability of model training. This is achieved through a linear transformation, i.e., (pixel value - minimum value) / (maximum value - minimum value). Size adjustment means uniformly adjusting all images to a preset fixed size to meet the input requirements of the visual encoder model. For example, a three-dimensional CT image can be adjusted to a voxel size of 256x256x256 to ensure the consistency of the input dimension. After the above preprocessing, the original image data is converted into preprocessed original image data.

[0024] Next, perform S22. Take the preprocessed original image data obtained in step S21 as input and feed it into the pre-trained MedNeXt model. It is worth mentioning that MedNeXt is a deep convolutional neural network specifically designed for medical image analysis. Its architecture includes multiple 3D convolutional layers, batch normalization layers, activation functions (such as ReLU), and pooling layers. By stacking these modules, hierarchical features of the image are gradually extracted. It uses 3D convolutional kernels to capture the three-dimensional spatial information of medical images and alleviates the vanishing gradient problem in the training of deep networks through mechanisms such as residual connections. The MedNeXt model has been pre-trained on a large number of medical image datasets, learning rich medical image feature representations. Its weight and bias parameters are obtained through supervised learning on large medical image classification, segmentation, or reconstruction tasks, aiming to minimize the prediction error. When the preprocessed original image data is input into the MedNeXt model, the data will go through a series of convolutions, poolings, and non-linear transformations, and finally output a vector with a fixed dimension before the global average pooling layer or the fully connected layer at the last layer of the model, that is, the image embedding vector.

[0025] Then, perform S23. Take the clinical background text as input and feed it into the pre-trained BERT model. Those skilled in the art should be aware that BERT is a bidirectional encoder based on the Transformer architecture. Its core is the multi-head self-attention mechanism, which can capture the complex dependencies and context information between words in the text. The BERT model has been pre-trained unsupervised on a large amount of general text corpora, learning rich language representations. Its weight and bias parameters are obtained through a large number of trainings. When the clinical background text is input into the BERT model, the text will first be tokenized and converted into word embeddings, and then processed by multiple layers of Transformer encoders. Finally, a vector with a fixed dimension is obtained at the output position corresponding to the [CLS] token of the model, that is, the background embedding vector.

[0026] In step S3, based on the background embedding vector and the image embedding vector, a prior-knowledge-based case retrieval is performed to obtain a list containing K historical case report texts. It should be understood that as pointed out in the background art, the existing methods fail to fully utilize the valuable experience of historical cases, resulting in the need to improve the accuracy and reliability of the reports. Based on this, the present application can, through the introduction of prior-knowledge-based case retrieval, intelligently find cases from a large historical image-report database that are highly similar to the current patient's image features and clinical background. These similar historical case reports contain the diagnostic ideas, descriptions of image findings, and diagnostic opinions of senior doctors, providing rich, high-quality, clinically relevant context information and professional guidance for the subsequent generation of reports by the large language model. This can not only significantly improve the professionalism and accuracy of the generated reports but also effectively make up for the deficiency of the large language model in the in-depth understanding of medical professional knowledge, thus solving the problems of report accuracy and reliability mentioned in the background art.

[0027] Specifically, in an exemplary embodiment of the present application, Figure 4 is a flowchart of step S3 in the method for generating a medical image report based on a large language model according to an embodiment of the present application. As Figure 4 shown, in step S3, based on the background embedding vector and the image embedding vector, a prior-knowledge-based case retrieval is performed to obtain a list containing K historical case report texts, including: S31, splicing the features of the background embedding vector and the image embedding vector to obtain a query embedding vector; S32, using the query embedding vector as a query to perform a similarity retrieval in the image-report vector database to obtain the list containing K historical case report texts.

[0028] It is worth mentioning that as pointed out in the background art, the writing of medical image reports needs to consider both image information and clinical background. Using only the image embedding vector or the background embedding vector for retrieval cannot comprehensively capture the complete features of a case, which may lead to inaccurate or insufficiently relevant historical cases being retrieved. Therefore, in order to create a richer and more comprehensive query embedding vector, the present application splices the features of these two heterogeneous but complementary embedding vectors to fuse the semantic information of the visual and text dual modalities. This fusion enables the subsequent similarity retrieval to more accurately match historical cases that are highly similar to the current case in terms of both image manifestations and clinical background, thereby providing more accurate prior knowledge for the large language model and effectively solving the problems of lack of effective prior knowledge and case reference mentioned in the background art.

[0029] For step S31, one possible implementation is as follows: It should be understood that feature concatenation is a simple and effective feature fusion method that directly concatenates two or more vectors along a certain dimension to form a longer new vector that contains all the original feature information. Specifically, during implementation, the image embedding vector and the background embedding vector are concatenated. For example, if the dimension of the image embedding vector V1 is D1 and the dimension of the background embedding vector V2 is D2, the concatenation operation will generate a query embedding vector Vq with a dimension of D1 + D2.

[0030] Correspondingly, the writing of medical image reports highly depends on the professional experience of radiologists and the reference to previous similar cases. As pointed out in the background art, existing methods fail to fully utilize the valuable experience of historical cases, resulting in the need to improve the accuracy and reliability of reports. By performing an efficient similarity search on the pre-constructed image-report vector database with the query embedding vector that fuses image and clinical background information obtained in step S31, historical cases that are highly similar to the current case in multi-modal features can be intelligently found. These similar historical case reports contain the diagnostic ideas, descriptions of image findings, and diagnostic opinions of senior doctors, providing rich, high-quality, clinically relevant context information and professional guidance for the subsequent generation of reports by the large language model. This can not only significantly improve the professionalism and accuracy of the generated reports but also effectively make up for the deficiency of the large language model in the in-depth understanding of medical expertise, thus solving the problems of report accuracy and reliability mentioned in the background art.

[0031] More specifically, in an exemplary embodiment of the present application, for step S3, using the query embedding vector as a query to perform a similarity search in the image-report vector database to obtain the list containing K historical case report texts includes: S321, calculating the cosine similarity between the query embedding vector and each case embedding vector in the image-report vector database as the case matching degree to obtain a set of case matching degrees; S322, extracting K historical case report texts from the image-report vector database based on the sorting of the set of case matching degrees.

[0032] For step S32, one possible implementation is as follows: First, perform S321. The image-report vector database is a structured data set that stores a large number of historical cases. Each entry in the database corresponds to a historical case and contains the case embedding vector of that case (similarly formed by converting historical images and historical clinical background texts into embedding vectors and concatenating them through methods similar to S2 and S31) and the corresponding original historical case report text.

[0033] For each historical case i in the database, its corresponding case embedding vector is denoted as Query the embedding vector and all case embedding vectors in the database The cosine similarity. Cosine similarity is a measure of the cosine value of the angle between two non-zero vectors, and its value ranges from -1 to 1. The closer the value is to 1, the closer the directions of the two vectors are, that is, the higher the similarity. The calculation formula of cosine similarity is: where represents and the dot product of the vectors, represents the Euclidean norm of the vector. By traversing all historical cases in the image-report vector database, calculate and each cosine similarity to obtain a set containing the matching degrees of all cases. For example, if there are 1 million historical cases in the database, 1 million cosine similarity values will be obtained.

[0034] Finally, perform S322. After obtaining the cosine similarities of all historical cases, that is, the case matching degree set, this step will sort these matching degrees in descending order. After the sorting is completed, select the K historical cases with the highest similarity. It should be noted that K is a preset integer value representing the number of historical cases to be retrieved. The setting of K needs to balance the retrieval efficiency and the report quality. If the value of K is too small, it may not provide rich enough reference information; if the value of K is too large, it will increase the processing burden of the subsequent large language model and may introduce irrelevant noise information. In practical applications, the value of K can be set according to experience. For example, K can be set to 5 or 10, that is, retrieve the 5 or 10 most similar historical cases. From the sorting results, extract the historical case report texts corresponding to these K highest similarities. These texts are the original and complete medical image reports, containing detailed contents such as the findings in the images and the diagnostic opinions. For example, if K is set to 5, 5 historical medical image report texts most similar to the current patient's image and clinical background will be extracted.

[0035] In step S4, the image embedding vector and the list containing K historical case report texts are input into the observation large language model to obtain visual evidence JSON. It can be understood that existing methods do not understand and utilize image information deeply enough, and the generated reports may lack sufficient details and accuracy. Directly inputting the original image data into the large language model for end-to-end generation often makes it difficult to ensure the objectivity and structurality of the output. Based on this, this application introduces an observation large language model and combines the image embedding vector and historical case reports, which can specifically train the model to focus on extracting key and objective lesion features from visual information and output them in a strict JSON format, thus ensuring the accuracy, structurality, and parsability of visual evidence. This phased processing method decouples image analysis and text generation, enabling the model to more effectively focus on the recognition and description of image features and providing high-quality and standardized input for subsequent report writing.

[0036] Specifically, in an exemplary embodiment of the present application, step S4 of inputting the image embedding vector and the list containing K historical case report texts into the observation large language model to obtain visual evidence JSON includes: S41, constructing a medical image observation prompt based on the image embedding vector and the list containing K historical case report texts, where the medical image observation prompt includes listing all objective abnormal findings in a strict JSON format; S42, inputting the medical image observation prompt into the observation large language model to obtain the visual evidence JSON.

[0037] It should be understood that as mentioned in the background art, existing methods do not understand image information deeply enough, and the report structurality needs to be improved. Directly inputting the original image embedding vector into the model, the model may have difficulty understanding its specific meaning and generating a structured description. Therefore, by constructing a clear medical image observation prompt, converting the abstract image embedding vector into a text instruction that the model can understand, and clearly requiring the output of objective abnormal findings in a strict JSON format, it can effectively guide the observation large language model to focus on the recognition and structured description of image features, avoid generating irrelevant information or subjective judgments, thus ensuring the accuracy, objectivity, and parsability of the subsequent visual evidence JSON and providing a high-quality intermediate representation for the generation of the final report.

[0038] For step S41, one possible implementation is as follows: First, texturize or tokenize the image embedding vector. Although large language models can handle text inputs, directly using numerical image embedding vectors as text inputs may be inefficient or semantically unclear. One implementation is to convert the image embedding vector into a short, semantically representative text description or special token sequence through a pre-trained lightweight encoder or mapping layer. For example, a small neural network can be trained to map a 512-dimensional image embedding vector to a descriptive text, such as the image shows abnormal signals in the lungs, possibly involving nodules and pleural effusion. Or, more commonly, the image embedding vector is directly used as a special visual token in the input sequence of the large language model, and the model will learn how to understand and utilize the visual information represented by this token during training.

[0039] Next, integrate the historical case report texts. Integrate the list of K historical case report texts (e.g., 5 complete historical reports) into the prompt. These historical reports provide rich medical terms, image description paradigms, and diagnostic logics, which can be used as references when the large language model generates objective findings. To avoid overly long prompts, the historical reports can be summarized, or only the key paragraphs related to the image findings can be selected. For example, the image findings part of each historical report can be extracted.

[0040] Next, instructions and format requirements are constructed. The core of the prompt is to clearly inform the observational language model of the task to be completed and the output format. This is achieved through pre-set templates or instruction statements. The instructions clearly require the model to: Play a role, for example, please play the role of a senior radiologist and focus on objectively describing imaging findings. Input information specifies the inputs the model needs to refer to, including image features, namely textual or tokenized representations via image embedding vectors, and historical case references. For example, based on the following image features: [Image feature description / tag], and referencing the following similar case reports: [Text list of historical case reports]. The task objective explicitly requires the model to list all objective abnormal findings. The emphasis on objectivity is to avoid the model from making subjective diagnoses or speculations. The output format is mandatory and must be in a strict JSON format. This ensures that subsequent steps can easily parse and utilize this visual evidence. The structure of the JSON format can be pre-defined, for example, containing an array named findings, each element of which is an object describing a single abnormal finding, with keys such as type, location, size, morphology, and density. An example medical image observation prompt might be as follows: "Please play the role of a senior radiologist. Based on the following image features: [Textual representation of the image embedding vector, for example: 'The image shows a solid nodule in the right upper lobe and a small amount of fluid in the left pleural cavity.'], and referring to the following similar case reports: [Image findings in historical case report 1: A round nodule of approximately 1.2x1.0cm in size, with rough edges and uniform density, is seen in the right upper lobe.; Image findings in historical case report 2: A small amount of fluid density is seen in the left pleural cavity, and no obvious enlarged lymph nodes are seen in the hilum.], please list all objective abnormal findings in a strict JSON format. The JSON format example is as follows: {"findings": [{"type": "nodule", "location": "right upper lobe", "size": "10x8mm", "morphology": "lobed", "density": "solid"},{"type": "pleural effusion", "location": "left", "amount": This constructed medical image observation prompt will serve as the input of step S42 to guide the observation language model to generate structured and objective visual evidence JSON.

[0041] Accordingly, although the steps in S41 construct explicit medical image observation prompts, transforming these prompts into structured and accurate visual evidence JSON still requires an intelligent entity with strong language understanding and generation capabilities. As pointed out in the background art, existing methods may have problems such as logical incoherence, inaccurate information, or lack of clinical relevance when generating reports. The observation large language model, as the core processing unit, can deeply understand the semantic meaning of the image features and historical case references contained in the prompts, and precisely identify, extract, and organize the objective abnormal findings in the image according to the preset JSON format requirements. This ensures that the output visual evidence is not only accurate in content but also standardized in format, providing a high-quality and parsable intermediate representation for the subsequent generation of a complete report by the large language model, effectively solving the problems of inaccurate report information and insufficient structuring mentioned in the background art.

[0042] For step S42, one possible implementation is as follows: It is worth mentioning that the observation large language model adopts a Transformer-based encoder-decoder architecture, and its weight and bias parameters are obtained through supervised learning on a large dataset of medical image-report pairs. During the training process, the model learns how to map image features and related text prompts to the corresponding structured visual discovery JSON.

[0043] First, the encoding stage: The medical image observation prompts constructed in step S41 are used as the input sequence and fed into the encoder of the observation large language model. The encoder is stacked by multiple layers of Transformer encoder blocks, and each block contains a multi-head self-attention mechanism and a feed-forward neural network. Each token in the input prompt, including the textual representation of the image features, the tokens of the historical case text, and the tokens of the JSON format instructions, will be converted into a token embedding. These token embeddings, together with the positional encoding, are input into the encoder. The encoder captures the complex dependencies and context information between each token in the input sequence through the self-attention mechanism, thereby encoding the entire medical image observation prompt into a high-dimensional and semantically rich feature vector. This feature vector synthesizes the visual information of the image, the expertise of the historical cases, and the requirements for the output format.

[0044] Secondly, the decoding stage: The feature vectors output by the encoder are passed to the decoder of the observational large language model. The decoder is also stacked by multiple layers of Transformer decoder blocks, each block containing a masked multi-head self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder generates the JSON text word by word in an autoregressive manner. When generating each word, the decoder simultaneously attends to the feature vectors output by the encoder (through the cross-attention mechanism) and the sequence of words that have already been generated (through the masked self-attention mechanism). The decoder predicts the next most likely word based on the context information provided by the encoder and combines the medical knowledge and JSON grammar rules it has learned during training. For example, when the model needs to generate the type field of a lesion, it predicts word tokens such as nodule or mass based on the imaging features and hints from historical cases.

[0045] Finally, the decoder continues to generate word tokens until it generates a special token indicating the end of the sequence. The generated sequence of word tokens is combined to form the final visual evidence JSON. This visual evidence JSON is structured data in text form, precisely listing all the objective abnormal findings observed from the image, such as: {"findings": [{"type": "nodule", "location": "upper right lobe of the lung", "size": "10x8mm", "morphology": "lobulated", "density": "solid"}, {"type": "pleural effusion", "location": "left side", "amount": "small amount"}]}.

[0046] Specifically, here, the input of the observational large model is a medical image observation prompt constructed from the image embedding vector and the list containing K historical case report texts, that is, a precise structural feature constructed from multi-modal data, and the output is a visual evidence JSON with a strict format. Therefore, since both the input and output of the observational large model are strictly structured / formatted data, its fixed attribute mapping performance under the ordered deterministic specification becomes an optimization focus. That is, after obtaining the feature vectors through the encoder as described above, the encoded feature vectors are further optimized.

[0047] Based on this, further, when the medical image observation prompt is input into the observational large language model for processing, fixed attribute optimization is performed on the intermediate generated encoded feature vectors to obtain optimized encoded feature vectors, including: performing self-sequence scale deterministic compression based on eigenvalues on the intermediate generated encoded feature vectors to obtain scale-graded compression feature vectors, that is: ; where are the respective eigenvalues in the intermediate generated encoded feature vectors, and are respectively the minimum eigenvalue and the maximum eigenvalue in the intermediate generated encoded feature vector, are the eigenvalues in the scale-division compressed feature vector. Here, , when , take , which can compress the feature distribution of the multimodal input based on the self-sequence scale division, so as to improve the certainty of the feature distribution at the expense of flexibility and adaptability.

[0048] Determine the uncertainty neighborhood of each eigenvalue in the scale-division compressed feature vector, and perform regional uncertainty constraint processing on the corresponding eigenvalue based on the uncertainty neighborhood of each eigenvalue to obtain a region-constrained feature vector, that is, perform further regional uncertainty constraint processing on the features after deterministic compression, that is, for each , determine its uncertainty neighborhood , within this uncertainty neighborhood , satisfy , where is the threshold for constructing the uncertainty region, that is: ; where, is corresponding uncertainty neighborhood, is the threshold for constructing the uncertainty region, and are both obtained by empirical setting or optimized by cross-validation on the validation set. For example, is 0.1, are the eigenvalues in the region-constrained feature vector, that is, for the neighborhood uncertainty of the encoded features, perform eigenvalue neighborhood region division to output a high-uncertainty risk region, and perform uncertainty region constraint premise evaluation relative to a single eigenvalue, that is, apply the passive constraint evaluation of the uncertainty neighborhood to the eigenvalue, so as to improve the coverage of the continuous distribution details of the features in the uncertainty region.

[0049] Perform overall continuous attribute scatter clustering inference on each eigenvalue in the region-constrained feature vector to obtain the optimized encoded feature vector, that is: ; where, and are respectively 's mean and standard deviation, is the value of the exponential function with the natural constant e as the base, It is to optimize each eigenvalue in the encoded feature vector. That is, for fixed attribute mapping, it is necessary to perform accessibility imbalance compensation between mapping error and coverage loss. That is, while improving the feature distribution coverage in the uncertain region, further improve the global coverage integrity through accessibility compensation clustering. In this way, the mapping performance of the observation large model from the ordered deterministic input specification based on multi-modal precise structure data to the output of strict format attributes is improved. Then decode it to improve the representation ability of the visual evidence JSON.

[0050] In step S5, the visual evidence JSON, the clinical background text, and the list containing K historical case report texts are input into the writing large language model to obtain a medical imaging report. That is, by jointly inputting the structured visual evidence JSON obtained in step S4, the clinical background text obtained in step S1, and the list of historical case reports retrieved in step S3 into the writing large language model, the deep integration of multi-source information can be achieved. The writing large language model can synthesize this information to generate a complete report containing imaging findings and diagnostic opinions, ensuring the accuracy, logical coherence, and clinical relevance of the report, thus making up for the deficiency of the traditional large language model in lacking the integration of vision and professional knowledge and solving the report quality problem mentioned in the background technology.

[0051] Specifically, in an exemplary embodiment of the present application, step S5 of inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into the writing large language model to obtain a medical imaging report includes: S51, constructing a final report generation prompt based on the visual evidence JSON, the clinical background text, and the list containing K historical case report texts; S52, inputting the final report generation prompt into the writing large language model to obtain the medical imaging report.

[0052] More specifically, in an exemplary embodiment of the present application, the final report generation prompt is "Please act as a senior radiologist and generate a complete medical imaging report containing [imaging findings] and [diagnostic opinions] based on the following [clinical background] and [list of imaging findings], and referring to [similar case reports]".

[0053] For step S5, one possible implementation is as follows: First, perform S51. If the obtained visual evidence JSON is {"findings": [{"type": "nodule", "location": "right upper lobe of the lung", "size": "10x8mm", "morphology": "lobulated", "density": "solid"},{"type": "pleural effusion", "location": "left side", "amount": "small amount"}]}. Clinical background text, for example, the patient is a 50-year-old male who was admitted to the hospital due to cough and expectoration for 2 weeks and has a history of smoking. A list containing K historical case report texts, for example, 5 historical medical report texts most similar to the current case).

[0054] First, parse and textify the visual evidence JSON. The visual evidence JSON is a structured data that needs to be converted into natural language text that the writing large language model can understand. This is achieved by traversing the findings array in the JSON and combining the attributes of each abnormal finding, such as type, location, size, morphology, density, etc. into one or more descriptive texts. For example, the first finding in the JSON {"type": "nodule", "location": "right upper lobe of the lung", "size": "10x8mm", "morphology": "lobulated", "density": "solid"} can be converted into "A lobulated solid nodule of 10x8mm is visible in the right upper lobe of the lung". The second finding {"type": "pleural effusion", "location": "left side", "amount": "small amount"} can be converted into "A small amount of fluid is visible in the left pleural cavity". All the converted descriptive texts can be combined into a string of imaging findings list.

[0055] Second, integrate the clinical background text. The clinical background text is the key information such as the patient's medical history and symptoms, which is crucial for writing the diagnosis opinion. The clinical background text can be directly used as part of the prompt without additional processing. For example, directly use "The patient is a 50-year-old male who was admitted to the hospital due to cough and expectoration for 2 weeks and has a history of smoking." as the clinical background part in the prompt.

[0056] Third, integrate the historical case report texts. The list of K historical case report texts provides rich professional knowledge and report examples. These reports can be used as references when the writing large language model generates reports. To avoid the prompt being too long, the historical reports can be appropriately screened or summarized. For example, only include the imaging findings and diagnosis opinion parts of each historical report, or splice all the historical reports together. These historical reports will be used as the similar case report part in the prompt.

[0057] Finally, fill it into a preset prompt template. According to an exemplary embodiment of the present application, the template for generating the final report prompt is: "Please act as a senior radiologist. Based on the following [clinical background] and [list of imaging findings], and referring to [similar case reports], generate a complete medical imaging report including [imaging findings] and [diagnostic opinions]". Precisely fill the processed list of imaging findings, clinical background, and content of similar case reports into the corresponding placeholders of this template.

[0058] Finally, perform S52. Input the final report generation prompt constructed in step S51 into the pre-trained writing large language model. The writing large language model is a general or fine-tuned large language model in the medical field, and its architecture is based on the decoder-only structure of Transformer (such as the GPT series) or the encoder-decoder structure. Its weight and bias parameters are obtained through unsupervised pre-training and supervised fine-tuning on a large medical report dataset. During the training process, the model learns the language patterns, logical structures, professional terms of medical reports, and how to generate coherent and accurate report content based on the input information. The specific processing process is that after receiving the final report generation prompt, the writing large language model first encodes the prompt and converts it into a numerical representation inside the model. Then, the model uses its powerful language generation ability to generate a medical imaging report word by word or token by token in an autoregressive manner. During the generation process, the model will comprehensively consider all the information in the prompt: it will understand the overall situation of the patient based on the clinical background; accurately describe the objective findings in the image according to the list of imaging findings and organize them into the imaging findings part of the report; at the same time, the model will deeply refer to the diagnostic ideas and expressions in similar case reports, combine its own medical knowledge, reason and judge the imaging findings, and finally generate a diagnostically significant diagnostic opinion part. For example, the model may generate a diagnostic opinion considering pulmonary malignancy and suggesting further examination based on a nodule in the upper lobe of the right lung, a smoking history, and the diagnosis in similar cases. Finally, the writing large language model outputs a complete medical imaging report text containing imaging findings and diagnostic opinions. For example, a generated medical imaging report may be as follows: Medical Imaging Report: Patient Information: Patient, male, 50 years old. Clinical Background: Admitted to the hospital due to cough and expectoration for 2 weeks, with a history of smoking. Imaging Findings: Chest CT plain scan shows: A round solid nodule shadow can be seen in the upper lobe of the right lung, about 10x8 mm in size, with a lobulated edge and uniform density. A small amount of liquid density shadow can be seen in the left thoracic cavity, and no obvious pleural thickening is seen. No obvious enlarged lymph nodes are seen in the bilateral hilar and mediastinum. Diagnostic Opinions: 1. Nodule in the upper lobe of the right lung, highly suspected of malignant lesion, it is recommended to further perform enhanced CT examination or PET-CT examination, and perform puncture biopsy if necessary to clarify the diagnosis. 2. A small amount of left pleural effusion, it is recommended to combine clinically. This report will be used as the final output of this method to assist and optimize doctors and provide comprehensive diagnostic basis for doctors.

[0059] In summary, the method for generating medical image reports based on large language models according to the embodiments of the present application is elucidated. It first obtains the original image data and clinical background text of a patient, and extracts image embedding vectors and background embedding vectors respectively, realizing the preliminary digital representation of multi-modal information and solving the problem that traditional LLMs are difficult to effectively fuse visual information. Then, using these embedding vectors for case retrieval based on prior knowledge, K highly relevant historical case reports are selected from a large amount of historical data, providing rich professional references for subsequent report generation and making up for the deficiency of existing methods in lacking the utilization of historical experience. Subsequently, the image embedding vectors and historical case reports are input into the observing large language model, enabling it to focus on extracting objective visual evidence from the images and outputting in a structured JSON format, ensuring the objectivity and accuracy of the reports. Finally, the writing large language model integrates the visual evidence JSON, clinical background text, and historical case reports to generate the final medical image report. Through a phased and multi-modal fusion approach, it effectively solves problems such as incoherent report logic and inaccurate information, significantly improving the quality and generation efficiency of the reports.

[0060] Figure 5 FIG. is a block diagram of a medical image report generation system based on large language models according to an embodiment of the present application. As Figure 5 shown, the medical image report generation system 100 based on large language models according to an embodiment of the present application includes: an image data background text acquisition module 110 for acquiring the original image data and clinical background text of a target patient object; an image data background text acquisition module 120 for extracting visual features from the original image data to obtain image embedding vectors, and extracting text semantic features from the clinical background text to obtain background embedding vectors; an image data background text acquisition module 130 for performing case retrieval based on prior knowledge based on the background embedding vectors and the image embedding vectors to obtain a list containing K historical case report texts; an image data background text acquisition module 140 for inputting the image embedding vectors and the list containing K historical case report texts into an observing large language model to obtain visual evidence JSON; an image data background text acquisition module 150 for inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

[0061] Here, those skilled in the art can understand that the specific operations of each step in the above medical image report generation system based on large language models have been introduced in detail in the description of the method for generating medical image reports based on large language models above, and therefore, the repeated description thereof will be omitted. Figures 1 to 4 of the method for generating medical image reports based on large language models above, and thus, the repeated description thereof will be omitted.

Claims

1. A method for generating medical image reports based on large language models, characterized in that, Including: Obtaining the original image data and clinical background text of a target patient object; Extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; performing prior-knowledge-based case retrieval based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

2. The method for generating a medical image report based on a large language model according to claim 1, wherein Extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector, including: preprocessing the original image data to obtain preprocessed original image data; inputting the preprocessed original image data into a pre-trained visual encoder to obtain the image embedding vector, where the pre-trained visual encoder is a MedNeXt model; inputting the clinical background text into a text editor to obtain the background embedding vector, and the text editor is a BERT model.

3. The method for generating a medical image report based on a large language model according to claim 2, wherein, Preprocessing the original image data to obtain preprocessed original image data, including: performing window width and window level adjustment, normalization, and size adjustment on the original image data to obtain the preprocessed original image data.

4. The method for generating a medical image report based on a large language model according to claim 1, wherein, Performing prior-knowledge-based case retrieval based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts, including: concatenating the background embedding vector and the image embedding vector to obtain a query embedding vector; using the query embedding vector as a query to perform similarity retrieval in an image-report vector database to obtain the list containing K historical case report texts.

5. The method for generating a medical image report based on a large language model according to claim 4, wherein, Using the query embedding vector as a query to perform similarity retrieval in an image-report vector database to obtain the list containing K historical case report texts, including: calculating the cosine similarity between the query embedding vector and each case embedding vector in the image-report vector database as a case matching degree to obtain a set of case matching degrees; extracting K historical case report texts from the image-report vector database based on the sorting of the set of case matching degrees.

6. The method for generating a medical image report based on a large language model according to claim 1, wherein Inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON, including: constructing a medical image observation prompt based on the image embedding vector and the list containing K historical case report texts, where the medical image observation prompt includes listing all objective abnormal findings in a strict JSON format; inputting the medical image observation prompt into the observation large language model to obtain the visual evidence JSON.

7. The method for generating a medical image report based on a large language model according to claim 6, wherein When processing the medical imaging observation prompt by inputting it into the observation large language model, perform fixed attribute optimization on the intermediate generated encoded feature vector to obtain an optimized encoded feature vector, including: performing self-sequence scale deterministic compression based on eigenvalues on the intermediate generated encoded feature vector to obtain a scale-division compressed feature vector; determining the uncertainty neighborhood of each eigenvalue in the scale-division compressed feature vector, and performing regional uncertainty constraint processing on the corresponding eigenvalue based on the uncertainty neighborhood of each eigenvalue to obtain a region-constrained feature vector; performing overall continuous attribute scatter clustering inference on each eigenvalue in the region-constrained feature vector to obtain the optimized encoded feature vector.

8. The method for generating a medical image report based on a large language model according to claim 1, wherein Input the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into the writing large language model to obtain a medical imaging report, including: constructing a final report generation prompt based on the visual evidence JSON, the clinical background text, and the list containing K historical case report texts; inputting the final report generation prompt into the writing large language model to obtain the medical imaging report.

9. The method for generating a medical image report based on a large language model according to claim 8, wherein The final report generation prompt is "Please act as a senior radiologist and generate a complete medical imaging report containing [imaging findings] and [diagnostic opinions] based on the following [clinical background] and [list of imaging findings], and referring to [similar case reports]".

10. A medical image report generation system based on a large language model, characterized in that, Including: An imaging data background text acquisition module for acquiring the original imaging data and clinical background text of a target patient object; An imaging data background text acquisition module for extracting visual features from the original imaging data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; an imaging data background text acquisition module for performing prior knowledge-based case retrieval based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; an imaging data background text acquisition module for inputting the image embedding vector and the list containing K historical case report texts into the observation large language model to obtain visual evidence JSON; an imaging data background text acquisition module for inputting the visual evidence JSON, the clinical background text, and the list containing K historical case report texts into the writing large language model to obtain a medical imaging report.

Images