Medical imaging report generation method and system based on large language model

Abstract

The present application discloses a method and system for generating medical imaging reports based on a large language model, which relates to the field of imaging report generation. It first obtains the patient's original imaging data and clinical background text, and extracts the image embedding vector and background embedding vector respectively. Then, these embedding vectors are used to perform case retrieval based on prior knowledge, and K highly relevant historical case reports are screened out from massive historical data. Subsequently, the image embedding vectors and historical case reports are input into the observation large language model and output in a structured JSON format. Finally, a large language model is written to integrate visual evidence JSON, clinical background text and historical case reports to generate the final medical imaging report. This method effectively solves problems such as incoherent report logic and inaccurate information through a phased and multimodal fusion approach, significantly improving the quality and generation efficiency of the report.

Description

Technical Field

[0001] The present application relates to the field of image report generation, and more specifically, to a method and system for generating medical image reports based on a large language model. Background Art

[0002] With the rapid advancement of medical technology, medical imaging plays an increasingly important role in disease diagnosis and treatment. However, the writing of medical imaging reports is often done manually by radiologists, which is not only time-consuming and labor-intensive, but also subject to variations in report quality and consistency based on physician experience and subjective judgment. Doctors often face tremendous workload pressure when faced with massive amounts of imaging data, making the need for automated, intelligent generation of high-quality medical imaging reports increasingly urgent.

[0003] Although some studies have attempted to utilize artificial intelligence technologies to assist in the generation of medical imaging reports, such as methods based on template matching or rule engines, these methods often lack flexibility and struggle to handle complex clinical situations and diverse imaging manifestations. In recent years, large language models (LLMs) have achieved breakthroughs in the field of natural language processing, demonstrating powerful text generation and comprehension capabilities, providing new approaches for the automated generation of medical imaging reports. However, the direct application of large language models to medical imaging report generation remains challenging. Traditional LLMs often struggle to effectively integrate visual information when processing medical imaging data, and lack a deep understanding of medical expertise. This can lead to reports that are logically incoherent, contain inaccurate information, or lack clinical relevance. Furthermore, existing methods often fail to fully leverage the valuable experience of historical cases when generating reports, leaving room for improvement in the accuracy and reliability of the reports.

[0004] Therefore, there is an urgent need for a medical imaging report generation method and system that can effectively integrate medical imaging data, clinical background information and historical case experience. Summary of the Invention

[0005] In view of the above-mentioned limitations of the existing methods, according to one aspect of the present application, a method for generating a medical imaging report based on a large language model is provided, which includes: obtaining original imaging data and clinical background text of a target patient object; extracting visual features from the original imaging data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; based on the background embedding vector and the image embedding vector, performing case retrieval based on prior knowledge to obtain a list containing K historical case report texts; inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; inputting the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical imaging report.

[0006] According to another aspect of the present application, a medical imaging report generation system based on a large language model is provided, which includes: an image data background text acquisition module for acquiring original imaging data and clinical background text of a target patient object; an image data background text acquisition module for extracting visual features from the original imaging data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; an image data background text acquisition module for performing case retrieval based on prior knowledge based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; an image data background text acquisition module for inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; an image data background text acquisition module for inputting the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical imaging report.

[0007] Compared with the existing technology, the present application provides a method and system for generating medical imaging reports based on a large language model. It first obtains the patient's original imaging data and clinical background text, and extracts image embedding vectors and background embedding vectors respectively, realizing a preliminary digital representation of multimodal information, solving the problem that traditional LLMs are difficult to effectively integrate visual information. Then, these embedding vectors are used to perform case retrieval based on prior knowledge, and K highly relevant historical case reports are screened from massive historical data, providing rich professional references for subsequent report generation, making up for the lack of historical experience utilization in existing methods. Subsequently, the image embedding vectors and historical case reports are input into the observation large language model, so that it focuses on extracting objective visual evidence from the images and outputs it in a structured JSON format, ensuring the objectivity and accuracy of the report. Finally, the large language model is written to integrate the visual evidence JSON, clinical background text and historical case reports to generate the final medical imaging report. Through a phased, multimodal fusion approach, it effectively solves problems such as incoherent report logic and inaccurate information, significantly improving the quality and generation efficiency of the report. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 The present invention provides a flowchart of a method for generating a medical imaging report based on a large language model according to an embodiment of the present application.

[0010] Figure 2 Schematic diagram of data flow of a method for generating a medical imaging report based on a large language model according to an embodiment of the present application.

[0011] Figure 3 This is a flowchart of step S2 in the method for generating a medical imaging report based on a large language model according to an embodiment of the present application.

[0012] Figure 4 This is a flowchart of step S3 in the method for generating a medical imaging report based on a large language model according to an embodiment of the present application.

[0013] Figure 5 This is a block diagram of a medical imaging report generation system based on a large language model according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] In response to the problems in the above-mentioned background technology, this application proposes a method for generating medical imaging reports based on a large language model. Figure 1 The present invention provides a flowchart of a method for generating a medical imaging report based on a large language model according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the method for generating medical image reports based on a large language model according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, a medical imaging report generation method based on a large language model includes: S1, obtaining original imaging data and clinical background text of a target patient object; S2, extracting visual features from the original imaging data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector; S3, based on the background embedding vector and the image embedding vector, performing a case retrieval based on prior knowledge to obtain a list containing K historical case report texts; S4, inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; S5, inputting the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical imaging report.

[0016] In step S1, the original imaging data and clinical background text of the target patient are obtained. It should be understood that the generation of medical imaging reports is essentially a process of multimodal information fusion and understanding. The original imaging data carries the visual characteristics of the patient's lesions and is the direct basis for diagnosis; while the clinical background text provides non-imaging information such as the patient's medical history, symptoms, and examination results, which plays an indispensable role in fully understanding the condition and assisting diagnosis. Traditional medical imaging report generation methods often find it difficult to effectively integrate these heterogeneous data, resulting in one-sided or inaccurate reports. Therefore, obtaining these two key information is the cornerstone of subsequent steps for feature extraction, case retrieval, and final report generation, ensuring that the model can obtain complete information about the patient from multiple dimensions, thereby generating more comprehensive and accurate medical imaging reports.

[0017] One possible implementation of step S1 is as follows: First, the original image data of the target patient can be obtained through various channels. For example, it can be directly retrieved from the hospital's Picture Archiving and Communication System (PACS). This data is stored in the DICOM (Digital Imaging and Communication in Medicine) format and includes the patient's original scan images, such as CT, MRI, and X-rays. In practice, when a doctor or other relevant personnel needs to generate a medical imaging report for a specific patient, they select or enter the patient ID through the user interface. The system then retrieves and downloads the corresponding original image data from the PACS server. For example, for a patient undergoing a lung CT scan, the system will obtain multiple DICOM image sequences in axial, coronal, and sagittal planes.

[0018] The clinical background text for the target patient is obtained by extracting it from the hospital's electronic medical record system (EHR). These text data may include the patient's chief complaint, current medical history, past medical history, family history, allergy history, physical examination results, laboratory test results, medication status, and other relevant clinical records. During the implementation process, the system will query and extract the clinical background text related to this imaging examination from the EHR database based on the patient ID or medical record. For example, for the above-mentioned lung CT patient, the system will obtain clinical information such as his chief complaint of coughing and sputum for 3 weeks, accompanied by fever, and a history of hypertension. These text data exist in unstructured or semi-structured text formats, such as plain text, XML, or JSON format.

[0019] In step S2, visual features are extracted from the raw image data to obtain an image embedding vector, and text semantic features are extracted from the clinical background text to obtain a background embedding vector. It is understandable that the original medical image data (such as DICOM images) and clinical background text (such as medical history records) are heterogeneous, high-dimensional raw information that cannot be effectively processed directly by a large language model. In order for these multimodal information to be understood and utilized by subsequent case retrieval and report generation modules, they need to be converted into a unified, low-dimensional, numerical representation with semantic information, namely an embedding vector. To this end, in the technical solution of the present application, by extracting visual features and text semantic features to map complex raw data into a continuous vector space, the lesion information in the image and the clinical semantics in the text are captured, providing standardized input for subsequent similarity calculation and model reasoning. This solves the problem mentioned in the background technology that traditional LLMs are difficult to effectively integrate visual information and lack a deep understanding of medical expertise, and is a key link in achieving multimodal information fusion and intelligent report generation.

[0020] Specifically, in an exemplary embodiment of the present application, Figure 3FIG is a flowchart of step S2 in the method for generating a medical imaging report based on a large language model according to an embodiment of the present application. Figure 3 As shown, step S2, extracting visual features from the original image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector, includes: S21, preprocessing the original image data to obtain preprocessed original image data; S22, inputting the preprocessed original image data into a pretrained visual encoder to obtain the image embedding vector, wherein the pretrained visual encoder is a MedNeXt model; S23, inputting the clinical background text into a text editor to obtain the background embedding vector, and the text editor is a BERT model.

[0021] The MedNeXt model's uniqueness as a visual encoder lies in its design specifically for medical image analysis, particularly for three-dimensional volumetric data (such as CT and MRI). Unlike general-purpose two-dimensional image models, MedNeXt inherits the superior performance of modern convolutional networks (ConvNeXt) and is optimized for the specific characteristics of medical imaging, effectively capturing the morphology, structure, and adjacency of lesions in three-dimensional space. This ensures that the image embedding vectors extracted from the raw image data are highly concentrated and accurately represent complex pathological features, providing a high-quality visual evidence foundation for subsequent similar case retrieval and report generation.

[0022] The BERT model was chosen to process clinical context text because, as a Transformer-based bidirectional encoder, BERT excels at understanding the deep contextual semantics of natural language. Clinical context text contains a large number of key medical terms and patient descriptions with complex logical relationships. BERT, through pre-training using its unique masked language model and next sentence prediction task, is able to capture the subtle differences in the meaning of words in different contexts, thereby converting clinical context text into a context embedding vector that is rich in semantic information and accurately reflects the patient's medical history and current condition. This high-quality text representation is crucial for subsequent large language models to link imaging findings with clinical context and generate clinically relevant diagnostic opinions.

[0023] One possible implementation of step S2 is as follows: First, perform S21. After acquiring the raw image data of the target patient (e.g., a CT image sequence in DICOM format), a series of preprocessing operations are first performed to eliminate noise, standardize the data format, and optimize image quality to make it more suitable for input into the deep learning model. Specifically, in an exemplary embodiment of the present application, preprocessing the raw image data to obtain preprocessed raw image data includes: adjusting the window width and window level, normalizing, and resizing the raw image data to obtain the preprocessed raw image data. Specifically, window width and window level adjustment refers to adjusting the grayscale value range of the image to highlight the tissue structure or lesion of interest based on different medical imaging modalities and observation purposes. For example, for lung CT images, the window width is set to 1500 HU (Hounsfield units) and the window level is set to -500 HU to clearly display the lung parenchyma and tracheobronchial structures; while for bone windows, the window width may be set to 2000 HU and the window level to 300 HU to observe skeletal details. These window width and window position values ​​are preset based on clinical experience and imaging standards. Normalization is to scale the pixel values ​​of the image to a standard range, such as [0,1] or [-1,1], to eliminate the grayscale differences caused by different scanning devices or scanning parameters and ensure the stability of model training. This is achieved through linear transformation, that is, (pixel value-minimum value) / (maximum value-minimum value). Resizing is to uniformly adjust all images to a preset fixed size to meet the input requirements of the visual encoder model. For example, a three-dimensional CT image can be adjusted to a voxel size of 256x256x256 to ensure the consistency of the input dimension. After the above preprocessing, the original image data is converted into preprocessed original image data.

[0024] Next, proceed to S22. The preprocessed raw image data obtained in step S21 is fed as input to the pretrained MedNeXt model. MedNeXt is a deep convolutional neural network specifically designed for medical image analysis. Its architecture comprises multiple 3D convolutional layers, batch normalization layers, activation functions (such as ReLU), and pooling layers. By stacking these modules, it gradually extracts hierarchical image features. It utilizes 3D convolutional kernels to capture the three-dimensional spatial information of medical images and mitigates the vanishing gradient problem in deep network training through mechanisms such as residual connections. The MedNeXt model is pretrained on a large number of medical imaging datasets and learns a rich representation of medical image features. Its weights and bias parameters are obtained through supervised learning on large-scale medical image classification, segmentation, or reconstruction tasks, aiming to minimize prediction error. Once the preprocessed raw image data is fed into the MedNeXt model, it undergoes a series of convolutions, pooling, and nonlinear transformations, ultimately outputting a fixed-dimensional vector, the image embedding vector, before the final global average pooling layer or fully connected layer.

[0025] Then, proceed to S23. The clinical background text is used as input and sent to the pre-trained BERT model. Those skilled in the art should know that BERT is a bidirectional encoder based on the Transformer architecture, the core of which is a multi-head self-attention mechanism that can capture the complex dependencies and contextual information between words in the text. The BERT model has been unsupervised pre-trained on a large amount of general text corpus and has learned rich language representations. Its weights and bias parameters are obtained through extensive training. When the clinical background text is input into the BERT model, the text will first be segmented and converted into word embeddings, and then processed by a multi-layer Transformer encoder, and finally a fixed-dimensional vector, namely the background embedding vector, is obtained at the output position corresponding to the [CLS] tag of the model.

[0026] In step S3, based on the background embedding vector and the image embedding vector, a case retrieval based on prior knowledge is performed to obtain a list containing K historical case report texts. It should be understood that the background technology points out that the existing methods have failed to fully utilize the valuable experience of historical cases, resulting in the accuracy and reliability of the reports need to be improved. Based on this, the present application introduces case retrieval based on prior knowledge, which can intelligently find cases that are highly similar to the current patient's imaging features and clinical background from a huge historical image-report database. These similar historical case reports contain the diagnostic ideas, descriptions of imaging findings and diagnostic opinions of senior doctors, providing rich, high-quality, clinically relevant contextual information and professional guidance for the subsequent large language model to generate reports. This can not only significantly improve the professionalism and accuracy of the generated reports, but also effectively make up for the lack of deep understanding of medical expertise in the large language model, thereby solving the report accuracy and reliability issues mentioned in the background technology.

[0027] Specifically, in an exemplary embodiment of the present application, Figure 4 FIG. 1 is a flowchart of step S3 in the method for generating a medical imaging report based on a large language model according to an embodiment of the present application. Figure 4 As shown, step S3, based on the background embedding vector and the image embedding vector, performs case retrieval based on prior knowledge to obtain a list containing K historical case report texts, including: S31, feature splicing of the background embedding vector and the image embedding vector to obtain a query embedding vector; S32, using the query embedding vector as a query, performing similarity retrieval in the image-report vector database to obtain the list containing K historical case report texts.

[0028] It is worth mentioning that the background technology points out that the writing of medical imaging reports needs to consider both imaging information and clinical background. Using image embedding vectors or background embedding vectors alone for retrieval cannot fully capture the complete characteristics of the case, which may cause the retrieved historical cases to be inaccurate or insufficiently relevant. Therefore, in order to create a richer and more comprehensive query embedding vector, this application combines the semantic information of visual and textual dual modalities by feature splicing of these two heterogeneous but complementary embedding vectors. This fusion enables subsequent similarity retrieval to more accurately match historical cases that are highly similar to the current case in both imaging manifestations and clinical background, thereby providing more accurate prior knowledge for the large language model, effectively solving the problem of lack of effective prior knowledge and case references mentioned in the background technology.

[0029] One possible implementation of step S31 is as follows: It should be understood that feature concatenation is a simple and effective feature fusion method that directly connects two or more vectors along a certain dimension to form a longer new vector that contains all the original feature information. In specific implementations, the image embedding vector and the background embedding vector are concatenated. For example, the dimension of the image embedding vector V1 is D1, and the dimension of the background embedding vector V2 is D2. The concatenation operation will generate a query embedding vector Vq with a dimension of D1+D2.

[0030] Accordingly, the writing of medical imaging reports is highly dependent on the professional experience of radiologists and reference to previous similar cases. The background technology points out that the existing methods have failed to make full use of the valuable experience of historical cases, resulting in the accuracy and reliability of the reports to be improved. By embedding the query vector obtained in step S31 that integrates the image and clinical background information, and performing efficient similarity retrieval in the pre-built image-report vector database, it is possible to intelligently find historical cases that are highly similar to the current case in multimodal features. These similar historical case reports contain the diagnostic ideas, descriptions of imaging findings and diagnostic opinions of senior doctors, providing rich, high-quality, clinically relevant contextual information and professional guidance for the subsequent large language model to generate reports. This can not only significantly improve the professionalism and accuracy of the generated reports, but also effectively make up for the lack of deep understanding of medical expertise in the large language model, thereby solving the report accuracy and reliability issues mentioned in the background technology.

[0031] More specifically, in an exemplary embodiment of the present application, step S32, using the query embedding vector as a query, performs a similarity search in the image-report vector database to obtain the list containing K historical case report texts, including: S321, calculating the cosine similarity between the query embedding vector and each case embedding vector in the image-report vector database as the case matching degree to obtain a set of case matching degrees; S322, extracting K historical case report texts from the image-report vector database based on the sorting of the set of case matching degrees.

[0032] One possible implementation of step S32 is as follows: First, proceed to S321. The image-report vector database is a structured data set that stores a large number of historical cases. Each entry in the database corresponds to a historical case and includes the case embedding vector (constructed by converting the historical images and historical clinical background text into embedding vectors and then concatenating them using a similar method to S2 and S31) and the corresponding original historical case report text.

[0033] For each historical case i in the database, its corresponding case embedding vector is recorded as Query embedding vector Embedding vectors of all cases in the database Cosine similarity between them. Cosine similarity is a measure of the cosine value of the angle between two non-zero vectors. Its value is between -1 and 1. The closer the value is to 1, the closer the directions of the two vectors are, that is, the higher the similarity. The calculation formula of cosine similarity is: ;in, express and The vector dot product of Represents the Euclidean norm of a vector. By traversing all historical cases in the image-report vector database, we can calculate With each The cosine similarity of is calculated, resulting in a set containing the matching scores of all cases. For example, if there are 1 million historical cases in the database, 1 million cosine similarity values ​​will be obtained.

[0034] Finally, proceed to S322. After obtaining the cosine similarity of all historical cases, that is, the case matching degree set, this step will sort these matching degrees in descending order. After the sorting is completed, the K historical cases with the highest similarity are selected. It is worth noting that K is a preset integer value representing the number of historical cases to be retrieved. The setting of K requires a balance between retrieval efficiency and report quality. If the K value is too small, it may not provide sufficiently rich reference information; if the K value is too large, it will increase the processing burden of the subsequent large language model and may introduce irrelevant noise information. In practical applications, the K value can be set based on experience. For example, K can be set to 5 or 10, that is, the 5 or 10 most similar historical cases are retrieved. From the sorting results, the historical case report texts corresponding to the K highest similarities are extracted. These texts are original, complete medical imaging reports, containing detailed content such as imaging findings and diagnostic opinions. For example, if K is set to 5, the 5 historical medical imaging report texts that are most similar to the current patient's images and clinical background will be extracted.

[0035] In step S4, the image embedding vector and the list containing K historical case report texts are input into the observation large language model to obtain visual evidence JSON. It is understandable that the existing methods do not have a deep enough understanding and utilization of image information, and the generated reports may lack sufficient details and accuracy. Directly inputting the original image data into the large language model for end-to-end generation often makes it difficult to ensure the objectivity and structure of the output. Based on this, the present application introduces the observation large language model, and combines the image embedding vector and historical case reports to specifically train the model to focus on extracting key, objective lesion features from visual information, and output it in a strict JSON format, thereby ensuring the accuracy, structure and parsability of the visual evidence. This staged processing method decouples image analysis from text generation, allowing the model to more effectively focus on the identification and description of image features, providing high-quality, standardized input for subsequent report writing.

[0036] Specifically, in an exemplary embodiment of the present application, step S4, inputting the image embedding vector and the list containing K historical case report texts into the observation large language model to obtain visual evidence JSON, includes: S41, constructing a medical image observation prompt based on the image embedding vector and the list containing K historical case report texts, wherein the medical image observation prompt includes listing all objective abnormal findings in a strict JSON format; S42, inputting the medical image observation prompt into the observation large language model to obtain the visual evidence JSON.

[0037] It should be understandable that, as mentioned in the background technology, existing methods do not have a deep enough understanding of imaging information, and the structuring of reports needs to be improved. Directly inputting the original image embedding vector into the model may make it difficult for the model to understand its specific meaning and generate a structured description. To this end, by constructing clear medical image observation prompts, converting abstract image embedding vectors into text instructions that the model can understand, and explicitly requiring the output of objective abnormal findings in a strict JSON format, it can effectively guide the observation of large language models to focus on the recognition and structured description of image features, avoiding the generation of irrelevant information or subjective judgments, thereby ensuring the accuracy, objectivity and parsability of subsequent visual evidence JSON, and providing a high-quality intermediate representation for the generation of the final report.

[0038] For step S41, a possible implementation method is as follows: First, the image embedding vector is textualized or tokenized. Although the large language model can process text input, directly using the numerical image embedding vector as text input may be inefficient or semantically unclear. One implementation method is to convert the image embedding vector into a short, semantically representative text description or special tag sequence through a pre-trained lightweight encoder or mapping layer. For example, a small neural network can be trained to map a 512-dimensional image embedding vector into a descriptive text, such as an image showing abnormal signals in the lungs, which may involve nodules and pleural effusion. Alternatively, a more common method is to use the image embedding vector directly as a special visual tag in the input sequence of the large language model. During training, the model will learn how to understand and utilize the visual information represented by this tag.

[0039] Next, integrate the historical case report text. A list of K historical case report texts (e.g., five complete historical reports) is integrated into the prompt. These historical reports provide a rich set of medical terminology, imaging description paradigms, and diagnostic logic, serving as a reference for observing the objective findings generated by the large language model. To reduce the length of the prompt, the historical reports can be summarized or only the key paragraphs relevant to the imaging findings can be selected. For example, the imaging findings section of each historical report can be extracted.

[0040] Next, instructions and format requirements are constructed. The core of the prompt is to clearly inform the observational language model of the task to be completed and the output format. This is achieved through pre-set templates or instruction statements. The instructions clearly require the model to: Play a role, for example, please play the role of a senior radiologist and focus on objectively describing imaging findings. Input information specifies the inputs the model needs to refer to, including image features, namely textual or tokenized representations via image embedding vectors, and historical case references. For example, based on the following image features: [Image feature description / tag], and referencing the following similar case reports: [Text list of historical case reports]. The task objective explicitly requires the model to list all objective abnormal findings. The emphasis on objectivity is to avoid the model from making subjective diagnoses or speculations. The output format is mandatory and must be in a strict JSON format. This ensures that subsequent steps can easily parse and utilize this visual evidence. The structure of the JSON format can be pre-defined, for example, containing an array named findings, each element of which is an object describing a single abnormal finding, with keys such as type, location, size, morphology, and density. An example medical image observation prompt might be as follows: "Please play the role of a senior radiologist. Based on the following image features: [Textual representation of the image embedding vector, for example: 'The image shows a solid nodule in the right upper lobe and a small amount of fluid in the left pleural cavity.'], and referring to the following similar case reports: [Image findings in historical case report 1: A round nodule of approximately 1.2x1.0cm in size, with rough edges and uniform density, is seen in the right upper lobe.; Image findings in historical case report 2: A small amount of fluid density is seen in the left pleural cavity, and no obvious enlarged lymph nodes are seen in the hilum.], please list all objective abnormal findings in a strict JSON format. The JSON format example is as follows: {"findings": [{"type": "nodule", "location": "right upper lobe", "size": "10x8mm", "morphology": "lobed", "density": "solid"},{"type": "pleural effusion", "location": "left", "amount": This constructed medical image observation prompt will serve as the input of step S42 to guide the observation language model to generate structured and objective visual evidence JSON.

[0041] Accordingly, although step S41 constructs clear medical image observation prompts, converting these prompts into structured and accurate visual evidence JSON still requires an intelligent entity with strong language understanding and generation capabilities. The background technology points out that existing methods may have problems such as logical incoherence, inaccurate information or lack of clinical relevance when generating reports. The observation large language model serves as the core processing unit, which can deeply understand the semantics of image features and historical case references contained in the prompts, and accurately identify, extract and organize objective abnormal findings in the images according to the preset JSON format requirements. This ensures that the output visual evidence is not only accurate in content, but also in a standardized format, providing a high-quality, parsable intermediate representation for the subsequent writing of a complete report generated by the large language model, effectively solving the problems of inaccurate and insufficiently structured report information mentioned in the background technology.

[0042] One possible implementation of step S42 is as follows: It's worth noting that the large language model uses a Transformer-based encoder-decoder architecture, whose weights and bias parameters are derived through supervised learning on a large dataset of medical image-report pairs. During training, the model learns how to map image features and associated textual cues to corresponding structured visual discovery JSON.

[0043] First, in the encoding phase, the medical image observation prompts constructed in step S41 are fed as the input sequence into the encoder of the observation large language model. The encoder consists of multiple layers of stacked Transformer encoder blocks, each of which incorporates a multi-head self-attention mechanism and a feedforward neural network. Each token in the input prompt—including textual representations of image features, tokens from historical case text, and tokens from JSON-formatted instructions—is converted into a word embedding. These word embeddings, along with positional encodings, are fed into the encoder. Using a self-attention mechanism, the encoder captures the complex dependencies and contextual information between tokens in the input sequence, encoding the entire medical image observation prompt into a high-dimensional, semantically rich feature vector. This feature vector integrates the visual information of the image, the expertise from the historical case, and the required output format.

[0044] Next, in the decoding phase, the feature vector output by the encoder is passed to a decoder that observes a large language model. The decoder is also composed of multiple layers of Transformer decoder blocks, each of which incorporates a masked multi-head self-attention mechanism, a criss-cross attention mechanism, and a feedforward neural network. The decoder generates JSON text token by token in an autoregressive manner. When generating each token, the decoder simultaneously attends to the feature vector output by the encoder (via the criss-cross attention mechanism) and the sequence of previously generated tokens (via the masked self-attention mechanism). Based on the contextual information provided by the encoder and combined with the medical knowledge and JSON grammar rules learned during training, the decoder predicts the next most likely token. For example, when the model needs to generate a lesion type field, it predicts tokens such as nodule or mass based on imaging features and historical case studies.

[0045] Finally, the decoder continues generating tokens until a special marker indicating the end of the sequence is generated. The resulting token sequences are combined to form the final visual evidence JSON. This visual evidence JSON is structured data in text form, precisely listing all objective abnormal findings observed in the image, for example: {"findings": [{"type":"nodule", "location": "right upper lobe", "size": "10x8mm", "morphology": "lobulated", "density": "solid"},{"type": "pleural effusion", "location": "left", "amount": "small"}]}.

[0046] Specifically, the input to the observation model is medical image observation prompts constructed from the image embedding vectors and the list of K historical case report texts—precise structural features constructed from multimodal data—and the output is visual evidence in a strictly formatted JSON format. Therefore, since both the input and output of the observation model are strictly structured / formatted data, its fixed attribute mapping performance under ordered deterministic specifications becomes a key optimization focus. Specifically, after obtaining the feature vectors through the encoder, the encoded feature vectors are further optimized.

[0047] Based on this, further, when the medical image observation prompt is input into the observation large language model for processing, the intermediate generated coding feature vector is subjected to fixed attribute optimization to obtain an optimized coding feature vector, including: performing self-sequence scale deterministic compression based on the eigenvalue on the intermediate generated coding feature vector to obtain a scale-indexed compressed feature vector, that is: ;in, are the eigenvalues ​​in the intermediate generated encoded feature vector, and are the minimum and maximum eigenvalues ​​in the intermediate generated encoding eigenvectors, is each eigenvalue in the scale-compressed eigenvector, where ,when When, take , which can compress the feature distribution of multimodal input based on self-sequence scaling, thereby improving the determinism of feature distribution at the expense of flexibility and adaptability.

[0048] Determine the uncertainty neighborhood of each eigenvalue in the scale-indexed compressed feature vector, and perform regional uncertainty constraint processing on the corresponding eigenvalue based on the uncertainty neighborhood of each eigenvalue to obtain a regional constraint feature vector, that is, further regional uncertainty constraint processing is performed on the features after deterministic compression, that is, for each , determine its uncertainty neighborhood , in the uncertainty neighborhood Inner, satisfied ,in Construct a threshold for the uncertain region, namely: ;in, yes The corresponding uncertainty neighborhood, Construct thresholds for uncertain regions, and They are all obtained through empirical settings or by cross-validation on the validation set, such as is 0.1, It is the individual eigenvalues ​​in the regional constraint feature vector, that is, the neighborhood uncertainty of the coded feature is divided into eigenvalue neighborhood regions to output high uncertainty risk regions, and the uncertainty region constraint premise evaluation is performed relative to a single eigenvalue, that is, the passive constraint evaluation of the uncertainty neighborhood is applied to the eigenvalue, so as to improve the coverage of the feature continuous distribution details in the uncertainty region.

[0049] Performing overall continuous attribute scatter clustering inference on each eigenvalue in the regional constraint feature vector to obtain the optimized coding feature vector, namely: ;in, and They are The mean and standard deviation of is the exponential function value with the natural constant e as the base, This involves optimizing the individual eigenvalues ​​in the encoded feature vector. Specifically, for fixed attribute mapping, accessibility imbalances must be compensated for between mapping errors and coverage loss. This means improving feature distribution coverage within uncertain regions while further enhancing global coverage integrity through accessibility compensation clustering. This improves the mapping performance of large observation models from ordered, deterministic input specifications based on multimodal, precisely structured data to outputs with strictly formatted attributes. This is then decoded to enhance the representational capabilities of visual evidence JSON.

[0050] In step S5, the visual evidence JSON, the clinical background text, and the list of K historical case report texts are input into the writing large language model to obtain a medical imaging report. That is to say, by inputting the structured visual evidence JSON obtained in step S4, the clinical background text obtained in step S1, and the list of historical case reports retrieved in step S3 into the writing large language model, a deep fusion of multi-source information can be achieved. The writing large language model can integrate this information to generate a complete report containing imaging findings and diagnostic opinions, ensuring the accuracy, logical coherence, and clinical relevance of the report, thereby making up for the lack of visual and professional knowledge fusion in traditional large language models and solving the report quality issues mentioned in the background technology.

[0051] Specifically, in an exemplary embodiment of the present application, step S5, inputting the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical imaging report, includes: S51, constructing a final report generation prompt based on the visual evidence JSON, the clinical background text and the list containing K historical case report texts; S52, inputting the final report generation prompt into the writing large language model to obtain the medical imaging report.

[0052] More specifically, in an exemplary embodiment of the present application, the final report generation prompt is "Please play the role of a senior radiologist, based on the following [clinical background] and [list of imaging findings], and refer to [similar case reports], generate a complete medical imaging report that includes [imaging findings] and [diagnostic opinion]."

[0053] For step S5, a possible implementation is as follows: First, perform S51. For example, the obtained visual evidence JSON is {"findings": [{"type": "nodule", "location": "right upper lobe", "size": "10x8mm", "morphology": "lobulated", "density": "solid"},{"type": "pleural effusion", "location": "left", "amount": "small"}]}. Clinical background text, for example, a 50-year-old male patient was admitted to the hospital with a 2-week history of cough and sputum and had a history of smoking. A list of K historical case reports, for example, the five historical medical reports most similar to the current case.

[0054] First, parse and textualize the visual evidence JSON. Visual evidence JSON is structured data that needs to be converted into natural language text that can be understood by a large language model. This is done by iterating over the "findings" array in the JSON and combining the attributes of each abnormal finding, such as type, location, size, morphology, and density, into one or more descriptive sentences. For example, the first finding in the JSON, {"type": "nodule", "location": "right upper lobe", "size": "10x8mm", "morphology": "lobulated", "density": "solid"}, can be converted to "a 10x8mm lobulated solid nodule in the right upper lobe." The second finding, {"type": "pleural effusion", "location": "left side", "amount": "small amount"}, can be converted to "a small amount of effusion in the left pleural cavity." All converted descriptive text can be combined into a single string listing the imaging findings.

[0055] Second, integrate clinical background text. This text contains key information about the patient's medical history, symptoms, and other key information, which is crucial for developing a diagnosis. It can be included directly as part of the prompt without any additional processing. For example, you could use "Patient, male, 50 years old, admitted to the hospital with a 2-week history of cough and sputum, with a history of smoking" as the clinical background section of the prompt.

[0056] Next, integrate historical case report text. A list of K historical case report texts provides a wealth of expertise and report examples. These reports can serve as a reference when writing reports generated by the large language model. To reduce the length of the prompt, the historical reports can be appropriately filtered or summarized. For example, only the imaging findings and diagnostic opinion sections of each historical report can be included, or all historical reports can be concatenated. These historical reports will be used as the similar case reports section in the prompt.

[0057] Finally, fill in the preset prompt template. According to an exemplary embodiment of the present application, the template for the final report generation prompt is: "Please play the role of a senior radiologist. Based on the following [clinical background] and [list of imaging findings], and referring to [similar case reports], generate a complete medical imaging report that includes [imaging findings] and [diagnostic opinion]." The processed list of imaging findings, clinical background, and similar case reports are accurately filled into the corresponding placeholders of this template.

[0058] Finally, proceed to S52. The final report generation prompt constructed in step S51 is input into the pre-trained large language model for writing. The large language model for writing is a general-purpose large language model or a fine-tuned large language model in the medical field. Its architecture is based on the decoder-only structure (such as the GPT series) or the encoder-decoder structure of the Transformer. Its weights and bias parameters are obtained through unsupervised pre-training and supervised fine-tuning on a large medical report dataset. During the training process, the model learns the language patterns, logical structure, and professional terminology of medical reports, as well as how to generate coherent and accurate report content based on the input information. The specific processing process is that after the large language model receives the final report generation prompt, it first encodes the prompt and converts it into a numerical representation within the model. Then, the model uses its powerful language generation capabilities to generate medical imaging reports word by word in an autoregressive manner. During the generation process, the model comprehensively considers all information in the prompt: it understands the patient's overall condition based on the clinical context; it accurately describes the objective findings in the image based on the list of imaging findings and organizes them into the imaging findings section of the report; Furthermore, the model deeply references the diagnostic thinking and expression methods in similar case reports, and combines its own medical knowledge to reason and judge the imaging findings, ultimately generating a clinically meaningful diagnostic opinion section. For example, based on a right upper lobe nodule and smoking history, as well as diagnoses in similar cases, the model might generate a diagnostic opinion suggesting a possible lung malignancy and recommending further testing. Finally, the large language model outputs a complete medical imaging report text that includes the imaging findings and diagnostic opinion. For example, a generated medical imaging report might look like this: Medical Imaging Report: Patient Information: Patient, male, 50 years old. Clinical Background: Admitted to hospital for a 2-week history of cough and sputum production, with a history of smoking. Imaging Findings: Chest CT scan shows a round, solid nodule in the right upper lobe, approximately 10x8 mm in size, with lobed margins and uniform density. A small amount of fluid density is visible in the left pleural cavity, with no significant pleural thickening. No obvious enlarged lymph nodes are observed in the bilateral hilum or mediastinum. Diagnostic Opinion: 1. The right upper lobe nodule is considered highly likely to be malignant. Further enhanced CT or PET-CT is recommended, with a puncture biopsy if necessary to confirm the diagnosis. 2. A small amount of pleural effusion is seen on the left side; clinical analysis is recommended. This report will serve as the final output of this method to assist and optimize physicians and provide comprehensive diagnostic support.

[0059] In summary, a method for generating medical imaging reports based on a large language model according to an embodiment of the present application is illustrated. It first obtains the original imaging data and clinical background text of the patient, and extracts the image embedding vector and the background embedding vector respectively, thereby realizing a preliminary digital representation of multimodal information and solving the problem that traditional LLMs are difficult to effectively integrate visual information. Next, these embedding vectors are used to perform case retrieval based on prior knowledge, and K highly relevant historical case reports are screened out from massive historical data, providing rich professional references for subsequent report generation, thus making up for the lack of historical experience utilization in existing methods. Subsequently, the image embedding vectors and historical case reports are input into the observation large language model, so that it focuses on extracting objective visual evidence from the images and outputs them in a structured JSON format, thereby ensuring the objectivity and accuracy of the report. Finally, a large language model is written to integrate the visual evidence JSON, clinical background text and historical case reports to generate the final medical imaging report. Through a phased, multimodal fusion approach, problems such as incoherent report logic and inaccurate information are effectively solved, significantly improving the quality and generation efficiency of the report.

[0060] Figure 5 FIG is a block diagram of a medical imaging report generation system based on a large language model according to an embodiment of the present application. Figure 5 As shown, according to an embodiment of the present application, a medical imaging report generation system 100 based on a large language model includes: an imaging data background text acquisition module 110, which is used to acquire the original imaging data and clinical background text of a target patient object; an imaging data background text acquisition module 120, which is used to extract visual features from the original imaging data to obtain an image embedding vector, and extract text semantic features from the clinical background text to obtain a background embedding vector; an imaging data background text acquisition module 130, which is used to perform case retrieval based on prior knowledge based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; an imaging data background text acquisition module 140, which is used to input the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; an imaging data background text acquisition module 150, which is used to input the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical imaging report.

[0061] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned medical imaging report generation system based on the large language model have been referred to above. Figures 1 to 4 It has been introduced in detail in the description of the medical imaging report generation method based on a large language model, and therefore, its repeated description will be omitted.

Claims

1. A method for generating medical imaging reports based on a large language model, characterized in that: include: Obtaining original imaging data and clinical background text of target patient subjects; Visual features are extracted from the original image data to obtain an image embedding vector, and text semantic features are extracted from the clinical background text to obtain a background embedding vector; based on the background embedding vector and the image embedding vector, a case retrieval based on prior knowledge is performed to obtain a list containing K historical case report texts; the image embedding vector and the list containing K historical case report texts are input into an observation large language model to obtain visual evidence JSON; the visual evidence JSON, the clinical background text and the list containing K historical case report texts are input into a writing large language model to obtain a medical imaging report.

2. The method for generating medical imaging reports based on a large language model according to claim 1, characterized in that: Extracting visual features from the raw image data to obtain an image embedding vector, and extracting text semantic features from the clinical background text to obtain a background embedding vector, including: preprocessing the raw image data to obtain preprocessed raw image data; inputting the preprocessed raw image data into a pretrained visual encoder to obtain the image embedding vector, wherein the pretrained visual encoder is a MedNeXt model; inputting the clinical background text into a text editor to obtain the background embedding vector, and the text editor is a BERT model.

3. The method for generating medical imaging reports based on a large language model according to claim 2, characterized in that: The original image data is preprocessed to obtain preprocessed original image data, including: adjusting the window width and window position, normalizing and resizing the original image data to obtain the preprocessed original image data.

4. The method for generating medical imaging reports based on a large language model according to claim 1, wherein: Based on the background embedding vector and the image embedding vector, a case retrieval based on prior knowledge is performed to obtain a list containing K historical case report texts, including: feature splicing of the background embedding vector and the image embedding vector to obtain a query embedding vector; using the query embedding vector as a query, performing a similarity search in an image-report vector database to obtain the list containing K historical case report texts.

5. The method for generating medical imaging reports based on a large language model according to claim 4, characterized in that: Using the query embedding vector as a query, a similarity search is performed in the image-report vector database to obtain the list of K historical case report texts, including: calculating the cosine similarity between the query embedding vector and each case embedding vector in the image-report vector database as the case matching degree to obtain a set of case matching degrees; based on the sorting of the set of case matching degrees, extracting K historical case report texts from the image-report vector database.

6. The method for generating medical imaging reports based on a large language model according to claim 1, wherein: Inputting the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON, including: constructing a medical image observation prompt based on the image embedding vector and the list containing K historical case report texts, wherein the medical image observation prompt includes listing all objective abnormal findings in a strict JSON format; inputting the medical image observation prompt into the observation large language model to obtain the visual evidence JSON.

7. The method for generating medical imaging reports based on a large language model according to claim 1, wherein: Inputting the visual evidence JSON, the clinical background text, and the list of K historical case report texts into a writing large language model to obtain a medical imaging report, including: constructing a final report generation prompt based on the visual evidence JSON, the clinical background text, and the list of K historical case report texts; and inputting the final report generation prompt into the writing large language model to obtain the medical imaging report.

8. The method for generating medical imaging reports based on a large language model according to claim 7, characterized in that: The prompt for generating the final report is "Please play the role of a senior radiologist. Based on the following [clinical background] and [list of imaging findings], and referring to [similar case reports], generate a complete medical imaging report that includes [imaging findings] and [diagnostic opinion]." 9. A medical imaging report generation system based on a large language model, characterized in that: include: An imaging data background text acquisition module is used to obtain the original imaging data and clinical background text of the target patient object; An image data background text acquisition module is used to extract visual features from the original image data to obtain an image embedding vector, and to extract text semantic features from the clinical background text to obtain a background embedding vector; an image data background text acquisition module is used to perform case retrieval based on prior knowledge based on the background embedding vector and the image embedding vector to obtain a list containing K historical case report texts; an image data background text acquisition module is used to input the image embedding vector and the list containing K historical case report texts into an observation large language model to obtain visual evidence JSON; an image data background text acquisition module is used to input the visual evidence JSON, the clinical background text and the list containing K historical case report texts into a writing large language model to obtain a medical image report.

Images