Chronic otitis media CT report generation system based on visual-language large model

Abstract

The invention discloses a chronic otitis media CT report automatic generation system based on a visual-language large model, and belongs to the technical field of radiation medicine. The system comprises a data acquisition and processing module for acquiring image data and text data and preprocessing the data; the vision-language large model training module is used for inputting the preprocessed data into a vision-language large model for training and adjusting and optimizing model parameters; and the report generation interaction module is used for generating a structured report by utilizing the trained vision-language large model. According to the system, the visual-language large model is applied to the field of CT report generation and typing diagnosis of chronic otitis media for the first time, the technical blank in the field is filled, and a new path is developed for intelligent diagnosis of chronic otitis media.

Description

Technical Field

[0001] This invention belongs to the field of radiomedical technology and relates to a method for generating medical image reports, specifically a system for generating CT reports for chronic otitis media based on a visual-language large model. Background Technology

[0002] Chronic otitis media, a common recurrent inflammatory disease of the tympanic cavity and mastoid cavity, is one of the leading causes of hearing loss in children and adults worldwide. Chronic suppurative otitis media and cholesteatoma of the middle ear are the most common types. Because these two diseases present similarly but have significantly different treatment plans and prognoses, accurate differential diagnosis is crucial for clinical diagnosis and prognostic assessment. Currently, high-resolution computed tomography (HRCT) of the temporal bone is the primary method for differential diagnosis of chronic otitis media. However, due to the intricate and complex anatomical structure of the temporal bone, HRCT image interpretation is extremely difficult, easily leading to missed diagnoses and misdiagnoses by junior ENT physicians and doctors in areas with limited medical resources. Artificial intelligence-assisted medical image information analysis and report generation can reduce the workload of radiologists and help improve the accuracy of the final diagnosis.

[0003] In recent years, artificial intelligence technology has made some progress in the imaging diagnosis of chronic otitis media, improving diagnostic accuracy. Existing diagnostic algorithms mostly employ deep convolutional neural networks (CNNs) to extract image features and perform subtyping diagnosis by outputting classification probability values, such as the 3D CNN-based model developed by Su Ri et al., and the method based on fast region convolutional neural networks proposed by Schilder et al. Although these models perform well in terms of accuracy (usually exceeding 0.8), their structural limitations mean they can only output disease subtyping probability values, lacking complete image report interpretation and diagnostic process explanations, resulting in severely insufficient interpretability and greatly limiting their application in clinical practice. While some studies have attempted to demonstrate the model's attention mechanism through images such as heatmaps to improve interpretability, there is still a significant gap compared to textual explanations that are easily understood by clinicians.

[0004] Meanwhile, research on the visual-language large model in CT imaging has been deepening and has been widely applied in various CT image report generation scenarios, demonstrating good interpretability and development potential. However, there is currently no relevant research on the temporal bone CT classification diagnosis of chronic otitis media, especially research on subsequent reasoning and classification diagnosis based on internal knowledge. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a CT report generation system for chronic otitis media based on a large vision-language model, thus solving the problems in the existing technologies.

[0006] The objective of this invention can be achieved through the following technical solutions: A CT report generation system for chronic otitis media based on a large vision-language model includes: Data acquisition and processing module: Acquires image and text data, and preprocesses the data; Visual-Language Large Model Training Module: Inputs preprocessed data into the visual-language large model for training and optimizes model parameters; Report generation interactive module: Utilizes a trained visual-language large model to generate structured reports.

[0007] Furthermore, the image data includes: DICOM format HRCT images; The text data includes: outpatient and inpatient medical records, imaging text reports, and postoperative pathology reports.

[0008] Furthermore, the image data preprocessing process includes: 1) Based on the canthal-auricular line, Reid baseline, and orbitoauricular line at the skull base, the B-spline interpolation algorithm was used to correct the scanning angle deviation to ≤1.5°; 2) Scale the HU values ​​of the temporal bone region to [-1000, 400] and add an offset of 1024 to convert them to unsigned integers; 3) The three-dimensional images were unified to 32×256×256 voxels by trilinear interpolation, while maintaining a coronal resolution of 0.4mm×0.4mm.

[0009] Furthermore, the text data preprocessing process includes: constructing a standardized library of otological imaging terminology, and checking and standardizing the terminology of the text data.

[0010] Furthermore, the visual-language large model includes a visual encoder and a large language model. The preprocessed CT images are stitched together layer by layer and then input into the visual encoder. After processing by the visual encoder, they enter the visual projection layer to obtain the features of the CT images. After being stitched together with tokens, they are input into the large language model. The preprocessed image report text is directly stitched to the end of the CT image tokens and input into the large language model together.

[0011] Furthermore, during the training phase of the visual-language large model, the parameters of the visual encoder are frozen, and only the parameters of the large language model are fine-tuned in a supervised manner. The text of the image report is used as the prediction target and a mask is set. The large language model outputs a report based on the input CT image features and updates its internal parameters based on the deviation from the given image report text, thereby achieving the training objective.

[0012] Furthermore, the visual encoder adopts the ViT architecture and uses the ViT-bigG model pre-trained by OpenCLIP. The visual encoder first segments the image into multiple image blocks, and then, for each image block, it transforms it into a token through a series of linear mappings, and then concatenates all the tokens into a sequence.

[0013] Furthermore, the loss function L during the training phase CE for: Where C represents the vocabulary length, y i It is the text of the label. It is the text predicted by the model.

[0014] Furthermore, after training the large visual-language model, the BLEU metric was used to validate the effectiveness of the generated reports: Where BP represents the penalty parameter; p n p represents the n-gram precision. n =Number of n-grams in the generated text that match the reference text / Total number of n-grams in the generated text; w n Indicates the weighting coefficient. N The maximum number of n-grams.

[0015] Furthermore, the report production interaction module adopts a human-computer interaction mechanism, including: Clicking on a field in the report triggers an edit menu, which supports selecting preset terms from a dropdown menu, dragging and dropping to adjust inference steps, adding literature citations, manually adjusting probability values, and triggering local model retraining.

[0016] The beneficial effects of this invention are: 1. The invention is the first to apply a large visual-language model to the generation and classification diagnosis of CT reports for chronic otitis media, filling a technological gap in the field and opening up a new path for the intelligent diagnosis of chronic otitis media.

[0017] 2. The report generation system of the present invention can significantly improve the efficiency, standardization and accuracy of image report generation, effectively reduce the workload of imaging diagnostic physicians; at the same time, it provides clinicians with more reliable and comprehensive diagnostic evidence, strongly assists clinical decision-making, and has extremely high clinical application value and broad market prospects. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a diagram of the CT report generation system for chronic otitis media according to the present invention; Figure 2 This is a schematic diagram of the visual encoder of the present invention; Figure 3 This is the input image for training the visual-language large model of this invention; Figure 4 This is a framework diagram of the large language model of this invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, the CT report generation system for chronic otitis media based on a large vision-language model includes: Data acquisition and processing module: Acquires image and text data, and preprocesses the data; Visual-Language Large Model Training Module: Inputs preprocessed data into the visual-language large model for training and optimizes model parameters; Report generation interactive module: Utilizes a trained visual-language large model to generate structured reports.

[0022] 1. Data Acquisition and Processing Module The sample data for this invention are derived from: preoperative CT images and case records of patients with chronic otitis media; The image data includes: DICOM format HRCT images (slice thickness 0.5-1.0mm, matrix 512×512), covering data acquired by mainstream equipment such as Siemens and GE.

[0023] Text data includes: outpatient and inpatient medical records, imaging text reports, and postoperative pathology reports (ICD-11 coded and labeled); The process of image data preprocessing includes: 1) Orientation Correction: Based on the three major planes of the skull base (canthal-auricular line, Reid baseline, and orbitoauricular line), a B-spline interpolation algorithm is used to correct the scanning angle deviation to ≤1.5°. 2) HU value normalization: Scale the HU values ​​of the temporal bone region (x:-150~150mm, y:-100~100mm, z:-50~150mm) to [-1000, 400] and add an offset of 1024 to convert them to unsigned integers.

[0024] 3) Size standardization: The three-dimensional images are standardized to 32×256×256 voxels (axial×coronal×sagittal) through trilinear interpolation, maintaining a coronal / sagittal resolution of 0.4mm×0.4mm.

[0025] The main preprocessing of text data is terminology standardization: a standardized library of otology imaging terminology is built, and experts manually check and standardize the terminology in the imaging report text.

[0026] 2. Visual-Language Large Model Training Module The visual-language large model comprises two main modules: a visual encoder and a large language model. Preprocessed CT images are stitched together layer by layer and then input into the visual encoder. After processing by the visual encoder, the images enter the visual projection layer to obtain CT image features (ct_img_tokens). These features are then stitched together using special tokens and input into the large language model, such as... ct_img_tokens. In addition, the preprocessed image report text is directly appended to the CT image tokens mentioned above and input into the large language model.

[0027] During the training phase, the parameters of the visual encoder are frozen, and only the parameters of the large language model are subjected to supervised fine-tuning (SFT). This involves using the text of the image report as the prediction target and setting a mask, allowing the large language model to output a report based on the features of the input CT image, and updating the internal parameters based on the deviation (loss) from the given image report text, thereby achieving the training objective.

[0028] The visual encoder employs the ViT (VisionTransformer) architecture. The network settings and initialization parameters of ViT utilize the pre-trained ViT-bigG model from OpenCLIP. Its principle is as follows: Figure 2 As shown, the image is first divided into multiple image patches. Then, for each patch, a series of linear mappings are used to convert it into a token. Finally, all the tokens are concatenated into a sequence.

[0029] In this embodiment, the length and width of each CT image are fixed at 448 pixels, and each image block is 14 pixels. The Transformer visual encoder has 32 layers, each with 16 attention heads, and the feature dimension of each output image block is 1280. Subsequently, a simple visual projection layer is used to map the feature dimension from 1280 to 8192, with the aim of aligning the dimension with the feature dimension of the large language model.

[0030] In this embodiment, a pre-trained large language model is used. For example... Figure 4 As shown, it includes three parts: input layer, decoder layer, and output layer. The input layer transforms the original text into vectors that the model can compute. Specifically, after word segmentation, the token embedding corresponding to each word is found, and each word is mapped to a fixed-dimensional vector (such as 8192-dimensional). At the same time, it is superimposed with the positional encoding that represents the text order, and the output is a "word vector matrix" (text length × vector dimension) with positional information.

[0031] The decoder layer contains multiple decoders to capture the contextual logic of the text. The internal structure of a single layer is as follows: Masked Multi-Head Attention calculates word-to-word association weights from multiple "attention heads" on the input word vector matrix, focusing on key contextual information. Residual Connections directly add the input and output of the attention layer, avoiding information loss during deep training. Normalization normalizes the result after the residual connections, making the data distribution more stable and accelerating model training. A Feedforward Neural Network (MLP) performs a non-linear transformation on the normalized vectors, extracting more complex semantic features through "compression and expansion."

[0032] The output layer transforms the "feature vector" output by the decoder layer into specific candidate words to generate text. It primarily involves a Softmax operation that converts the vector output by the linear layer into a "probability distribution" on the vocabulary, selects the word with the highest probability as the currently generated word, and then re-inputs the vector of that word into the decoder, repeating the process to generate the next word (autoregressive generation).

[0033] In this embodiment, 80 stacked layers, 16 attention heads, and a hidden layer vector size of 8192 are set.

[0034] The training process of the large-scale vision-language model includes: Step 1: Divide the preprocessed data into training and validation sets. Input the training set data into the vision-language large-scale model for training. Each data point consists of multiple CT images and an image report annotated by experts, as shown in the example below: Table 1 Examples of Training Samples During the training phase, the parameters of the visual encoder are frozen, and only the parameters of the large language model are subjected to supervised fine-tuning (SFT). This involves using the text of the image report as the prediction target and setting a mask, allowing the large language model to output a report based on the input CT image features. The internal parameters are updated based on the deviation (loss) from the given image report text, thus achieving the training objective. The loss function used is cross-entropy loss Li. CE ,as follows: In this context, C represents the word length. y i It is the text of the label. This is the text predicted by the model. The final loss function is obtained by averaging the loss for each text in each sample. The Adam optimizer is used to optimize the parameters of the large model. In this case, the initial learning rate of the optimizer is set to 10. -5 .

[0035] Step 2: Validate the trained visual-language large model using validation set data.

[0036] After training, validation was performed on the validation set. The BLEU metric was used to automatically validate the effectiveness of the generated reports, as detailed below: Here, BP represents the penalty parameter. If the length of the generated text (c) is much shorter than the length of the reference text (r), a penalty will be triggered, i.e., c≤r, BP=exp(1-r / c); if c>r, BP=1 with no penalty. n This represents the precision of n-grams, where n-gram refers to a combination of n consecutive words. BLEU typically calculates the precision of 1-grams to 4-grams (i.e., p1 to p4). Precision formula: p n =Number of n-grams in the generated text that match the reference text / Total number of n-grams in the generated text. w n This represents the weighting coefficient.

[0037] 3. Report generation interactive module In the report production interaction module, after data preprocessing, it is input into the trained model in DICOM / NIfTI format, automatically parsing metadata such as patient ID and scan parameters to generate a structured report; the generated structured report conforms to the HL7FHIR standard and supports integration with electronic medical record systems.

[0038] The report production interaction module adopts a human-computer interaction mechanism, specifically including: Doctors can click on the yellow highlighted field in the report (such as "soft tissue shadow in the left tympanic cavity") to trigger the modification menu, which supports selecting preset terms from the drop-down menu, dragging and adjusting the inference steps, adding references from literature such as "Ear Microsurgery", and manually adjusting probability values ​​to trigger local model retraining.

[0039] Furthermore, the report's production interaction module employs a differential update strategy, fine-tuning only model parameters related to modified fields. Using an RTX 4090 GPU, update time is ≤15 seconds.

[0040] In this embodiment, the quality of the report generated by the above system is evaluated and verified. The subtyping diagnosis results generated by the model are compared with the postoperative pathological gold standard. The subtyping accuracy, site localization accuracy, and other indicators are calculated. The evaluation indicators and calculation methods are shown in Table 2 below: Table 2 Evaluation Indicators and Objectives In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0041] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A chronic otitis media CT report generation system based on a visual-linguistic large model, characterized by, The application relates to an ear imaging report generation method and system. The data acquisition and processing module acquires image data and text data and pre-processes the data; The visual-language large model training module inputs the pre-processed data into a visual-language large model for training and optimizes model parameters; The report generation and interaction module generates a structured report by using the trained visual-language large model.

2. The visual-linguistic large model-based chronic otitis media CT report generation system of claim 1, wherein, The image data comprises DICOM format HRCT images; The text data comprises outpatient and inpatient medical records, imaging text reports and postoperative pathological reports. 3.The visual-linguistic large model based chronic otitis media CT report generation system of claim 1, wherein, The image data pre-processing process comprises the following steps: 1) Based on the canthus-ear line, Reid baseline and auditory-orbital line of the skull base, a B-spline interpolation algorithm is used to correct the scanning angle deviation to less than or equal to 1.5 degrees; 2) The HU value of the temporal bone region is scaled to [-1000, 400] and a 1024 offset is added to convert it into an unsigned integer; 3) The three-dimensional image is unified into 32x256x256 voxels by trilinear interpolation, and the coronal plane resolution is kept at 0.4mmx0.4mm.

4. The visual-linguistic large model-based chronic otitis media CT report generation system of claim 1, wherein, The text data pre-processing process comprises the following steps: a standardized library of otological imaging terms is constructed, and the text data is checked and standardized. 5.The visual-linguistic large model based chronic otitis media CT report generation system of claim 1, wherein, The visual-language large model comprises a visual encoder and a large language model. The pre-processed CT image is sequentially spliced and input into the visual encoder, and the CT image features are obtained after the visual encoder processing and visual projection layer. After token splicing, the large language model is transmitted; the pre-processed image report text is directly spliced behind the CT image token and input into the large language model together. 6.The visual-linguistic large model based chronic otitis media CT report generation system of claim 5, wherein, During the training of the visual-language large model, the parameters of the visual encoder are frozen, and only the parameters of the large language model are supervised and fine-tuned. The text of the image report is used as the prediction target and a mask is set. The large language model outputs the report according to the input CT image features, and the internal parameters are updated according to the deviation from the given image report text, so that the training target is achieved. 7.The visual-linguistic large model based chronic otitis media CT report generation system of claim 5, wherein, The visual encoder adopts the ViT architecture and uses the OpenCLIP pre-trained ViT-bigG model. The visual encoder first divides the picture into multiple image blocks, then converts each image block into a token through a series of linear mappings, and finally splices all the tokens into a sequence. 8.The visual-linguistic large model based chronic otitis media CT report generation system of claim 6, wherein, The loss function L of the training phase CE is: where C is the length of the vocabulary, y i is the text of the label, is the text predicted by the model. 9.The visual-linguistic large model based chronic otitis media CT report generation system of claim 6, wherein, After the visual-language large model is trained, the effect of the generated report is verified by using the BLEU index. where BP denotes a penalty parameter; p n denotes n-gram precision, p n = the number of n-grams in the generated text that match the reference text / the total number of n-grams in the generated text; w n denotes a weight coefficient, N is the maximum number of n-grams. 10.The visual-linguistic large model based chronic otitis media CT report generation system of claim 1, wherein, The report production and interaction module adopts a human-computer interaction mechanism, which comprises the following steps: Clicking a field in the report triggers a modification menu, supports a drop-down menu to select preset terms, drag-and-drop adjustment of inference steps, addition of literature references and manual adjustment of probability values, and triggers local retraining of the model.

Images