Brain tumor diagnosis and report generation method and system based on large language model
Through multimodal diagnostic methods based on large language model, feature extraction and report generation of brain tumor images is solved, and the efficiency and accuracy of brain tumor diagnosis in the existing technology is achieved, and rapid and accurate diagnostic information extraction and report generation are achieved. , improves the efficiency of medical services and diagnostic reliability.
Patent Information
- Application Number
- CN202411257500.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-09
AI Technical Summary
In the prior art, the diagnosis of brain tumors depends on the experience of doctors and complex and complicated tissue section examinations, making it difficult to diagnose efficiently and accurately, especially for the processing and clinical research of large numbers of tissue specimens, which have errors and time-consuming problems.
Using brain tumor diagnosis and report generation methods based on large language model, the characteristics of the craniocerebral MRI images and image reports are extracted, combined with the shared weight alignment of visual channels and text channels, and input them into the pre-trained large language model to generate medical reports. and diagnostic results, use multimodal models to reduce the burden on doctors and improve diagnostic accuracy.
It realizes rapid and accurate extraction of diagnostic information and generates structured medical reports, which reduces the burden on doctors, improves the efficiency and reliability of diagnosis, and reduces the possibility of human error.
Smart Images

Figure CN119049638A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a brain tumor diagnosis and report generation method and system based on a large language model. Background Art
[0002] Medical imaging plays a vital role in the process of medical diagnosis, especially brain Magnetic Resonance Imaging, which can provide a lot of information about brain structure and function. Compared with imaging technologies such as CT and X-ray, MRI provides better soft tissue contrast, higher resolution, and no radiation risk. Therefore, MRI provides strong data support for doctors' diagnosis and report writing. However, due to the large differences in the morphology of similar brain tumors and the similarity of information such as edge details of different types of brain tumors, it is difficult to determine the type of tumor only by MRI, and even experienced radiologists cannot make a true diagnosis of the tumor type. At present, the diagnosis of tumors in the medical system mainly relies on biopsy of the tumor lesion, that is, histopathological examination. When formulating a patient's medical plan, the doctor needs to integrate the patient's medical history, pathological sections, radiological scan results and imaging reports. These decisions often rely on the doctor's experience and take a long time.
[0003] Currently, the diagnosis and grading of brain tumors is mainly performed by pathologists who examine tissue sections on glass slides with the help of a microscope. Although this method is widely used in the clinic, it becomes complicated and error-prone for translational and clinical research that requires processing hundreds or even thousands of tissue specimens. Summary of the invention
[0004] Based on the technical problems existing in the background technology, the present invention proposes a brain tumor diagnosis and report generation method and system based on a large language model, which reduces the burden on doctors, improves the efficiency of medical services, and improves the accuracy and reliability of diagnosis.
[0005] The brain tumor diagnosis and report generation method based on the large language model proposed in the present invention inputs the cranial MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and a diagnosis result;
[0006] The training process of the diagnosis and report generation model is as follows:
[0007] Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features;
[0008] Step 2: The alignment module is set up with a visual channel and a text channel, and the global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain aligned visual features, and the aligned visual features are spliced in front of the embedding layer of the aligned text features output by the text channel to obtain aligned visual-text features;
[0009] Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report, input the medical report into the diagnosis module to output the diagnosis result, and construct a loss function to adjust the model parameters in the diagnosis and report generation model.
[0010] Further, in step 2, the visual channel includes a plurality of visual attention blocks arranged in series, each visual attention block includes a visual self-attention layer, a visual cross-attention layer and a visual feedforward network layer connected in sequence, the input of the visual self-attention layer of the current visual attention block is connected to the output of the visual feedforward network layer of the previous visual attention block, and the output of the visual feedforward network layer of the current visual attention block is connected to the input of the visual self-attention layer of the next visual attention block;
[0011] The text channel includes a plurality of text attention blocks arranged in series, each text attention block includes a text self-attention layer and a text feedforward network layer connected in sequence, the input of the text self-attention layer of the current text attention block is connected to the output of the text feedforward network layer of the previous text attention block, and the output of the text feedforward network layer of the current text attention block is connected to the input of the text self-attention layer of the next text attention block.
[0012] Furthermore, in step 2, weights are shared between the visual channel and the text channel, specifically: the same number of visual attention blocks and text attention blocks are set, and the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block.
[0013] Furthermore, the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block, specifically:
[0014] In the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated, and the query vector is shared with the text self-attention layer through the visual self-attention layer and transmitted to the visual channel to realize interaction with the text in the visual channel.
[0015] Furthermore, in step three, the medical report is input into the diagnosis module to output the diagnosis result, specifically:
[0016] The medical report is converted into a text feature vector through a text preprocessing layer;
[0017] The context information in the text feature vector is captured through a bidirectional long short-term memory network, and the output is a hidden state vector for each time step;
[0018] Perform a full connection mapping on the hidden state vector of the last time step output by the bidirectional long short-term memory network to generate a high-dimensional feature vector;
[0019] The ReLU activation function is used to increase nonlinearity, and then another fully connected layer is used to map the high-dimensional feature vector to a vector of the number of categories. Finally, the Softmax activation function is used to convert the vector of the number of categories into a probability distribution for predicting tumor categories, thereby generating a diagnostic result.
[0020] Furthermore, a loss function is constructed to adjust the model parameters in the diagnosis and report generation model, which is specifically:
[0021] The contrast loss in the alignment module is constructed based on the InfoNCE loss function. The contrast loss acts on the aligned visual-text features output by the alignment module and the text semantic representation corresponding to the medical report output by the large language model. The specific formula is as follows:
[0022]
[0023] Among them, L c represents the contrast loss, N is the number of samples in a batch, including positive sample pairs and negative sample pairs, q i is the aligned visual feature corresponding to the i-th sample, k + is the text semantic representation of the medical report corresponding to the positive sample, τ is the temperature coefficient, k j is the text semantic representation corresponding to the medical report of the jth sample in the batch.
[0024] Furthermore, by constructing a loss function to adjust the model parameters in the diagnosis and report generation models, the large language model in the report generation module adopts the cross entropy loss L ac , specifically:
[0025]
[0026] Among them, w t is the target word at the current time step t, w <t is all words before the current time step, p θ (w t |w<t ) is the large language model given w <t Under the condition of predicting w t The probability of , T is the total time step;
[0027] The diagnosis module uses the cross entropy loss function L diag , used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label;
[0028] The loss function constructed by the diagnosis and report generation model is as follows:
[0029] L total =λ align L c +λ gen L ac +λ diag L diag
[0030] Among them, λ align ,λ gen ,λ diag They are contrast loss L c , cross entropy loss L ac , cross entropy loss function L diag The corresponding weight hyperparameters.
[0031] Furthermore, in step 1, feature extraction is performed on the cranial MRI medical image, specifically:
[0032] Normalize the cranial MRI medical images and cut them into image blocks of fixed size;
[0033] Each image block is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. A position code is added to each embedded image block to retain the position information in the original image.
[0034] The position-encoded image blocks are input into the Transformer encoder, and the features output by the last Transformer layer are used as global visual features.
[0035] The brain tumor diagnosis and report generation system based on the large language model inputs the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results;
[0036] The diagnosis and report generation model includes a preprocessing module, an alignment module, a report generation module, a diagnosis module and a loss module; the training process of the diagnosis and report generation model is as follows:
[0037] The preprocessing module is used to obtain cranial MRI medical images and corresponding imaging reports, and to extract features from the cranial MRI medical images and imaging reports to obtain global visual features and text features;
[0038] The alignment module is used to align the visual channel and the text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain aligned visual features, and the aligned visual features are spliced in front of the embedding layer of the aligned text features output by the text channel to obtain aligned visual-text features.
[0039] The report generation module is used to input the aligned visual-text features into the pre-trained large language model to generate a medical report.
[0040] The diagnosis module is used to input the medical report into the diagnosis module to output the diagnosis result;
[0041] The loss module is used to construct loss functions to adjust model parameters in diagnosis and report generation models.
[0042] The advantages of the brain tumor diagnosis and report generation method and system based on a large language model provided by the present invention are: the brain tumor diagnosis and report generation method and system based on a large language model provided in the structure of the present invention utilizes the deep learning visual encoder for tumor image information, and utilizes the ability of the language large model to perform the medical report generation task. Therefore, this embodiment utilizes the ability of the multimodal large model (diagnosis and report generation model) to perform the medical report generation task, and uses the medical report to assist the tumor diagnosis task; it can quickly and accurately extract key diagnostic information and generate a structured medical report. This not only greatly reduces the burden on doctors and improves the efficiency of medical services, but also reduces the possibility of human errors, thereby improving the accuracy and reliability of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic diagram of the process of the present invention;
[0044] Figure 2 Flowchart for training models for diagnosis and report generation;
[0045] Figure 3 Schematic diagram of the structure of the model for diagnosis and report generation. DETAILED DESCRIPTION
[0046] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0047] In view of the technical problems in the background technology, we began to explore the combination of natural language processing (NLP) technology and medical image analysis. In this way, the present embodiment has developed a multimodal diagnostic report generation model that can automatically generate medical diagnostic reports. The significance of the technology of automatically generating medical reports lies in its significant contribution to improving the efficiency and accuracy of medical diagnosis and the treatment results of patients. By analyzing massive medical image data using deep learning and machine learning algorithms, this technology can quickly and accurately extract key diagnostic information and generate structured medical reports. This not only greatly reduces the burden on doctors and improves the efficiency of medical services, but also reduces the possibility of human errors, thereby improving the accuracy and reliability of diagnosis, as follows.
[0048] like Figures 1 to 3 As shown, the brain tumor diagnosis and report generation method based on the large language model proposed by the present invention inputs the cranial MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and a diagnosis result;
[0049] The training process of the diagnosis and report generation model is as follows:
[0050] Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features;
[0051] The ViT (Vision Transformer) model is used to extract features from cranial MRI medical images. Through the multi-head self-attention mechanism, the ViT model can capture the long-range dependencies between image blocks, thereby extracting global visual features. After these features are processed by layers of encoders, a final global visual feature will be generated.
[0052] Specifically, the ViT model first preprocesses the MRI medical image, including normalization and cutting into fixed-size image blocks. For example, a 224x224 MRI medical image is cut into 16x16 image blocks, with a total of 14x14=196 image blocks. Each image block is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. After embedding, a position code is added to each image block to retain its position information in the original image. After that, these image blocks are input into multiple Transformer encoder layers. Each encoder layer consists of a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer can establish long-range dependencies between image blocks, while the feedforward neural network can further extract deep features. Finally, the feature representation output from the last Transformer encoder layer is used as a global visual feature and can be input into the subsequent alignment module. In this way, by using the ViT model, the advantages of the Transformer architecture can be fully utilized to extract high-quality MRI image visual features and improve the accuracy and robustness of medical image analysis tasks.
[0053] Step 2: The alignment module is set up with a visual channel and a text channel, and the global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain aligned visual features, and the aligned visual features are spliced in front of the embedding layer of the aligned text features output by the text channel to obtain aligned visual-text features;
[0054] Alignment module, the global visual features learned from the ViT model cannot be directly understood by the large language model. The alignment module is the bridge that aligns the global visual features to the text embedding feature space. It is divided into two channels, the first channel is the visual channel, and the second channel is the text channel. The visual attention blocks and the text attention blocks are set to the same number. The visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block; that is: in the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and the text vector output by the current text self-attention layer are separated, and the query vector is transmitted to the visual channel through the visual self-attention layer and the text self-attention layer to interact with the text in the visual channel.
[0055] Specifically, the visual channel extracts visual features by interacting with the visual encoder. The input of the visual channel is a learnable query vector. These query vectors interact with themselves through the visual self-attention layer SA (selfattention) and interact with the visual features through the visual cross-attention layer CA (cross-attention) to learn which tokens to extract from the visual features. At the same time, they can also interact with the text through the shared self-attention layer.
[0056] The other channel is the text channel, whose main function is to interact with the visual features through the text self-attention layer shared with the visual channel. The weight sharing method here is to splice the learnable query vector of the visual channel and the output of the text encoder and input them into a self-attention layer. The mask is used to control the interaction between the visual channel and the text channel. After the text self-attention layer, the query vector and the text vector are separated for subsequent operations. After these two channels, text features need to be generated from visual features. The output of the visual channel is connected to a fully connected layer to generate an aligned visual feature with the same dimension as the embedding of the large language model. The aligned visual features are spliced in front of the embedding layer (embedding) of the aligned text features output by the text channel to obtain the aligned visual-text features.
[0057] Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report, input the medical report into the diagnosis module to output the diagnosis result, and construct a loss function to adjust the model parameters in the diagnosis and report generation model.
[0058] The report generation module uses a large language model to receive the aligned visual text features as input and outputs the report text corresponding to the image. The network structure of the large language model is composed of multiple layers of Transformer stacking, which can well model the contextual relationship of the text. The large language model is usually pre-trained on a large-scale text corpus and can learn a general language representation. This pre-trained knowledge can be transferred to applications in specific fields, such as the medical field. By further fine-tuning on a domain-specific data set, the large language model can recognize specific medical terms and representations, and better understand the professional knowledge related to brain tumor diagnosis. Since the corpus provided by the data set is all in Chinese, this embodiment can use the Chinese large language model ChatGLM for training and fine-tuning. ChatGLM is a large language model based on the Transformer architecture, pre-trained on a large amount of Chinese text corpus, and has good Chinese understanding and generation capabilities. By using this model, its advantages in natural language processing can be fully utilized to generate high-quality Chinese reports for MRI images. In this way, the report generation module can convert complex MRI image data into structured, easy-to-understand Chinese medical reports, helping doctors make diagnostic decisions faster and improve clinical efficiency and diagnostic quality.
[0059] The task of the tumor diagnosis module is to convert the medical report text output by the report generation module into the diagnosis result of the tumor category. First, the text preprocessing layer converts the image report text into a text feature vector. This step uses a pre-trained word embedding model (such as Word2Vec or GloVe) to convert each word into a vector representation of fixed dimension and output a two-dimensional matrix. Next, the text features are processed through a bidirectional long short-term memory network (BiLSTM), which can capture the contextual information in the text sequence and output a hidden state vector for each time step. Then, enter the fully connected layer and perform a fully connected mapping on the hidden state of the last time step output by the BiLSTM layer to generate a high-dimensional feature vector. Subsequently, the nonlinearity is increased through the ReLU activation function, and the high-dimensional feature vector is mapped to a vector of the number of categories through another fully connected layer. Finally, it is converted into a probability distribution through the Softmax activation function for predicting the tumor category, thereby generating a diagnosis result.
[0060] Three different optimization objectives are used for the alignment module, report generation module, and diagnosis module:
[0061] (1) Alignment module: The contrast loss function in this module adopts the InfoNCE loss function. The contrast loss function acts on the aligned visual-text features output by the alignment module and the text semantic representation corresponding to the medical report output by the large language model. It is defined as follows:
[0062]
[0063] Among them, L c represents the contrast loss, N is the number of samples in a batch, including positive sample pairs and negative sample pairs, q i is the aligned visual feature corresponding to the i-th sample, k + is the text semantic representation of the medical report corresponding to the positive sample, τ is the temperature coefficient, k j is the text semantic representation corresponding to the medical report of the jth sample in the batch.
[0064] InfoNCE loss is used in this embodiment scenario to promote the consistency between aligned visual features and text semantic representations. By maximizing the similarity of text representations that match the image and minimizing the similarity with other irrelevant text representations, this loss function helps learn a diagnosis and report generation model that can correctly align visual images and text descriptions.
[0065] (2) Report generation module: In large language models, the autoregressive loss function is used to train the model to predict the next word or character in sequence. The main purpose is to maximize the probability of the next word given the conditional probability of the previous part of the sequence. This embodiment uses cross-entropy loss:
[0066]
[0067] Among them, w t is the target word at the current time step t, w <t is all words before the current time step, p θ (w t |w <t ) is the large language model given w <t Under the condition of predicting w t The probability of T is the total time step. Through the loss function L ac ,The diagnosis and report generation model is trained to predict the probability of the next word to guide the generation of medical reports.
[0068] (3) Tumor diagnosis module: In order to optimize the performance of the tumor diagnosis module, the cross-entropy loss function is used. The cross-entropy loss function is used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label:
[0069]
[0070] Among them, L diag is the cross entropy loss function, y cis the true label of the cth sample, p c Generates the probability distribution of the model output for the cth sample after diagnosis and reporting.
[0071] Therefore, the loss function constructed by the diagnosis and report generation model in this embodiment is as follows:
[0072] L total =λ align L c +λ gen L ac +λ diag L diag
[0073] Among them, λ align ,λ gen ,λ diag They are contrast loss L c , cross entropy loss L ac , cross entropy loss function L diag The corresponding weight hyperparameter is used to balance the loss of different modules to the total loss L total The contribution of align =3λ gen =3λ diag By optimizing the comprehensive loss function, the performance of the alignment module, report generation module, and tumor diagnosis module can be jointly improved, ultimately achieving efficient and accurate tumor diagnosis.
[0074] Through steps one to three, this embodiment uses deep learning visual encoders for tumor image information and the ability of a large language model to perform medical report generation tasks. Therefore, this embodiment uses the ability of a multimodal large model (diagnosis and report generation model) to perform medical report generation tasks and use medical reports to assist tumor diagnosis tasks. The proposed method can provide convenience for clinicians to diagnose and formulate treatment plans.
[0075] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for diagnosing and reporting brain tumors based on a large language model, characterized in that: Input the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results; The training process of the diagnosis and report generation model is as follows: Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features; Step 2: The alignment module is provided with a visual channel and a text channel. The global visual feature is input into the visual channel as a learnable query vector, and the text feature is input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain aligned visual features, and the aligned visual features are spliced in front of the embedding layer of the aligned text features output by the text channel to obtain aligned visual-text features. Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report, input the medical report into the diagnosis module to output the diagnosis result, and construct a loss function to adjust the model parameters in the diagnosis and report generation model.
2. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, characterized in that: In step 2, the visual channel includes a plurality of visual attention blocks arranged in series, each visual attention block includes a visual self-attention layer, a visual cross-attention layer and a visual feedforward network layer connected in sequence, the input of the visual self-attention layer of the current visual attention block is connected to the output of the visual feedforward network layer of the previous visual attention block, and the output of the visual feedforward network layer of the current visual attention block is connected to the input of the visual self-attention layer of the next visual attention block; The text channel includes a plurality of text attention blocks arranged in series, each text attention block includes a text self-attention layer and a text feedforward network layer connected in sequence, the input of the text self-attention layer of the current text attention block is connected to the output of the text feedforward network layer of the previous text attention block, and the output of the text feedforward network layer of the current text attention block is connected to the input of the text self-attention layer of the next text attention block.
3. The method for diagnosing and reporting brain tumors based on a large language model according to claim 2, characterized in that: In step 2, weights are shared between the visual channel and the text channel. Specifically, the same number of visual attention blocks and text attention blocks are set, and the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block.
4. The method for diagnosing and reporting brain tumors based on a large language model according to claim 3, characterized in that: The visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block, specifically: In the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated, and the query vector is shared with the text self-attention layer through the visual self-attention layer and transmitted to the visual channel to realize interaction with the text in the visual channel.
5. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, characterized in that: In step three, the medical report is input into the diagnosis module to output the diagnosis result, specifically: The medical report is converted into a text feature vector through a text preprocessing layer; The context information in the text feature vector is captured through a bidirectional long short-term memory network, and the output is a hidden state vector for each time step; Perform a full connection mapping on the hidden state vector of the last time step output by the bidirectional long short-term memory network to generate a high-dimensional feature vector; The ReLU activation function is used to increase nonlinearity, and then another fully connected layer is used to map the high-dimensional feature vector to a vector of the number of categories. Finally, the Softmax activation function is used to convert the vector of the number of categories into a probability distribution for predicting tumor categories, thereby generating a diagnostic result.
6. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, characterized in that: The loss function is constructed to adjust the model parameters in the diagnosis and report generation model, which is specifically: The contrast loss in the alignment module is constructed based on the InfoNCE loss function. The contrast loss acts on the aligned visual-text features output by the alignment module and the text semantic representation corresponding to the medical report output by the large language model. The specific formula is as follows: Among them, L c represents the contrast loss, N is the number of samples in a batch, including positive sample pairs and negative sample pairs, q i is the aligned visual feature corresponding to the i-th sample, k + is the text semantic representation of the medical report corresponding to the positive sample, τ is the temperature coefficient, k j is the text semantic representation corresponding to the medical report of the jth sample in the batch.
7. The method for diagnosing and reporting brain tumors based on a large language model according to claim 6, characterized in that: By constructing a loss function to adjust the model parameters in the diagnosis and report generation models, the large language model in the report generation module uses the cross entropy loss L ac , specifically: Among them, w t is the target word at the current time step t, w <t is all words before the current time step, p θ (w t |w <t ) is the large language model given w <t Under the condition of predicting w t The probability of , T is the total time step; The diagnosis module uses the cross entropy loss function L diag , used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label; The loss function constructed by the diagnosis and report generation model is as follows: L total =λ align L c +λ gen L ac +λ diag L diag Among them, λ align ,λ fen ,λ diag They are contrast loss L c , cross entropy loss L ac , cross entropy loss function L diag The corresponding weight hyperparameters.
8. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, characterized in that: In step 1, feature extraction is performed on the brain MRI medical image, specifically: Normalize the cranial MRI medical images and cut them into image blocks of fixed size; Each image block is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. A position code is added to each embedded image block to retain the position information in the original image. The position-encoded image blocks are input into the Transformer encoder, and the features output by the last Transformer layer are used as global visual features.
9. A brain tumor diagnosis and report generation system based on a large language model, characterized in that: Input the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results; The diagnosis and report generation model includes a preprocessing module, an alignment module, a report generation module, a diagnosis module and a loss module; the training process of the diagnosis and report generation model is as follows: The preprocessing module is used to obtain cranial MRI medical images and corresponding imaging reports, and to extract features from the cranial MRI medical images and imaging reports to obtain global visual features and text features; The alignment module is used to align the visual channel and the text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain aligned visual features, and the aligned visual features are spliced in front of the embedding layer of the aligned text features output by the text channel to obtain aligned visual-text features. The report generation module is used to input the aligned visual-text features into the pre-trained large language model to generate a medical report. The diagnosis module is used to input the medical report into the diagnosis module to output the diagnosis result; The loss module is used to construct a loss function to adjust model parameters in the diagnosis and report generation models.
Citation Information
Patent Citations
Ultrasonic diagnosis intelligent interaction system based on liver attribute analysis
CN117333462A
Interrogation interaction method and device based on large language model and related equipment
CN117558439A
Image report generation method, apparatus and device, and computer readable storage medium
CN118136197A
Belt tearing detection method and system based on discrete state selectable space model
CN118333979A
Medical report generation method based on double-layer semantic collaboration and reasoning
CN118506958A
Cited By
Automatic thyroid ultrasound image diagnosis report generation method
CN119920393A
Medical image analysis method based on prior position information
CN119941648A
Vision and language-based X-ray film analysis method and device
CN120411702A
A method and apparatus for analyzing x-ray films based on vision and language
CN120411702B
Report generation method and device based on thinking chain and large model
CN120495213A
