Brain tumor diagnosis and report generation method and system based on large language model

Abstract

The present invention discloses a brain tumor diagnosis and report generation method and system based on a large language model, which relates to the field of artificial intelligence technology. The brain MRI medical image to be diagnosed is input into a trained diagnosis and report generation model to generate a medical report and a diagnosis result. The training process of the diagnosis and report generation model is as follows: Step 1, obtain the brain MRI medical image and the corresponding image report, and extract features of the brain MRI medical image and the image report respectively to obtain global visual features and text features; input the global visual features and text features into an alignment module to output aligned visual-text features; input the aligned visual-text features into a report generation module to generate a medical report, and input the medical report into a diagnosis module to output a diagnosis result. The brain tumor diagnosis and report generation method and system improve the efficiency of medical services and improve the accuracy and reliability of diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for brain tumor diagnosis and report generation based on a large language model. Background Art

[0002] Medical imaging plays a crucial role in medical diagnosis, particularly brain MRI (Magnetic Resonance Imaging), which provides extensive information about brain structure and function. Compared to imaging techniques such as CT and X-rays, MRI offers superior soft tissue contrast, higher resolution, and no radiation risk. Therefore, MRI provides strong data support for doctors' diagnoses and report preparation. However, due to significant morphological variations among similar brain tumors and similarities in margin details between different types, it is difficult to determine tumor type based solely on MRI. Even experienced radiologists struggle to accurately diagnose tumor types. Currently, tumor diagnosis in the medical system primarily relies on biopsies of tumor lesions, known as histopathological examinations. When developing a patient's treatment plan, doctors must integrate the patient's medical history, pathology slides, radiological scans, and imaging reports. These decisions often rely on the doctor's experience and are time-consuming.

[0003] Currently, the diagnosis and grading of brain tumors is primarily performed by pathologists who examine tissue sections on glass slides using a microscope. While this approach is widely used in the clinic, it becomes tedious and error-prone for translational and clinical research, which requires processing hundreds or even thousands of tissue specimens. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a brain tumor diagnosis and report generation method and system based on a large language model, which reduces the burden on doctors, improves the efficiency of medical services, and improves the accuracy and reliability of diagnosis.

[0005] The brain tumor diagnosis and report generation method based on a large language model proposed in this invention inputs the brain MRI medical image to be diagnosed into a trained diagnosis and report generation model to generate a medical report and diagnosis results.

[0006] The training process of the diagnosis and report generation model is as follows:

[0007] Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features;

[0008] Step 2: The alignment module sets up a visual channel and a text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain the aligned visual features. The aligned visual features are then spliced ​​in front of the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features.

[0009] Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report, input the medical report into the diagnosis module to output the diagnosis result, and construct a loss function to adjust the model parameters in the diagnosis and report generation model.

[0010] Further, in step 2, the visual channel includes a plurality of visual attention blocks arranged in series, each visual attention block includes a visual self-attention layer, a visual cross-attention layer, and a visual feedforward network layer connected in sequence, the input of the visual self-attention layer of the current visual attention block is connected to the output of the visual feedforward network layer of the previous visual attention block, and the output of the visual feedforward network layer of the current visual attention block is connected to the input of the visual self-attention layer of the next visual attention block;

[0011] The text channel includes multiple text attention blocks arranged in series, each text attention block includes a text self-attention layer and a text feedforward network layer connected in sequence, the input of the text self-attention layer of the current text attention block is connected to the output of the text feedforward network layer of the previous text attention block, and the output of the text feedforward network layer of the current text attention block is connected to the input of the text self-attention layer of the next text attention block.

[0012] Furthermore, in step 2, weights are shared between the visual channel and the text channel, specifically: the same number of visual attention blocks and text attention blocks are set, and the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block.

[0013] Furthermore, the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block, specifically:

[0014] In the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated, and the query vector is shared with the text self-attention layer through the visual self-attention layer and transmitted to the visual channel to realize interaction with the text in the visual channel.

[0015] Furthermore, in step three, the medical report is input into the diagnosis module to output the diagnosis result, specifically:

[0016] The medical report is converted into a text feature vector through a text preprocessing layer;

[0017] The context information in the text feature vector is captured through a bidirectional long short-term memory network, and the output is a hidden state vector for each time step;

[0018] Perform a fully connected mapping on the hidden state vector of the last time step output by the bidirectional long short-term memory network to generate a high-dimensional feature vector;

[0019] Nonlinearity is added through the ReLU activation function, and then the high-dimensional feature vector is mapped to a vector of the number of categories through another fully connected layer. Finally, the vector of the number of categories is converted into a probability distribution through the Softmax activation function to predict the tumor category and generate a diagnostic result.

[0020] Furthermore, a loss function is constructed to adjust the model parameters in the diagnosis and report generation model. Specifically, in the alignment module, it is:

[0021] The contrast loss in the alignment module is constructed based on the InfoNCE loss function. The contrast loss acts on the aligned visual-text features output by the alignment module and the text semantic representation of the medical report output by the large language model. The specific formula is as follows:

[0022] ;

[0023] in, represents the contrast loss, is the number of samples in a batch, including positive sample pairs and negative sample pairs, For the The aligned visual features corresponding to the samples are is the text semantic representation of the medical report corresponding to the positive sample, is the temperature coefficient, For the batch The text semantic representation of the medical reports of samples.

[0024] Furthermore, by constructing a loss function to adjust the model parameters in the diagnosis and report generation model, the large language model in the report generation module adopts the first cross entropy loss function , specifically:

[0025] ;

[0026] in, is the current time step The target word, are all words before the current time step, For a large language model given Prediction under the conditions The probability of is the total time step;

[0027] The diagnosis module uses the second cross entropy loss function , used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label;

[0028] The loss function constructed by the diagnosis and report generation model is as follows:

[0029] ;

[0030] in, They are contrast loss , the first cross entropy loss function , the second cross entropy loss function The corresponding weight hyperparameters.

[0031] Furthermore, in step 1, feature extraction is performed on the brain MRI medical image, specifically:

[0032] Normalize the brain MRI medical images and cut them into image blocks of fixed size;

[0033] Each image block is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. A position code is added to each embedded image block to retain the position information in the original image.

[0034] The position-encoded image blocks are input into the Transformer encoder, and the features output by the last Transformer layer are used as global visual features.

[0035] A brain tumor diagnosis and report generation system based on a large language model inputs the brain MRI medical image to be diagnosed into a trained diagnosis and report generation model to generate a medical report and diagnosis results;

[0036] The diagnosis and report generation model includes a preprocessing module, an alignment module, a report generation module, a diagnosis module, and a loss module. The training process of the diagnosis and report generation model is as follows:

[0037] The preprocessing module is used to obtain cranial MRI medical images and corresponding imaging reports, and extract features from the cranial MRI medical images and imaging reports to obtain global visual features and text features;

[0038] The alignment module is used to align the visual channel and the text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through the fully connected layer to obtain the aligned visual features. The aligned visual features are then spliced ​​in front of the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features.

[0039] The report generation module is used to input the aligned visual-text features into the pre-trained large language model to generate a medical report.

[0040] The diagnosis module is used to input the medical report into the diagnosis module to output the diagnosis result;

[0041] The loss module is used to construct loss functions to adjust model parameters in diagnosis and report generation models.

[0042] The advantages of the large language model-based brain tumor diagnosis and report generation method and system provided by the present invention are that they utilize a deep learning visual encoder to analyze tumor image information and simultaneously leverage the power of a large language model to generate medical reports. This embodiment leverages the power of a multimodal large model (diagnosis and report generation model) to generate medical reports and uses medical reports to assist in tumor diagnosis. Key diagnostic information can be quickly and accurately extracted and structured medical reports generated. This significantly reduces the burden on physicians and improves the efficiency of medical services while also reducing the likelihood of human error, thereby enhancing the accuracy and reliability of diagnoses. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the process of the present invention;

[0044] Figure 2 Flowchart for training models for diagnosis and report generation;

[0045] Figure 3 Schematic diagram of the model structure for diagnosis and reporting. DETAILED DESCRIPTION

[0046] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0047] To address the technical issues mentioned in the background technology, we began exploring the integration of natural language processing (NLP) technology with medical image analysis. This approach led to the development of a multimodal diagnostic report generation model capable of automatically generating medical diagnostic reports. The significance of automated medical report generation technology lies in its significant contribution to improving the efficiency and accuracy of medical diagnoses and patient treatment outcomes. By leveraging deep learning and machine learning algorithms to analyze massive amounts of medical imaging data, this technology can quickly and accurately extract key diagnostic information and generate structured medical reports. This not only significantly reduces the burden on physicians and improves the efficiency of medical services, but also reduces the potential for human error, thereby improving the accuracy and reliability of diagnoses. The details are as follows.

[0048] like Figures 1 to 3 As shown, the brain tumor diagnosis and report generation method based on the large language model proposed by the present invention inputs the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results;

[0049] The training process of the diagnosis and report generation model is as follows:

[0050] Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features;

[0051] The ViT (Vision Transformer) model is used to extract features from brain MRI medical images. Through a multi-head self-attention mechanism, the ViT model captures long-range dependencies between image patches, thereby extracting global visual features. These features are processed by layers of encoders to generate a final global visual feature.

[0052] Specifically, the ViT model first preprocesses MRI images, including normalization and slicing them into fixed-size patches. For example, a 224x224 MRI image is sliced ​​into 16x16 patches, for a total of 14x14 = 196 patches. Each patch is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. After embedding, a positional encoding is added to each patch to preserve its position in the original image. These patches are then fed into multiple Transformer encoder layers. Each encoder layer consists of a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer establishes long-range dependencies between patches, while the feedforward neural network further extracts deep-level features. Finally, the feature representation output from the last Transformer encoder layer is used as the global visual feature and is fed into the subsequent alignment module. This allows the ViT model to fully leverage the advantages of the Transformer architecture to extract high-quality visual features from MRI images, improving the accuracy and robustness of medical image analysis tasks.

[0053] Step 2: The alignment module sets up a visual channel and a text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain the aligned visual features. The aligned visual features are then spliced ​​in front of the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features.

[0054] The alignment module, which bridges the gap between global visual features learned from the ViT model and the text embedding feature space, is a bridge. It consists of two channels: the visual channel and the text channel. The visual attention blocks and text attention blocks have the same number of blocks. The visual self-attention layer in each visual attention block shares weights with the text self-attention layer in the corresponding text attention block. Specifically, in the text channel, the input to the current text self-attention layer is the output of the previous text attention block and the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated and fed into the visual channel through the shared visual and text self-attention layers, enabling interaction with text in the visual channel.

[0055] Specifically, the visual channel extracts visual features by interacting with the visual encoder. The input of the visual channel is a learnable query vector. These query vectors interact with themselves through the visual self-attention layer SA and interact with the visual features through the visual cross-attention layer CA to learn which tokens to extract from the visual features. They can also interact with the text through the shared self-attention layer.

[0056] The other channel is the text channel. Its main function is to interact with visual features through a text self-attention layer shared with the visual channel. The weight sharing approach here is to concatenate the learnable query vector of the visual channel and the output of the text encoder and input them into a self-attention layer. A mask is used to control the interaction between the visual and text channels. After the text self-attention layer, the query vector and text vector are separated for subsequent operations. These two channels then need to generate text features from visual features. The output of the visual channel is connected to a fully connected layer to generate aligned visual features with the same dimensions as the embedding of the large language model. The aligned visual features are then concatenated before the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features.

[0057] Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report, input the medical report into the diagnosis module to output the diagnosis result, and construct a loss function to adjust the model parameters in the diagnosis and report generation model.

[0058] The report generation module uses a large language model to receive aligned visual-text features as input and outputs report text corresponding to the image. The large language model's network structure consists of multiple layers of stacked Transformers, which effectively model the contextual relationships of text. Large language models are typically pre-trained on large text corpora to learn universal language representations. This pre-trained knowledge can be transferred to specific domain applications, such as medicine. By further fine-tuning on domain-specific datasets, the large language model can recognize specific medical terms and representations and better understand the expertise related to brain tumor diagnosis. Since the corpus provided in the dataset is all in Chinese, this embodiment can use the Chinese large language model ChatGLM for training and fine-tuning. ChatGLM is a large language model based on the Transformer architecture, pre-trained on a large Chinese text corpus, and has excellent Chinese language understanding and generation capabilities. By using this model, its advantages in natural language processing can be fully utilized to generate high-quality Chinese reports for MRI images. In this way, the report generation module can convert complex MRI image data into structured, easy-to-understand Chinese medical reports, helping doctors make diagnostic decisions more quickly and improving clinical efficiency and diagnostic quality.

[0059] The tumor diagnosis module is tasked with converting the medical report text output by the report generation module into a tumor category diagnosis. First, the text preprocessing layer converts the imaging report text into a text feature vector. This step uses a pretrained word embedding model (such as Word2Vec or GloVe) to convert each word into a fixed-dimensional vector representation, outputting a two-dimensional matrix. Next, a bidirectional long short-term memory (BiLSTM) network processes the text features, capturing contextual information within the text sequence and outputting a hidden state vector for each time step. The fully connected layer then performs a fully connected mapping on the hidden state of the last time step output by the BiLSTM layer, generating a high-dimensional feature vector. Nonlinearity is then added using the ReLU activation function, followed by another fully connected layer that maps the high-dimensional feature vector to a vector representing the number of categories. Finally, a Softmax activation function is used to convert this into a probability distribution for predicting the tumor category and generating a diagnosis.

[0060] Three different optimization objectives are used for the alignment module, report generation module, and diagnosis module:

[0061] (1) Alignment module: The contrast loss function in this module adopts the InfoNCE loss function. The contrast loss function acts on the aligned visual-text features output by the alignment module and the text semantic representation corresponding to the medical report output by the large language model. It is defined as follows:

[0062] ;

[0063] in, represents the contrast loss, is the number of samples in a batch, including positive sample pairs and negative sample pairs, For the The aligned visual features corresponding to the samples are is the text semantic representation of the medical report corresponding to the positive sample, is the temperature coefficient, For the batch The text semantic representation of the medical reports of samples.

[0064] In this example scenario, the InfoNCE loss is used to promote consistency between aligned visual-context features and textual semantic representations. By maximizing the similarity between textual representations that match the image and minimizing the similarity with other unrelated textual representations, this loss function helps learn diagnosis and report generation models that correctly align visual images and textual descriptions.

[0065] (2) Report Generation Module: In large language models, the autoregressive loss function is used to train the model to predict the next word or character in sequence. The main goal is to maximize the probability of the next word given the conditional probability of the previous part of the sequence. This embodiment uses the first cross-entropy loss function (Cross-Entropy Loss):

[0066] ;

[0067] in, is the current time step The target word, are all words before the current time step, For a large language model given Prediction under the conditions The probability of is the total time step. Through the loss function ,The diagnosis and report generation model is trained to predict the probability of the next word to guide the generation of medical reports.

[0068] (3) Tumor diagnosis module: In order to optimize the performance of the tumor diagnosis module, the second cross-entropy loss function is used. The second cross-entropy loss function is used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label:

[0069] ;

[0070] in, is the second cross entropy loss function, For the The true labels of samples, For the The samples are diagnosed and the probability distribution of the model output is generated.

[0071] Therefore, the loss function constructed by the diagnosis and report generation model in this embodiment is specifically as follows:

[0072] ;

[0073] in, They are contrast loss , the first cross entropy loss function , the second cross entropy loss function The corresponding weight hyperparameters are used to balance the losses of different modules to the total loss The contribution of By optimizing the comprehensive loss function, the performance of the alignment module, report generation module, and tumor diagnosis module can be jointly improved, ultimately achieving efficient and accurate tumor diagnosis.

[0074] Through steps one through three, this embodiment leverages deep learning visual encoders for tumor image information and the capabilities of large language models to generate medical reports. This embodiment leverages the capabilities of a large multimodal model (diagnosis and report generation model) to generate medical reports and use medical reports to assist in tumor diagnosis. The proposed method can facilitate clinicians' diagnosis and treatment plan development.

[0075] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A brain tumor diagnosis and report generation method based on a large language model, characterized in that: Input the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results; The training process of the diagnosis and report generation model is as follows: Step 1: Obtain cranial MRI medical images and corresponding imaging reports, and perform feature extraction on the cranial MRI medical images and imaging reports to obtain global visual features and text features; Step 2: The alignment module is equipped with a visual channel and a text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain the aligned visual features. The aligned visual features are then spliced ​​in front of the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features. Step 3: Input the aligned visual-text features into the pre-trained large language model to generate a medical report. The medical report is then input into the diagnosis module to output the diagnosis result. A loss function is then constructed to adjust the model parameters in the diagnosis and report generation models. In step 2, the visual channel includes a plurality of visual attention blocks arranged in series, each visual attention block includes a visual self-attention layer, a visual cross-attention layer, and a visual feedforward network layer connected in sequence, the input of the visual self-attention layer of the current visual attention block is connected to the output of the visual feedforward network layer of the previous visual attention block, and the output of the visual feedforward network layer of the current visual attention block is connected to the input of the visual self-attention layer of the next visual attention block; The text channel includes a plurality of text attention blocks arranged in series, each text attention block includes a text self-attention layer and a text feedforward network layer connected in sequence, the input of the text self-attention layer of the current text attention block is connected to the output of the text feedforward network layer of the previous text attention block, and the output of the text feedforward network layer of the current text attention block is connected to the input of the text self-attention layer of the next text attention block; By sharing weights between the visual channel and the text channel, specifically: the same number of visual attention blocks and text attention blocks are set, and the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block, specifically: In the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated, and the query vector is shared with the text self-attention layer through the visual self-attention layer and transmitted to the visual channel to realize interaction with the text in the visual channel.

2. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, wherein: In step three, the medical report is input into the diagnosis module to output the diagnosis results, specifically: The medical report is converted into a text feature vector through a text preprocessing layer; The context information in the text feature vector is captured through a bidirectional long short-term memory network, and the output is a hidden state vector for each time step; Perform a fully connected mapping on the hidden state vector of the last time step output by the bidirectional long short-term memory network to generate a high-dimensional feature vector; Nonlinearity is added through the ReLU activation function, and then the high-dimensional feature vector is mapped to a vector of the number of categories through another fully connected layer. Finally, the vector of the number of categories is converted into a probability distribution through the Softmax activation function to predict the tumor category and generate a diagnostic result.

3. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, wherein: By constructing a loss function to adjust the model parameters in the diagnosis and report generation model, specifically in the alignment module: The contrast loss in the alignment module is constructed based on the InfoNCE loss function. The contrast loss acts on the aligned visual-text features output by the alignment module and the text semantic representation of the medical report output by the large language model. The specific formula is as follows: in, represents the contrast loss, is the number of samples in a batch, including positive sample pairs and negative sample pairs, For the The aligned visual features corresponding to the samples are is the text semantic representation of the medical report corresponding to the positive sample, is the temperature coefficient, For the batch The text semantic representation of the medical reports of samples.

4. The method for diagnosing and reporting brain tumors based on a large language model according to claim 3, wherein: By constructing a loss function to adjust the model parameters in the diagnosis and report generation model, the large language model in the report generation module adopts the first cross entropy loss function , specifically: in, is the current time step The target word, are all words before the current time step, For a large language model given Prediction under the conditions The probability of is the total time step; The diagnosis module uses the second cross entropy loss function , used to evaluate the difference between the output probability distribution of the diagnosis and report generation model and the true label; The loss function constructed by the diagnosis and report generation model is as follows: in, They are contrast loss , the first cross entropy loss function , the second cross entropy loss function The corresponding weight hyperparameters.

5. The method for diagnosing and reporting brain tumors based on a large language model according to claim 1, wherein: In step 1, feature extraction is performed on the brain MRI medical image, specifically: Normalize the brain MRI medical images and cut them into image blocks of fixed size; Each image block is flattened into a 256-dimensional vector and embedded into a 768-dimensional high-dimensional space through a linear transformation layer. A position code is added to each embedded image block to retain the position information in the original image. The position-encoded image blocks are input into the Transformer encoder, and the features output by the last Transformer layer are used as global visual features.

6. A brain tumor diagnosis and report generation system based on a large language model, characterized by: Input the brain MRI medical image to be diagnosed into the trained diagnosis and report generation model to generate a medical report and diagnosis results; The diagnosis and report generation model includes a preprocessing module, an alignment module, a report generation module, a diagnosis module, and a loss module. The training process of the diagnosis and report generation model is as follows: The preprocessing module is used to obtain cranial MRI medical images and corresponding imaging reports, and extract features from the cranial MRI medical images and imaging reports to obtain global visual features and text features; The alignment module is equipped with a visual channel and a text channel. The global visual features are input into the visual channel as a learnable query vector, and the text features are input into the text channel. By sharing weights between the visual channel and the text channel, the features output by the visual channel are passed through a fully connected layer to obtain the aligned visual features. The aligned visual features are then spliced ​​before the embedding layer of the aligned text features output by the text channel to obtain the aligned visual-text features. The report generation module is used to input the aligned visual-text features into the pre-trained large language model to generate a medical report; The diagnosis module is used to input the medical report into the diagnosis module to output the diagnosis result; The loss module is used to construct the loss function to adjust the model parameters in the diagnosis and report generation model; In the alignment module, the visual channel includes a plurality of visual attention blocks arranged in series, each visual attention block includes a visual self-attention layer, a visual cross-attention layer, and a visual feedforward network layer connected in sequence, the input of the visual self-attention layer of the current visual attention block is connected to the output of the visual feedforward network layer of the previous visual attention block, and the output of the visual feedforward network layer of the current visual attention block is connected to the input of the visual self-attention layer of the next visual attention block; The text channel includes a plurality of text attention blocks arranged in series, each text attention block includes a text self-attention layer and a text feedforward network layer connected in sequence, the input of the text self-attention layer of the current text attention block is connected to the output of the text feedforward network layer of the previous text attention block, and the output of the text feedforward network layer of the current text attention block is connected to the input of the text self-attention layer of the next text attention block; By sharing weights between the visual channel and the text channel, specifically: the same number of visual attention blocks and text attention blocks are set, and the visual self-attention layer in each visual attention block shares the weight setting with the text self-attention layer in the corresponding text attention block, specifically: In the text channel, the input of the current text self-attention layer is the output of the previous text attention block and the output of the previous visual attention block. The query vector and text vector output by the current text self-attention layer are separated, and the query vector is shared with the text self-attention layer through the visual self-attention layer and transmitted to the visual channel to realize interaction with the text in the visual channel.

Images