Medical image report generation method based on multi-modal fusion large language model

Abstract

The invention discloses a medical image report generation method based on a multi-modal fusion large language model. The method comprises the following steps: fusing local visual features of a medical image with a structured knowledge label through a multi-channel attention mechanism to generate a customized prompt vector; meanwhile, a context dependence matrix is introduced, a cross-graph convolution attention network is constructed, and evolution path modeling is carried out on the multi-time-point image; the prompts and contextual information are input into a large language model to automatically generate a continuous, professional image report. During training, a semantic consistency regular item loss function is combined, so that the report is kept consistent with the image in the semantic level. The system supports seamless connection and private deployment with the multi-mode PACS, and meets different medical environment requirements. According to the method, the quality and the continuity of automatic reports can be remarkably improved, and the clinical requirements on high-quality reports are met.

Description

Technical Field

[0001] This invention relates to the field of radiology and automatic generation of medical image diagnostic reports, specifically to a method for generating medical image reports based on a multimodal fusion large language model. Background Technology

[0002] With the rapid development of artificial intelligence in the medical field, automatic generation of medical image reports has become an important research direction for assisting clinical diagnosis. Existing research has applied deep learning techniques such as convolutional neural networks (CNNs) and recurrent neural networks (LSTMs) to report generation. For example, Jing et al. combined CNNs and hierarchical LSTMs and introduced an attention mechanism to improve report descriptive capabilities. However, these existing methods typically rely solely on training with a single image, neglecting the temporal information within the patient's longitudinal image sequence. In fact, radiologists often refer to previous images of patients during diagnosis to compare lesion changes, and most existing automatic generation models struggle to capture this temporal evolution information, thus failing to generate professional reports containing key comparative statements. Furthermore, most existing technologies do not fully utilize structured medical knowledge to guide the generation process, resulting in insufficient use of professional terminology and clinical logic in the generated reports. In summary, there is currently a lack of a medical image report generation mechanism that deeply integrates multi-timepoint image evolution modeling with multimodal knowledge prompts, which provides innovative opportunities for this invention. Summary of the Invention

[0003] Technical Objective: To address the shortcomings of existing technologies, this invention discloses a medical image report generation method based on a multimodal fusion large language model, which can automatically generate structured, continuous, and highly professional medical image reports, improve the work efficiency of radiologists, and meet clinical needs.

[0004] Technical solution: To achieve the above technical objectives, the present invention adopts the following technical solution:

[0005] A method for generating medical image reports based on a multimodal fusion large language model, specifically including the following steps:

[0006] The medical image sequence of the same subject is obtained, and local visual features are extracted from each image to obtain a set of image region embedding vectors. At the same time, a set of medical knowledge tags corresponding to the subject or examination site is retrieved from the structured medical knowledge base.

[0007] A dual learnable conditional cue is constructed, which includes a visual cue channel and a structured label cue channel. The outputs of the two channels are input to a multi-channel attention guidance module for weighted fusion to generate a customized cue vector. A dynamic reconstruction strategy is adopted to form a nested hierarchical cue structure based on the input image content.

[0008] A context dependency matrix is ​​established based on image sequences, and its elements represent the correlation between images at different time points. The context dependency matrix is ​​then input into a cross-image convolutional attention network to obtain lesion evolution path features.

[0009] Customized prompt vectors and lesion evolution path features are input into a pre-trained large language model to generate a target medical image report.

[0010] Preferably, the multi-channel attention guidance module calculates attention weights for the visual cue channel output and the structured label cue channel output respectively, and performs weighted fusion of the features of the two channels according to the weights to construct a customized cue vector.

[0011] Preferably, the dynamic reconstruction strategy organizes the customized prompt vector into a hierarchical prompt structure containing anatomical location slots, abnormal feature slots, trend slots, and treatment suggestion slots, and enables or suppresses each slot based on the channel confidence threshold.

[0012] Preferably, the matrix elements of the context-dependent matrix are obtained by linear weighting of two parts: feature similarity based on visual embedding and positional information similarity based on standardized spatial displacement, wherein the weighting coefficients of the two parts are non-negative real numbers and their sum is 1.

[0013] Preferably, the cross-graph convolutional attention network includes at least two layers of graph convolution and self-attention units, distinguishes between two edge types: forward and backward, and normalizes the attention weights for different edge types to extract lesion evolution path features.

[0014] Preferably, during the training phase, the large language model uses the sum of cross-entropy loss and semantic consistency regularization loss as the optimization objective, wherein the semantic consistency regularization loss... Depend on Received, among which This is the cosine similarity difference term between the image region embedding vector and its corresponding text description embedding vector. This represents the absolute deviation between the target location index and the actual output location index for anatomical site-related terms in the report. The Kullback-Leibler divergence between the concept distribution induced by structured medical knowledge tags and the concept distribution of generated text. , , The weight parameters are positive real numbers, satisfying 0.1 ≤ , , ≤10.

[0015] A medical image report generation system based on a multimodal fusion large language model is provided to implement the medical image report generation method based on a multimodal fusion large language model as described above, including:

[0016] The medical image data acquisition and preprocessing module is used to acquire image sequences and output local visual features;

[0017] The medical knowledge tag extraction module is used to output a set of tags related to examinations from a structured medical knowledge base;

[0018] A dual learnable conditional cue generation module, including a visual cue channel and a structured label cue channel;

[0019] A multi-channel attention guidance module is used to weightedly fuse the outputs of two channels to generate a custom cue vector and perform dynamic reconstruction;

[0020] The context dependency building module is used to build the context dependency matrix;

[0021] A cross-graph convolutional attention network module is used to extract lesion evolution path features from the context dependency matrix;

[0022] The large language model module is used to generate medical image reports based on customized cue vectors and evolutionary path features;

[0023] The training and optimization module is used to train the system based on cross-entropy loss and semantic consistency regularization loss.

[0024] Preferably, the dual learnable conditional cue generation module and the multi-channel attention guidance module jointly update the visual cue channel and the structured label cue channel through a shared parameterized gating unit to output a customized cue vector.

[0025] Preferably, the context dependency building module is configured to calculate a context dependency matrix obtained by linear weighting of feature similarity and location information similarity, and input it into a cross-graph convolutional attention network module to output lesion evolution path features.

[0026] Preferably, the training and optimization module is configured to calculate the semantic consistency regularization loss and sum it with the cross-entropy loss in a weighted manner to update the trainable parameters of the dual learnable conditional cue generation module, the multi-channel attention guidance module, the cross-graph convolutional attention network module, and the large language model module.

[0027] Beneficial Effects: The medical image report generation method based on a multimodal fusion large language model provided by this invention has the following beneficial effects:

[0028] 1. This invention designs a multi-channel attention-guided module that uses local visual features and structured medical knowledge tags as dual inputs. These are fused through an attention mechanism and embedded into a customized prompt vector, guiding the language model to generate report text with clinical logic. Unlike directly using existing model prompting techniques, this invention employs a dynamic reconstruction strategy, continuously adjusting the prompt structure based on the input image content to generate prompts with nested levels. For example, disease location and diagnostic information are dynamically injected into the prompts to enhance the model's ability to perceive medical semantics.

[0029] 2. This invention introduces a medical image context dependency matrix to represent the spatial-temporal relationships between multiple image sequences of the same patient. Based on this, a cross-image convolutional attention network is constructed to perform convolution and attention operations on the context dependency matrix, thereby extracting the evolutionary path features of lesions along the time axis. These evolutionary features are explicitly embedded into the report generation process, enabling the generated multi-time-point image reports to coherently describe the trend of lesion changes, improving the continuity and professionalism of the reports.

[0030] 3. To ensure consistency between the generated report and the visual-textual semantic level, this invention constructs a semantic consistency regularization loss function Φ. This loss function integrates three similarity metrics: image region embedding vector similarity ρ, relative positional offset of medical terms Δ, and semantic nesting error with knowledge tags ψ. The weights of each loss metric are adjusted independently through parameters. , , This allows the loss function to align image descriptions with report text, terminology spatial distribution, and semantic consistency of knowledge labels, thereby effectively reducing the semantic gap between images and text during training and improving the accuracy and credibility of reports. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a system block diagram of the present invention;

[0034] Figure 3 A schematic diagram of context dependency matrix and cross-graph convolutional attention network;

[0035] Figure 4 This is a schematic diagram of dual learnable condition prompts and dynamically reconfigurable slots. Detailed Implementation

[0036] The present invention will now be described more clearly and completely by way of a preferred embodiment in conjunction with the accompanying drawings, but this does not limit the invention to the scope of the described embodiment.

[0037] like Figure 1 As shown, a method for generating medical image reports based on a multimodal fusion large language model includes the following steps:

[0038] Step 1: Obtain the medical image sequence of the same subject and extract local visual features from each image to obtain the image region embedding vector set. At the same time, retrieve the set of medical knowledge tags corresponding to the subject or examination site from the structured medical knowledge base.

[0039] This step processes image sequences obtained from one or more examinations of the same subject. After performing geometric and intensity normalization, noise suppression, and contrast enhancement on each image, the system uses a local feature extraction network to generate several region-level features around the anatomical site and suspected abnormalities. This results in an image region embedding vector set composed of multiple region vectors, retaining the pixel coordinates and slice index of each vector for subsequent alignment. Simultaneously, the system retrieves a standardized terminology set corresponding to the examination site, anatomical structure, and common abnormal signs from a structured medical knowledge base. This set serves as the medical knowledge tag set required for subsequent prompt construction, thus providing foundational data for the consistency between visual semantics and knowledge semantics.

[0040] Step 2: Construct dual learnable conditional prompts, which include a visual prompt channel and a structured label prompt channel. The outputs of the two channels are input to a multi-channel attention guidance module for weighted fusion to generate a customized prompt vector. A dynamic reconstruction strategy is then used to form a nested hierarchical prompt structure based on the input image content.

[0041] This step, guided by the conditions for report generation, constructs a dual learnable conditional cue system. The visual cue channel receives the image region embedding vector obtained in step 1 and obtains visual cue candidates through learnable projection and in-channel attention. The structured label cue channel receives a set of medical knowledge labels and obtains label cue candidates through label embedding and relational encoding. The outputs of both channels enter a multi-channel attention guidance module. This module calculates the weights of each channel based on the sample content and performs weighted fusion to generate a customized cue vector to drive text decoding. To adapt to the expression needs of different lesions and locations, the system implements a dynamic reconstruction strategy for the cue vector, organizing the cue content into hierarchical slots such as "anatomical location slot—abnormal feature slot—change trend slot—treatment suggestion slot." The system automatically enables or suppresses corresponding slots based on channel confidence, forming a nested hierarchical cue structure that matches the input image content.

[0042] Step 3: Establish a context dependency matrix based on the image sequence. The matrix elements represent the correlation between images at different time points. Input the context dependency matrix into a cross-image convolutional attention network to obtain the lesion evolution path features.

[0043] This step involves explicit modeling of time-series information. Based on images of the same object at different time points, the system calculates the correlation between any two time points, constructing a context dependency matrix. This matrix considers factors such as visual feature similarity and temporal proximity to reflect the correlation and comparability of lesions over time. Using the context dependency matrix as the edges of a graph and the frame-level features obtained in the previous step as the nodes, the system inputs a cross-graph convolutional attention network. Distinguishing between forward (temporally progressive) and backward (temporally retrospective) edges, it combines graph convolution and self-attention to extract lesion evolution path features that describe patterns such as new onset, progression, stabilization, or absorption, and outputs structured evolutionary information for comparison with previous data.

[0044] Preferably, the matrix elements of the context-dependent matrix are obtained by linear weighting of two parts: feature similarity based on visual embedding and positional information similarity based on standardized spatial displacement, wherein the weighting coefficients of the two parts are non-negative real numbers and their sum is 1.

[0045] Preferably, the cross-graph convolutional attention network includes at least two layers of graph convolution and self-attention units, distinguishes between two edge types: forward and backward, and normalizes the attention weights for different edge types to extract lesion evolution path features.

[0046] Step 4: Input the customized prompt vector and lesion evolution path features into the pre-trained large language model to generate the target medical image report.

[0047] This step aligns the customized cue vector obtained in step 2 with the lesion evolution path features obtained in step 3 into the input embedding space of the pre-trained large language model. The input model is then organized into a conditional sequence according to the following order: system cue (including slot structure) + sample conditions (cue and evolution features) + necessary fields (examination method, location, and comparative description). Under this conditional constraint, the large language model performs autoregressive generation, first outputting standardized examination findings, then outputting diagnostic conclusions consistent with the findings, ensuring a clear mapping relationship between location, signs, and conclusions, and consistent logic. If necessary, synonym merging is performed using a glossary to ensure standardized terminology, ultimately forming a target medical imaging report for clinical review.

[0048] Preferably, during the training phase, the large language model uses the sum of cross-entropy loss and semantic consistency regularization loss as the optimization objective, wherein the semantic consistency regularization loss... Depend on Received, among which This is the cosine similarity difference term between the image region embedding vector and its corresponding text description embedding vector. This represents the absolute deviation between the target location index and the actual output location index for anatomical site-related terms in the report. The Kullback-Leibler divergence between the concept distribution induced by structured medical knowledge tags and the concept distribution of generated text. , , The weight parameters are positive real numbers, satisfying 0.1 ≤ , , ≤10.

[0049] like Figure 2 As shown, a medical image report generation system based on a multimodal fusion large language model is used to implement the medical image report generation method based on a multimodal fusion large language model as described above, including:

[0050] The medical image data acquisition and preprocessing module is used to acquire image sequences and output local visual features;

[0051] This module interfaces with a picture archiving and communication system (PACS) or a local data source to acquire image sequences for the same subject. After de-identification, format parsing, and basic quality control, it performs geometric / intensity normalization, noise suppression, and contrast enhancement on the images. Under anatomical prior constraints, it extracts local visual features, forming an image region embedding vector set composed of multiple region-level vectors. This set is used for subsequent cue construction and temporal modeling, and maintains learnable alignment with the input embedding dimensions of downstream modules to ensure the stability of cross-module fusion.

[0052] The medical knowledge tag extraction module is used to output a set of tags related to examinations from a structured medical knowledge base;

[0053] This module retrieves standardized terms corresponding to examination methods, examination sites, and common signs from a structured medical knowledge base, and outputs a set of medical knowledge tags related to the current examination. The module disambiguates and removes duplicates of synonyms and hypernyms, and assigns learnable embeddings and relational encodings to the tags, providing machine-readable prior knowledge for prompt construction and semantic consistency constraints.

[0054] A dual learnable conditional cue generation module, including a visual cue channel and a structured label cue channel;

[0055] This module constructs two types of learnable conditional cues in parallel. The visual cue channel takes an image region embedding vector set as input, and generates candidate visual cues through projection and in-channel attention aggregation. The structured label cue channel takes a set of medical knowledge labels as input, and generates candidate label cuees through term embedding and relation encoding. These two types of cue candidates serve as the main input to the multi-channel attention guidance module, forming a conditional prior consistent with the sample content. Preferably, this module and the multi-channel attention guidance module jointly update the visual cue channel and the structured label cue channel through a shared parameterized gating unit: the gating unit adaptively adjusts the participation ratio and feature scale of the two channels based on sample confidence and training feedback, thereby stably outputting customized cue vectors.

[0056] A multi-channel attention guidance module is used to weightedly fuse the outputs of two channels to generate a custom cue vector and perform dynamic reconstruction;

[0057] This module calculates cross-channel attention weights for the outputs of the visual cue channel and the structured label cue channel, and then performs weighted fusion of the features of the two channels to generate a customized cue vector. Subsequently, dynamic reconstruction is performed to organize the customized cue vector into hierarchical slots (including anatomical location slots, abnormal feature slots, trend slots, and treatment suggestion slots). Based on the channel confidence, the corresponding slots are enabled or suppressed at the sample level to ensure that the cue structure is consistent with the semantics of the input image, thereby improving the terminology standardization and narrative logic of the subsequent text generation.

[0058] A context dependency building module is used to build a context dependency matrix; preferably, the context dependency building module is configured to calculate a context dependency matrix obtained by linear weighting of feature similarity and location information similarity, and input it into a cross-graph convolutional attention network module to output lesion evolution path features.

[0059] This module is used to establish a context dependency matrix describing the correlation between images of the same object at different time points. Preferably, the elements of the context dependency matrix are obtained by linearly weighting feature similarity and location information similarity with non-negative coefficients, so as to simultaneously reflect appearance changes and spatial correspondence. The context dependency matrix is ​​used as a weighted adjacency input to the cross-graph convolutional attention network module to extract temporal evolution features that reflect patterns such as emergence, progression, stabilization, or absorption.

[0060] A cross-graph convolutional attention network module is used to extract lesion evolution path features from the context dependency matrix;

[0061] This module uses time frames as graph nodes and context dependency matrices as weighted edges to distinguish between forward and backward edge types. Under a multi-head attention mechanism, edge weights are normalized and graph convolutional propagation is performed to obtain lesion evolution path features that reflect the trend of lesion changes over time. These features provide traceable quantitative evidence in comparison with previous representations and are semantically complementary to the cue vectors.

[0062] The large language model module is used to generate medical image reports based on customized cue vectors and evolutionary path features;

[0063] This module receives a customized prompt vector and lesion evolution path features after alignment and mapping, and concatenates them with system prompts and necessary fields (examination method, location, and comparison description) in a predetermined order to form a conditional sequence, driving autoregressive decoding to generate a medical imaging report. The generation order prioritizes the examination findings followed by the diagnostic conclusion, ensuring a clear mapping relationship between location, sign, and conclusion, a coherent temporal description, and consistent wording with knowledge tags.

[0064] The training and optimization module is used to train the system based on cross-entropy loss and semantic consistency regularization loss.

[0065] During the training phase, this module optimizes the sum of cross-entropy loss and semantic consistency regularization loss. The semantic consistency regularization comprehensively measures consistency in three aspects: the similarity difference between image region embeddings and text fragment embeddings, the deviation between the target location and the actual output location of anatomical-related terms, and the difference between the concept distribution induced by medical knowledge tags and the concept distribution of the generated text. During optimization, the trainable parameters of the dual learnable conditional cue generation module, the multi-channel attention guidance module, the cross-graph convolutional attention network module, and the large language model module are jointly updated. In the validation phase, model selection is based on metrics such as terminology consistency and temporal integrity to improve the accuracy and reliability of report generation.

[0066] Example

[0067] The system acquires multimodal medical image sequences (e.g., series of slices from X-ray, CT, or MRI) of patients from PACS. For each image, a pre-trained visual model (e.g., Faster R-CNN) is applied to extract local visual features and location coordinates of candidate lesion regions. Simultaneously, structured medical tags (e.g., disease name, anatomical location, examination site, lesion nature, etc.) corresponding to the image are retrieved from clinical databases or medical knowledge bases. These medical tags can be predefined and learned through dictionaries or knowledge graphs.

[0068] The extracted visual features and medical knowledge tags are input into a multi-channel attention guidance module. This module consists of two parallel channels: a visual cue channel and a structured label cue channel. The visual cue channel applies attention weights to local features of the image, focusing on potential lesion locations; the structured label cue channel integrates structured medical terminology information. Through an attention fusion layer, the feature vectors of both are added or concatenated, and then mapped using a multi-layer perceptron (MLP) to generate a customized cue vector. The customized cue vector does not directly copy existing cue vectors but employs a dynamic reconstruction strategy: it changes the information hierarchy and format contained in the cue vector in real time based on the characteristics of the current image. For example, when multiple lesions are detected, diagnostic terms containing each lesion can be nested as hierarchical cue vectors to guide the large language model in generating multi-segment structured diagnostic descriptions.

[0069] like Figure 4 This invention illustrates the dual learnable conditional prompting and hierarchical dynamic reconstruction mechanism. The upper part is a visual prompting channel, which acquires visual prompt candidates through the projection of region embedding vectors and in-channel attention; the lower part is a structured label prompting channel, which acquires label prompt candidates through term embedding and term relation encoding. The two candidate channels are weighted by a multi-channel attention guidance module and then weighted and fused to obtain a customized prompt vector. Based on this, the dynamic reconstruction unit adaptively opens or expands hierarchical slots such as anatomical location slots, abnormal feature slots, trend slots, and treatment suggestion slots according to the image content, aligning the prompt structure with the salient features of the current case. The output is then connected to a report generation decoder via a large language model interface. This structure ensures that the prompts contain robust visual evidence and carry semantic priors of medical knowledge, thereby improving the accuracy and consistency of the report.

[0070] For multiple images with a time series, construct a context dependency matrix M, where the elements of the matrix M... i,j This represents the similarity or correlation between the i-th and j-th frames. This similarity can be calculated based on feature comparison, location mapping, or embedding similarity based on a pre-trained model. The context dependency matrix can be viewed as a multi-channel two-dimensional matrix that captures the spatial correlation between images at different time points.

[0071] The context-dependent matrix is ​​input into a cross-graph convolutional attention network, which further extracts evolutionary features from the matrix through convolutional layers and attention mechanisms. For example, a one-dimensional convolution can be applied in the time dimension, combined with a self-attention layer, to identify patterns of lesion change over time (such as lesion enlargement, shrinkage, or regression). The feature representation output by the network describes the evolutionary path of the patient's lesions. These evolutionary features are then further projected into contextual cue vectors, which are merged with the aforementioned dual cue vectors and used as auxiliary input for language model generation.

[0072] like Figure 3 The structure of the context dependency matrix M and the cross-graph convolutional attention network is shown. The rectangular grid on the left represents M, and its elements are M... i,j The similarity is obtained by linear weighting of feature similarity and location information similarity, and is used to characterize the correlation between two frames of images with time indices i and j. The right side is a cross-graph convolutional attention network, where nodes represent frame-level features, and edges include both forward and backward categories to simultaneously encode progression / retrospection relationships. Multi-head attention and graph convolution are jointly used to model lesion connections across time and frames. Under the constraint of M, this network outputs lesion evolution path features to characterize temporal evolution patterns such as lesion new development, expansion, stabilization, or absorption, providing interpretable temporal evidence for subsequent report generation.

[0073] The generated custom cue vectors and context evolution vectors are input into a pre-trained Large Language Model (LLM). The LLM uses the cue vectors as conditions to guide the generation of report text. During decoding, the structured information in the custom cue vectors prompts the model to output descriptions that conform to clinical logic, while the context evolution vectors ensure the coherence of information across multiple time points. The final output is a structured medical imaging report text, including technical parameters, descriptions of findings, and diagnostic conclusions.

[0074] During the model training phase, a semantic consistency regularization loss function Φ is introduced to align the semantics of the image and the text. Specifically, the following definition is provided: Where ρ is the similarity (e.g., cosine similarity) between the image region embedding vector and the corresponding text description in the report; Δ is the relative spatial offset of the position of the medical term in the report, used to measure the coherence between the word order in the report and the actual lesion location; and ψ is the nesting error between the semantics of the cue label and the semantics of the generated text, measuring whether the label concept is accurately reflected. Three parameters. , , The weights of each loss term are adjusted individually to ensure that the model optimizes image feature matching, terminology position consistency, and concept coverage during training. Combining the cross-entropy loss and this regularization term for backpropagation significantly improves the professionalism and accuracy of the report.

[0075] Corresponding to the above method, this invention also provides a system embodiment. The system is deployed on a hospital server or a private cloud platform, equipped with a high-performance GPU accelerator, large-capacity storage, and network communication modules, and pre-installed with a large language model and knowledge base that can be updated online. The system defines interface specifications with the hospital's PACS and information systems, conforming to the DICOM / HL7 standard, enabling automatic exchange of image data and patient information. Users can view automatically generated report drafts through the interface and, when necessary, have doctors edit and supplement them to enhance the reliability of the results. This system is adaptable to various medical environments and can expand hardware resources and software functions according to hospital needs.

[0076] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating medical image reports based on a multimodal fusion large language model, characterized in that, Specifically, the following steps are included: The medical image sequence of the same subject is obtained, and local visual features are extracted from each image to obtain a set of image region embedding vectors. At the same time, a set of medical knowledge tags corresponding to the subject or examination site is retrieved from the structured medical knowledge base. A dual learnable conditional cue is constructed, which includes a visual cue channel and a structured label cue channel. The outputs of the two channels are input to a multi-channel attention guidance module for weighted fusion to generate a customized cue vector. A dynamic reconstruction strategy is adopted to form a nested hierarchical cue structure based on the input image content. A context dependency matrix is ​​established based on image sequences, and its elements represent the correlation between images at different time points. The context dependency matrix is ​​then input into a cross-image convolutional attention network to obtain lesion evolution path features. Customized prompt vectors and lesion evolution path features are input into a pre-trained large language model to generate a target medical image report.

2. The method for generating medical image reports based on a multimodal fusion large language model according to claim 1, characterized in that, The multi-channel attention guidance module calculates attention weights for the visual cue channel output and the structured label cue channel output respectively, and then performs weighted fusion of the features of the two channels according to the weights to construct a customized cue vector.

3. The method for generating medical image reports based on a multimodal fusion large language model according to claim 1, characterized in that, The dynamic reconstruction strategy organizes the customized prompt vectors into a hierarchical prompt structure containing slots for anatomical locations, abnormal features, trends, and treatment suggestions, and enables or suppresses each slot based on the channel confidence threshold.

4. The method for generating medical image reports based on a multimodal fusion large language model according to claim 1, characterized in that, The elements of the context-dependent matrix are obtained by linearly weighting two parts: feature similarity based on visual embedding and positional information similarity based on standardized spatial displacement. The weighting coefficients of the two parts are non-negative real numbers and their sum is 1.

5. The method for generating medical image reports based on a multimodal fusion large language model according to claim 1, characterized in that, The cross-graph convolutional attention network contains at least two layers of graph convolution and self-attention units, distinguishes between two edge types: forward and backward, and normalizes the attention weights for different edge types to extract lesion evolution path features.

6. The method for generating medical image reports based on a multimodal fusion large language model according to claim 1, characterized in that, During the training phase, the large language model uses the sum of cross-entropy loss and semantic consistency regularization loss as the optimization objective, where the semantic consistency regularization loss... Depend on Received, among which This is the cosine similarity difference term between the image region embedding vector and its corresponding text description embedding vector. This represents the absolute deviation between the target location index and the actual output location index for anatomical site-related terms in the report. The Kullback-Leibler divergence between the concept distribution induced by structured medical knowledge tags and the concept distribution of generated text. , , The weight parameters are positive real numbers, satisfying 0.1 ≤ , , ≤10.

7. A medical image report generation system based on a multimodal fusion large language model, characterized in that, A method for generating medical image reports based on a multimodal fusion large language model as described in any one of claims 1-6, comprising: The medical image data acquisition and preprocessing module is used to acquire image sequences and output local visual features; The medical knowledge tag extraction module is used to output a set of tags related to examinations from a structured medical knowledge base; A dual learnable conditional cue generation module, including a visual cue channel and a structured label cue channel; A multi-channel attention guidance module is used to weightedly fuse the outputs of two channels to generate a custom cue vector and perform dynamic reconstruction; The context dependency building module is used to build the context dependency matrix; A cross-graph convolutional attention network module is used to extract lesion evolution path features from the context dependency matrix; The large language model module is used to generate medical image reports based on customized cue vectors and evolutionary path features; The training and optimization module is used to train the system based on cross-entropy loss and semantic consistency regularization loss.

8. A medical image report generation system based on a multimodal fusion large language model according to claim 7, characterized in that, The dual learnable conditional cue generation module and the multi-channel attention guidance module jointly update the visual cue channel and the structured label cue channel through a shared parameterized gating unit to output a customized cue vector.

9. A medical image report generation system based on a multimodal fusion large language model according to claim 7, characterized in that, The context dependency building module is configured to calculate a context dependency matrix obtained by linear weighting of feature similarity and location information similarity, and input it into a cross-graph convolutional attention network module to output lesion evolution path features.

10. A medical image report generation system based on a multimodal fusion large language model according to claim 7, characterized in that, The training and optimization module is configured to calculate the semantic consistency regularization loss and sum it with the cross-entropy loss in a weighted manner to update the trainable parameters of the dual learnable conditional cue generation module, the multi-channel attention guidance module, the cross-graph convolutional attention network module, and the large language model module.

Images