Multi-modal large language model for generating hepatocellular carcinoma key pathological diagnosis report

Abstract

The invention provides a multi-modal large language model for generating a hepatocellular carcinoma key pathological diagnosis report, a framework main body is a visual coding module, and a multi-modal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism and a structured output decoding layer are also introduced. The visual coding module is constructed on the basis of a Swin Transform architecture, visual pre-training is completed on hepatocellular carcinoma MRI data, and after a task specific classification head is stripped, a trunk feature extraction network is reserved to serve as an image modal representation encoder. The multi-modal feature alignment module guides the model to learn a cross-modal semantic mapping relation between a hepatocellular carcinoma MRI image and a key pathological diagnosis report language, image modal input is a visual feature sequence, and text modal output is a structured description text; and the structured output decoding layer generates six types of liver cancer focus attributes. According to the method, the pre-operative multi-parameter and multi-stage enhanced MRI image is utilized, and the open-source large model is finely adjusted to generate a matched liver cancer postoperative pathology report.

Description

TECHNICAL FIELD

[0001] The present application relates to a multimodal large language model, in particular to a multimodal large language model for generating a key pathological diagnosis report of hepatocellular carcinoma. BACKGROUND

[0002] Hepatocellular carcinoma (HCC) is one of the most common malignant tumors in China, ranking second in the causes of death from malignant tumors, causing huge health and economic losses[1]. Surgical resection is the preferred treatment for liver cancer in China, but the overall recurrence rate 5 years after surgery is as high as 70%[2]. Neoadjuvant therapy reduces the risk of postoperative recurrence and metastasis in high-risk patients by reducing tumor volume, improving radical resection rate, and eliminating potential micro-metastases or subclinical lesions through preoperative systemic or combined local treatment. According to the latest Chinese expert consensus, patients with resectable CNLC Ib / IIa stage and resectable CNLC IIb / IIIa stage with recurrence risk factors are recommended to receive neoadjuvant therapy. Preoperative recurrence risk factors include incomplete tumor capsule, tumor adjacent to blood vessels, and alpha-fetoprotein > 400 μg / L.

[0003] However, the above-mentioned high-risk factors for recurrence are based on preoperative imaging reports and tumor markers, and do not include pathological indicators, making it difficult to accurately assess the risk of recurrence. Microvascular invasion (MVI) causes hematogenous metastasis and is the main cause of early postoperative recurrence and metastasis of liver cancer. In addition, histological type, degree of differentiation, and satellite lesions are also closely related to liver cancer recurrence. Due to the breakthrough progress of immunotherapy combined with targeted therapy in neoadjuvant therapy, the main pathological remission rate is as high as 27.3%, and the characteristics of the immune microenvironment related to its efficacy (PD-L1 expression, lymphocyte infiltration, tertiary lymphoid structure formation type and degree) are also of concern in clinical practice. The above-mentioned key pathological indicators are dependent on postoperative pathological diagnosis, and how to obtain a pathological diagnosis report preoperatively has become a difficult problem to be solved.

[0004] Multimodal large language models (MLLMs) are generative artificial intelligence technologies based on the expansion of large language models (LLMs), which can receive, understand, and reason about multiple modalities of data and generate text or other modalities of output. However, there is currently no multimodal large language model on the market that can generate a matching postoperative pathological report for liver cancer using preoperative multi-parameter, multi-phase enhanced MRI images. SUMMARY

[0005] The technical problem to be solved by this invention is to provide a multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma. It utilizes preoperative multi-parameter, multi-phase enhanced MRI images and fine-tunes the latest open-source large model to generate matching postoperative pathological reports for hepatocellular carcinoma.

[0006] This invention provides a multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma. The main architecture of the multimodal large language model is a visual encoding module, and it also incorporates a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer.

[0007] The visual encoding module is built on the Swing Transformer architecture and pre-trained on hepatocellular carcinoma MRI data. Then, the task-specific classification head is stripped off, and only the backbone feature extraction network is retained as the representation encoder of the image modality, outputting a visual feature sequence.

[0008] The multimodal feature alignment module guides the multimodal large language model to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and the language of key pathological diagnostic reports for hepatocellular carcinoma. The image modal input is the visual feature sequence output by the visual encoding module, and the text modal output is the structured descriptive text of the key pathological diagnostic report for hepatocellular carcinoma. The visual feature sequence and the structured descriptive text constitute a one-to-one pairing of training samples, and the training sample format is as follows:

[0009] Input: [The visual feature sequence output by the visual encoding module];

[0010] Output: [Structured description text of key pathological diagnosis report for hepatocellular carcinoma];

[0011] The multi-head low-rank attention mechanism is used to solve the memory and computational efficiency problems of long sequence modeling, while accurately capturing cross-modal medical semantic associations;

[0012] The enhanced medical MoE mechanism provides routing for liver cancer specialists;

[0013] The structured output decoding layer is used to generate six types of liver cancer lesion attributes.

[0014] Furthermore, the visual pre-training uses the MVI image slice grading of hepatocellular carcinoma pathology reports as a weak supervision signal to carry out global semantic discrimination learning; by training the Swin Transformer model, the Swin Transformer model acquires strong semantic recognition ability and can distinguish the risk level of lesions. The feature vectors extracted after training are used to generalize to downstream multi-task objectives.

[0015] The image-label mapping strategy used in the visual pre-training is as follows: each MVI image slice of a hepatocellular carcinoma pathology report is assigned a global label according to the microvascular invasion of the patient to which it belongs, and a learning pathway from local visual patterns to global diagnostic labels is constructed.

[0016] After removing the task-specific classification header, the loss function used is the three-class cross-entropy loss function, with the objective being:

[0017]

[0018] in, It is the value of the cross-entropy loss function for each category;

[0019] N represents the batch size of each category of samples in the visual pre-training;

[0020] y i This represents the true label of each class sample i, where the true label is either 0 or 1;

[0021] p i This represents the predicted label for each category sample i.

[0022] Furthermore, after completing visual pre-training, the visual encoding module also undergoes fine-tuning and optimization training to improve the model's transferability and generalization performance. This fine-tuning and optimization training includes:

[0023] Transfer learning initialization: The Swin Transformer model loads ImageNet pre-trained weights, freezes the underlying encoding modules, and only fine-tunes the mid-to-high-level semantic abstraction layers and task headers;

[0024] Layer-by-layer unfreezing strategy: In the early stage of training, only high-level feature modules are optimized, and then phased unfreezing is used to achieve gradient propagation from local optimization to global fine-tuning;

[0025] Activation and regularization control: Transformer uses the default GELU activation function; the MLP classification head uses Softmax activation; weight decay is introduced to control model complexity; overfitting is suppressed to improve model generalization ability;

[0026] Learning rate scheduling mechanism: A cosine annealing adjustment strategy is adopted to improve training stability. The expression is:

[0027]

[0028] Where, η max η represents the initial learning rate. max =1×10 -3 ;

[0029] η min Represents the minimum learning rate ηmin =0;

[0030] T cur Indicates the current iteration step;

[0031] T represents the total period, and T = 96.

[0032] Furthermore, the multimodal feature alignment module introduces a linear alignment layer to map the visual embedding dimension to the text embedding dimension, thereby achieving dimensional unification through a fully connected layer. The expression is as follows:

[0033]

[0034] In the formula, E t Represents the target feature vector;

[0035] E v This represents the input feature vector;

[0036] "where" refers to further explanation of "W";

[0037] Let W represent a learnable weight matrix, and D... t D represents the text embedding dimension; v Indicates the visual embedding dimension.

[0038] Furthermore, the construction process of the multimodal feature alignment module includes:

[0039] Image encoding: Each hepatocellular carcinoma MRI image is input into the visual encoding module for encoding, and the output is an MRI image embedding vector. This vector is then integrated into a global image representation vector z through a fully connected mapping. I ;

[0040] Text Encoding: Standardized microvascular invasion is structurally described using a text encoder, and then one-hot encoded by a text embedding module to generate a global text representation vector z. T ;

[0041] Positive sample construction: Each pair of image-text paired samples (z I ,z T (From the same patient;)

[0042] Negative sample construction: Constructing in-batch negative samples

[0043] Contrastive learning loss function design: InfoNCE loss is used, which performs softmax normalization on the similarity between each image sample and all text samples, and then maximizes the probability of positive sample pairs. The overall loss function expression is:

[0044]

[0045] In the formula, The encoded features representing the modality of the i-th image and the i-th text;

[0046] The encoded features representing the modality of the i-th image and the j-th text;

[0047] cos(·,·) represents the cosine similarity;

[0048] τ represents the temperature hyperparameter;

[0049] N represents the amount of data used in each training iteration of the contrastive learning module, and its size is consistent with the global batch size.

[0050] Training details design: When training the contrastive learning module, the remaining parameters are kept frozen, the backbone of the visual encoding module and the text encoder are frozen, and only the image-text modality alignment layer is optimized.

[0051] Furthermore, after the architecture of the multimodal large language model is implemented, it enters the instruction fine-tuning stage. The instruction fine-tuning stage aims to further optimize the response capability of the multimodal large language model in the hepatocellular carcinoma MRI image and text generation scenario through structured medical instruction tasks, so as to automatically complete the structured medical report writing task based on image features.

[0052] The instruction fine-tuning stage is achieved by constructing image-text-instruction-response data for training. The training objective is to enhance the model’s language generation ability, accuracy of medical terminology, and consistency of structural expression based on image understanding.

[0053] The instruction fine-tuning stage uses "visual coding features + natural language task instructions" as input and a structured diagnostic report as the output target.

[0054] The input includes image embeddings and task prompts; the image embeddings are visual feature sequences output by the visual encoding module, which serve as deep semantic inputs; the task prompts adopt a standard system prompting structure to prompt the multimodal large language model to output structured medical descriptions.

[0055] Output a structured key pathological diagnosis report for hepatocellular carcinoma, including six types of hepatocellular carcinoma lesion attributes;

[0056] The instruction fine-tuning stage employs a joint loss function, with the main task being the cross-entropy loss function, used to maximize the likelihood of the target text token; knowledge preservation uses the KL divergence loss function, used to constrain the current model output distribution from deviating from the original pre-trained knowledge representation;

[0057] The joint objective function is as follows:

[0058]

[0059] in, Cross-entropy loss is the loss function for the task.

[0060] KL divergence loss is the loss function for knowledge preservation.

[0061] λ is a hyperparameter that adjusts the weight of the regularization term and is used to control the degree of influence of knowledge retention loss on the total loss;

[0062] The joint objective function is a weighted sum of cross-entropy loss and KL divergence loss, used to simultaneously consider generation accuracy and knowledge preservation during fine-tuning.

[0063] The present invention has the following technical effects or advantages:

[0064] The multimodal large language model architecture of this invention is mainly based on the DeepSeek structure, employing an improved multi-layer Transformer backbone as the visual encoding module for visual feature extraction. It also introduces a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer to achieve automatic generation of standardized diagnostic reports from hepatocellular carcinoma MRI images. During the training phase, a Supervised Fine-Tuning (SFT) strategy is used to construct multimodal training samples with image features as input and structured diagnostic text as output.

[0065] The visual encoding module of this invention employs a multi-layer Transformer backbone and performs visual pre-training on hepatocellular carcinoma MRI data. Subsequently, the task-specific classification head is stripped, and only the backbone feature extraction network is retained as the image modality representation encoder, which is used to input to the downstream open-source multimodal large language model (such as the DeepSeek language model) to prepare for cross-modal joint modeling.

[0066] This invention introduces a multimodal alignment mechanism based on contrastive learning through a multimodal feature alignment module, guiding the model to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and diagnostic language, thereby enhancing the semantic consistency between visual and linguistic modalities in the multimodal large language model of this invention.

[0067] This invention is based on a large-sample cohort of liver cancer surgery and uses preoperative multi-parameter, multi-phase enhanced MRI images to generate matching postoperative pathology reports for liver cancer, which has significant guiding significance for clinical treatment decisions.

[0068] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Detailed Implementation

[0069] This application provides a multimodal large language model for generating key pathological diagnostic reports for hepatocellular carcinoma. Based on a large sample cohort of liver cancer surgeries, it utilizes preoperative multi-parameter, multi-phase enhanced MRI images and fine-tunes the latest open-source large model to generate matching postoperative pathological reports for liver cancer.

[0070] The overall concept of the technical solution in this application is as follows: The multimodal large language model architecture of this invention mainly references the DeepSeek structure, adopts an improved multi-layer Transformer backbone as the visual encoding module for visual feature extraction, and introduces a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer; to achieve the automatic generation of standardized diagnostic reports from hepatocellular carcinoma MRI images. During the training phase, a Supervised Fine-Tuning (SFT) strategy is used to construct multimodal training samples with image features as input and structured diagnostic text as output, thereby filling the technical gap in the automatic generation of standardized diagnostic reports from hepatocellular carcinoma MRI images.

[0071] Example

[0072] This application provides a multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma.

[0073] I. Architecture of a Multimodal Large Language Model

[0074] The main body of the multimodal large language model of this invention references the DeepSeek structure. The main architecture adopts an improved multi-layer Transformer backbone to implement a visual encoding module, and introduces a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer.

[0075] The visual encoding module is used to distinguish the risk level of lesions, enabling the Swin Transformer to obtain strong semantic recognition capabilities and output a visual feature sequence (i.e., a 768-dimensional space-semantic feature sequence), which is then generalized to downstream multi-task objectives.

[0076] The multimodal feature alignment module is used to guide the multimodal large language model of the present invention to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and diagnostic language;

[0077] The multi-head low-rank attention mechanism is used to solve the memory and computational efficiency problems of long sequence modeling, while accurately capturing cross-modal medical semantic associations;

[0078] The enhanced medical MoE mechanism provides routing for liver cancer specialists;

[0079] The structured output decoding layer is used to generate six types of liver cancer lesion attributes.

[0080] The construction of the multimodal large language model of this invention mainly includes two key stages: the construction of the visual encoding module and the construction of the multimodal feature alignment module. After obtaining the visual encoding features of the image modality and the structured text embedding, a graph-text joint generation architecture with high expressiveness and inference efficiency is constructed.

[0081] 1. Construction of the visual encoding module

[0082] The visual encoding module is built on the Swing Transformer architecture and pre-trained on hepatocellular carcinoma MRI data. Then, the task-specific classification head is stripped off, and only the backbone feature extraction network is retained as the image modality representation encoder. The output visual feature sequence (i.e., 768-dimensional spatial-semantic feature sequence) can be generalized to downstream multi-task objectives (such as description generation, structured reporting, etc.).

[0083] The Swin Transformer architecture employs a layered and modular visual Transformer structure, with its key mechanism being Shifted Window MSA (Shifted Window Multi-Head Self-Attention), which offers several advantages:

[0084] Hierarchical semantic representation: Through stage-wise feature extraction, a multi-scale semantic encoding from local texture to global structure is gradually formed to adapt to the heterogeneous changes in the morphology of hepatocellular carcinoma lesions;

[0085] Sliding window mechanism: Window offset operation in each layer realizes spatial interaction, effectively avoids the problem of window islands, and realizes long-range dependency modeling while maintaining local awareness;

[0086] Computational efficiency optimization: Limiting the attention range with a window reduces the complexity from O(n^2) to O(n^2). 2 The complexity is reduced to O(n), providing resource feasibility for high-resolution MRI image modeling.

[0087] Furthermore, the visual pre-training uses the MVI image slice grading (M0 / M1 / M2) from hepatocellular carcinoma pathology reports as a weak supervision signal to conduct global semantic discrimination learning; by training the Swin Transformer model, the Swin Transformer model acquires strong semantic recognition ability and can distinguish the risk level of lesions. The feature vectors extracted after training are used to generalize to downstream multi-task objectives; wherein, the visual pre-training is based on a large sample of liver cancer surgery disease cohorts, using preoperative multi-parameter, multi-phase enhanced MRI images.

[0088] The image-label mapping strategy used in the visual pre-training is as follows: each MVI image slice of a hepatocellular carcinoma pathology report (which can be the largest cross-section or a multi-channel fusion of multiple cross-sections) is assigned a global label according to the microvascular invasion of the patient to which it belongs, thus constructing a learning pathway from local visual patterns to global diagnostic labels.

[0089] After removing the task-specific classification header, the loss function used is the three-class cross-entropy loss function, with the objective being:

[0090]

[0091] in, It is the value of the cross-entropy loss function for each category;

[0092] N represents the batch size of each category of samples in the visual pre-training;

[0093] y i This represents the true label of each class sample i, where the true label is either 0 or 1;

[0094] p i This represents the predicted label for each category sample i.

[0095] After completing visual pre-training, the visual encoding module also undergoes fine-tuning and optimization training to improve the model's transferability and generalization performance. This fine-tuning and optimization training includes:

[0096] Transfer learning initialization: The Swin Transformer model loads ImageNet pre-trained weights, freezes the underlying encoding modules, and only fine-tunes the mid-to-high-level semantic abstraction layers and task headers;

[0097] Layer-by-layer unfreezing strategy: In the early stage of training, only high-level feature modules are optimized, and then phased unfreezing is used to achieve gradient propagation from local optimization to global fine-tuning;

[0098] Activation and regularization control: Transformer uses the default GELU activation function; the MLP classification head uses Softmax activation; weight decay is introduced to control model complexity; overfitting is suppressed to improve model generalization ability;

[0099] Learning rate scheduling mechanism: A cosine annealing adjustment strategy is adopted to improve training stability. The expression is:

[0100]

[0101] Where, η max η represents the initial learning rate. max =1×10 -3 ;

[0102] η min Represents the minimum learning rate η min =0;

[0103] T cur Indicates the current iteration step;

[0104] T represents the total period, and T = 96.

[0105] 2. Construction of the multimodal feature alignment module

[0106] The multimodal feature alignment module is used to guide the multimodal large language model to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and the language of key pathological diagnostic reports of hepatocellular carcinoma.

[0107] The image modal input is the visual feature sequence output by the visual encoding module, and the text modal output is the structured descriptive text of a key pathological diagnostic report for hepatocellular carcinoma; the visual feature sequence and the structured descriptive text constitute a one-to-one paired training sample, and the training sample format is as follows:

[0108] Input: [The visual feature sequence output by the visual encoding module];

[0109] Output: [Structured description text of key pathological diagnosis report for hepatocellular carcinoma];

[0110] Furthermore, to achieve the mapping from image encoding to the language model input space, the multimodal feature alignment module introduces a linear alignment layer to map the visual embedding dimension to the text embedding dimension, thereby completing dimensional unification through a fully connected layer. The expression is as follows:

[0111]

[0112] In the formula, E t Represents the target feature vector;

[0113] Ev This represents the input feature vector;

[0114] "where" refers to further explanation of "W";

[0115] Let W represent a learnable weight matrix, and D... t D represents the text embedding dimension; v Indicates the visual embedding dimension.

[0116] This fully connected layer can be extended to a multi-layer nonlinear structure. However, in order to simplify the learning of alignment relationships and to enhance the semantic alignment of images and text, visual features and language embeddings are co-optimized in a shared semantic space during the SFT stage, thereby achieving explicit cross-modal expression consistency.

[0117] The construction process of the multimodal feature alignment module includes:

[0118] Image encoding: Each hepatocellular carcinoma MRI image is input into the visual encoding module for encoding, and the output is an MRI image embedding vector. This vector is then integrated into a global image representation vector z through a fully connected mapping. I ;

[0119] Text Encoding: Standardized microvascular invasion is structurally described using a text encoder, and then one-hot encoded by a text embedding module to generate a global text representation vector z. T ;

[0120] Positive sample construction: Each pair of image-text paired samples (z I ,z T (From the same patient;)

[0121] Negative sample construction: Constructing in-batch negative samples That is, in a training batch, any unpaired combination between other images and text is considered a negative example;

[0122] Contrastive learning loss function design: InfoNCE loss is used, which performs softmax normalization on the similarity between each image sample and all text samples, and then maximizes the probability of positive sample pairs. The overall loss function expression is:

[0123]

[0124] In the formula, The encoded features representing the modality of the i-th image and the i-th text;

[0125] The encoded features representing the modality of the i-th image and the j-th text;

[0126] cos(·,·) represents the cosine similarity;

[0127] τ represents the temperature hyperparameter;

[0128] N represents the amount of data used in each training iteration of the contrastive learning module, and its size is consistent with the global batch size.

[0129] Training details design: When training the contrastive learning module, the remaining parameters are kept frozen (the backbone of the visual encoding module and the text encoder are frozen), and only the image-text modality alignment layer is optimized.

[0130] Freezing Mechanism Explanation: During the contrastive learning training phase, only the image-text modality alignment layer (i.e., the fully connected mapping layer from visual features to the text embedding space) is optimized. All other module parameters are frozen through gradient truncation; this design, by separating modality alignment optimization from backbone parameter updates, ensures semantic consistency learning between modalities while avoiding damage to the representational capabilities of the pre-trained model. The benefits of parameter freezing include preventing overfitting (preserving the pre-trained knowledge of the large model from noise interference), improving computational efficiency (reducing the computational cost of backpropagation), and enhancing stability (avoiding gradient conflicts during multimodal joint training).

[0131] II. Fine-tuning and optimization of multimodal large language models

[0132] After the architecture of the multimodal large language model is implemented, it enters the instruction fine-tuning stage. The instruction fine-tuning stage aims to further optimize the response capability of the multimodal large language model in the hepatocellular carcinoma MRI image and text generation scenario through structured medical instruction tasks, so as to automatically complete the structured medical report writing task based on image features.

[0133] The instruction fine-tuning stage is achieved by constructing image-text-instruction-response data for training. The training objective is to enhance the model’s language generation ability, accuracy of medical terminology, and consistency of structural expression based on image understanding.

[0134] The instruction fine-tuning stage uses "visual coding features + natural language task instructions" as input and a structured diagnostic report as the output target.

[0135] The input includes image embeddings and task prompts; the image embeddings are visual feature sequences output by the visual encoding module, which serve as deep semantic inputs; the task prompts adopt a standard system prompting structure to prompt the multimodal large language model to output structured medical descriptions.

[0136] The output is a structured key pathological diagnosis report for hepatocellular carcinoma, including six types of hepatocellular carcinoma lesion attributes. The specific content of the six types of hepatocellular carcinoma lesion attributes is as follows:

[0137] (1) Tissue types: hepatocellular carcinoma, intrahepatic cholangiocarcinoma, mixed hepatocellular-cholangiocarcinoma, other types;

[0138] (2) Edmondson-Steiner classification (I, II, III, IV);

[0139] (3) Grading of microvascular invasion (M0, M1, M2);

[0140] (4) PD-L1 expression (low, high);

[0141] (5) Whether tertiary lymphoid structures were observed and their distribution location (e.g., within the tumor, at the tumor margin, or in adjacent liver tissue);

[0142] (6) Properties of the tertiary lymphoid structure (maximum diameter, number, maturity).

[0143] The instruction fine-tuning stage employs a joint loss function, with the main task being the cross-entropy loss function, used to maximize the likelihood of the target text token; knowledge preservation uses the KL divergence loss function, used to constrain the current model output distribution from deviating from the original pre-trained knowledge representation;

[0144] The joint objective function is as follows:

[0145]

[0146] in, Cross-entropy loss is the loss function for the task.

[0147] KL divergence loss is the loss function for knowledge preservation.

[0148] λ is a hyperparameter that adjusts the weight of the regularization term and is used to control the degree of influence of knowledge retention loss on the total loss;

[0149] The joint objective function is a weighted sum of cross-entropy loss and KL divergence loss, used to simultaneously consider generation accuracy and knowledge preservation during fine-tuning.

[0150] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0151] The multimodal large language model architecture of this invention is mainly based on the DeepSeek structure, employing an improved multi-layer Transformer backbone as the visual encoding module for visual feature extraction. It also introduces a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer to achieve automatic generation of standardized diagnostic reports from hepatocellular carcinoma MRI images. During the training phase, a Supervised Fine-Tuning (SFT) strategy is used to construct multimodal training samples with image features as input and structured diagnostic text as output.

[0152] The visual encoding module of this invention employs a multi-layer Transformer backbone and performs visual pre-training on hepatocellular carcinoma MRI data. Subsequently, the task-specific classification head is stripped, and only the backbone feature extraction network is retained as the image modality representation encoder, which is used to input to the downstream open-source multimodal large language model (such as the DeepSeek language model) to prepare for cross-modal joint modeling.

[0153] This invention introduces a multimodal alignment mechanism based on contrastive learning through a multimodal feature alignment module, guiding the model to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and diagnostic language, thereby enhancing the semantic consistency between visual and linguistic modalities in the multimodal large language model of this invention.

[0154] This invention is based on a large-sample cohort of liver cancer surgery and uses preoperative multi-parameter, multi-phase enhanced MRI images to generate matching postoperative pathology reports for liver cancer, which has significant guiding significance for clinical treatment decisions.

[0155] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma, characterized in that: The main architecture of the multimodal large language model is a visual encoding module, and it also incorporates a multimodal feature alignment module, a multi-head low-rank attention mechanism, an enhanced medical MoE mechanism, and a structured output decoding layer. The visual encoding module is built on the Swing Transformer architecture and pre-trained on hepatocellular carcinoma MRI data. Then, the task-specific classification head is stripped off, and only the backbone feature extraction network is retained as the representation encoder of the image modality, outputting a visual feature sequence. The multimodal feature alignment module guides the multimodal large language model to learn the cross-modal semantic mapping relationship between hepatocellular carcinoma MRI images and the language of key pathological diagnostic reports for hepatocellular carcinoma. The image modal input is the visual feature sequence output by the visual encoding module, and the text modal output is the structured descriptive text of the key pathological diagnostic report for hepatocellular carcinoma. The visual feature sequence and the structured descriptive text constitute a one-to-one pairing of training samples, and the training sample format is as follows: Input: [The visual feature sequence output by the visual encoding module]; Output: [Structured description text of key pathological diagnosis report for hepatocellular carcinoma]; The multi-head low-rank attention mechanism is used to solve the memory and computational efficiency problems of long sequence modeling, while accurately capturing cross-modal medical semantic associations; The enhanced medical MoE mechanism provides routing for liver cancer specialists; The structured output decoding layer is used to generate six types of liver cancer lesion attributes.

2. The multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma according to claim 1, characterized in that: The visual pre-training uses the MVI image slice grading of hepatocellular carcinoma pathology reports as a weak supervision signal to carry out global semantic discrimination learning; by training the Swin Transformer model, the Swin Transformer model acquires strong semantic recognition ability and can distinguish the risk level of lesions. The feature vectors extracted after training are used to generalize to downstream multi-task objectives. The image-label mapping strategy used in the visual pre-training is as follows: each MVI image slice of a hepatocellular carcinoma pathology report is assigned a global label according to the microvascular invasion of the patient to which it belongs, and a learning pathway from local visual patterns to global diagnostic labels is constructed. After removing the task-specific classification header, the loss function used is the three-class cross-entropy loss function, with the objective being: in, It is the value of the cross-entropy loss function for each category; N represents the batch size of each category of samples in the visual pre-training; y i This represents the true label of each class sample i, where the true label is either 0 or 1; p i This represents the predicted label for each category sample i.

3. The multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma according to claim 1, characterized in that: After completing visual pre-training, the visual encoding module also undergoes fine-tuning and optimization training to improve the model's transferability and generalization performance. This fine-tuning and optimization training includes: Transfer learning initialization: The Swin Transformer model loads ImageNet pre-trained weights, freezes the underlying encoding modules, and only fine-tunes the mid-to-high-level semantic abstraction layers and task headers; Layer-by-layer unfreezing strategy: In the early stage of training, only high-level feature modules are optimized, and then phased unfreezing is used to achieve gradient propagation from local optimization to global fine-tuning; Activation and regularization control: Transformer uses the default GELU activation function; the MLP classification head uses Softmax activation; weight decay is introduced to control model complexity; overfitting is suppressed to improve model generalization ability; Learning rate scheduling mechanism: A cosine annealing adjustment strategy is adopted to improve training stability. The expression is: Where, η max η represents the initial learning rate. max =1×10 -3 ; η min Represents the minimum learning rate η min =0; T cur Indicates the current iteration step; T represents the total period, and T = 96.

4. The multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma according to claim 1, characterized in that: The multimodal feature alignment module introduces a linear alignment layer to map the visual embedding dimension to the text embedding dimension, thereby achieving dimensional unification through a fully connected layer. The expression is as follows: In the formula, E t Represents the target feature vector; E v This represents the input feature vector; "where" refers to further explanation of "W"; Let W represent a learnable weight matrix, and D... t D represents the text embedding dimension; v Indicates the visual embedding dimension.

5. The multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma according to claim 1, characterized in that: The construction process of the multimodal feature alignment module includes: Image encoding: Each hepatocellular carcinoma MRI image is input into the visual encoding module for encoding, and the output is an MRI image embedding vector. This vector is then integrated into a global image representation vector z through a fully connected mapping. I ; Text Encoding: Standardized microvascular invasion is structurally described using a text encoder, and then one-hot encoded by a text embedding module to generate a global text representation vector z. T ; Positive sample construction: Each pair of image-text paired samples (z I ,z T (From the same patient;) Negative sample construction: Constructing in-batch negative samples Contrastive learning loss function design: InfoNCE loss is used, which performs softmax normalization on the similarity between each image sample and all text samples, and then maximizes the probability of positive sample pairs. The overall loss function expression is: In the formula, The encoded features representing the modality of the i-th image and the i-th text; The encoded features representing the modality of the i-th image and the j-th text; cos(·,·) represents the cosine similarity; τ represents the temperature hyperparameter; N represents the amount of data used in each training iteration of the contrastive learning module, and its size is consistent with the global batch size. Training details design: When training the contrastive learning module, the backbone of the visual encoding module and the text encoder is frozen, and only the image-text modality alignment layer is optimized.

6. The multimodal large language model for generating key pathological diagnostic reports of hepatocellular carcinoma according to claim 1, characterized in that: After the architecture of the multimodal large language model is implemented, it enters the instruction fine-tuning stage. The instruction fine-tuning stage aims to further optimize the response capability of the multimodal large language model in the hepatocellular carcinoma MRI image and text generation scenario through structured medical instruction tasks, so as to automatically complete the structured medical report writing task based on image features. The instruction fine-tuning stage is achieved by constructing image-text-instruction-response data for training. The training objective is to enhance the model’s language generation ability, accuracy of medical terminology, and consistency of structural expression based on image understanding. The instruction fine-tuning stage uses "visual coding features + natural language task instructions" as input and a structured diagnostic report as the output target. The input includes image embeddings and task prompts; the image embeddings are visual feature sequences output by the visual encoding module, which serve as deep semantic inputs; the task prompts adopt a standard system prompting structure to prompt the multimodal large language model to output structured medical descriptions. Output a structured key pathological diagnosis report for hepatocellular carcinoma, including six types of hepatocellular carcinoma lesion attributes; The instruction fine-tuning stage employs a joint loss function, with the main task being the cross-entropy loss function, used to maximize the likelihood of the target text token; knowledge preservation uses the KL divergence loss function, used to constrain the current model output distribution from deviating from the original pre-trained knowledge representation; The joint objective function is as follows: in, Cross-entropy loss is the loss function for the task. KL divergence loss is the loss function for knowledge preservation. λ is a hyperparameter that adjusts the weight of the regularization term and is used to control the degree of influence of knowledge retention loss on the total loss; The joint objective function is a weighted sum of cross-entropy loss and KL divergence loss, used to simultaneously consider generation accuracy and knowledge preservation during fine-tuning.

Images