Pre-training methods, diagnostic methods and systems for large-scale brain tumor imaging diagnostic models

Abstract

This invention discloses a pre-training method, diagnostic method, and system for a large-scale model for brain tumor imaging diagnosis. The pre-training method includes: acquiring image data of brain tumor patients and corresponding diagnostic text, wherein the image data includes multiple sequences; constructing a visual unification model, using the image data to perform full-sequence standard training and missing sequence distillation training on the visual unification model, learning a unified visual representation of image data with arbitrary sequence combinations; the missing sequence distillation training uses features of the full-sequence image data as teacher features and features of the missing sequence image data as student features, constraining the student features and teacher features to align in the feature space; and constructing a visual language model, based on multi-task objectives, using the unified visual representation output by the trained visual unification model as input, and combining it with the diagnostic text to train the visual language model. The model trained by this invention can effectively handle sequence missing cases, improving the accuracy and robustness of diagnosis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for pre-training a large model for brain tumor imaging diagnosis, a diagnostic method, and a system. Background Technology

[0002] Accurate diagnosis of brain tumors is crucial for clinical treatment decisions and prognostic assessment. Magnetic resonance imaging (MRI), due to its excellent soft tissue resolution, has become the primary imaging modality for brain tumor diagnosis. Routine clinical MRI examinations typically include multiple sequences, such as T1-weighted images (T1), T2-weighted images (T2), T1-enhanced images (T1CE), and T2-FLAIR sequences. Different sequences provide complementary diagnostic information, collectively outlining the complete characteristics of the tumor.

[0003] Despite the wealth of information provided by multi-sequence MRI, effectively integrating this information to achieve accurate diagnosis remains a significant challenge in clinical practice and research. Firstly, sequence loss is a widespread problem. In clinical practice, due to various practical considerations, patients may not be able to complete all four MRI sequences, but only a portion, resulting in missing image sequences. Our sampling statistics from the PACS database of well-known tertiary hospitals in China showed that only 17.19% of the sampled patients (n=31,245) had complete four sequences (T1, T2, T1CE, and T2 FLAIR), indicating that the vast majority of patient samples did not meet the requirement of complete four sequences. This sequence loss severely disrupts the complementarity between different sequence information, leading to a significant decline in the performance of most existing AI diagnostic models that rely on full-sequence data, thus limiting their applicability and robustness in the real world. Summary of the Invention

[0004] In view of this, this application provides a pre-training method, diagnostic method and system for a large model for brain tumor imaging diagnosis, in order to solve the problem of excessive reliance of existing brain tumor diagnostic models on full-sequence magnetic resonance imaging data.

[0005] To address the aforementioned technical problems, this application adopts the following technical solution: A method for pre-training a large-scale model for brain tumor imaging diagnosis is provided, comprising: acquiring image data of brain tumor patients and corresponding diagnostic text, wherein the image data includes T1 sequences, T2 sequences, T1CE sequences, and T2 FLAIR sequences; constructing a visual unification model, using the image data to perform full-sequence standard training and missing-sequence distillation training on the visual unification model, learning the unified visual representation of image data with arbitrary sequence combinations, wherein the missing-sequence distillation training uses fusion features extracted from the full-sequence image data as teacher features and features extracted from the missing-sequence image data as student features, and constrains the student features and teacher features to align their distribution in the feature space based on a distillation loss function; and constructing a visual language model, based on a multi-task objective that includes at least image-text alignment learning and text generation, using the unified visual representation output by the trained visual unification model as input, and combining it with diagnostic text to train the visual language model.

[0006] As a further improvement of this application, after obtaining the image data of the brain tumor patient and the corresponding diagnostic text, the method further includes: preprocessing the image data; extracting brain tumor diagnostic information and lesion region description from the diagnostic text; and constructing a structured diagnostic text using the brain tumor diagnostic information and lesion region description.

[0007] As a further improvement of this application, the visual unified model includes a shared encoder, an adapter layer, a feature interaction layer, and a decoder layer. The feature interaction layer includes a first fusion module for processing full sequence image data and a second fusion module for missing sequence image data. The first fusion module is constructed based on a multilayer perceptron structure, and the second fusion module is constructed based on a multi-head attention mechanism.

[0008] As a further improvement of this application, a full-sequence standard training of the visual unified model is performed using image data, including: Step 1: Extracting the first feature representation corresponding to each sequence from the first image data including all sequences using a shared encoder; Step 2: Adjusting the feature dimension and distribution of the first feature representation of each sequence using an adapter layer; Step 3: Integrating the first feature representations with adjusted feature dimension and distribution using a first fusion module to obtain a first fused feature; Step 4: Reconstructing the image using a decoder layer based on the first fused feature; Step 5: Updating the parameters of the shared encoder, adapter layer, first fusion module, and decoder layer through backpropagation based on the reconstructed image, the first image data, and a first preset loss function; Steps 1 to 5 are executed iteratively until a preset number of training iterations is reached.

[0009] As a further improvement of this application, the first preset loss function is obtained by weighted summation of L1 loss, L2 loss and structural similarity loss.

[0010] As a further improvement to this application, a missing sequence distillation training of the visual unification model is performed using image data, including: Step 6: Constructing a second image data containing missing sequences using the first image data, and determining the number of sequences in the second image data; Step 7: Extracting the second feature representation corresponding to each sequence from the second image data using a shared encoder; Step 8: Adjusting the feature dimension and distribution of the second feature representation of each sequence using an adapter layer; Step 9: Controlling the second fusion module to adaptively adjust the number of attention heads according to the number of sequences, and then integrating the second feature representations with adjusted feature dimensions and distribution to obtain the second fused feature; Step 10: Using the first fused feature as the teacher feature and the second fused feature as the student feature, and combining it with a preset distillation loss function, the second fusion module is distilled and trained.

[0011] As a further improvement to this application, before step nine, the method further includes: assigning an independent embedding vector to the second feature representation corresponding to each sequence.

[0012] As a further improvement to this application, the distillation loss function is obtained based on the weighted sum of L1 loss, L2 loss, first cosine similarity loss and KL divergence loss.

[0013] As a further improvement to this application, the visual language model includes an image feature encoder, a text encoder, and a text generation module.

[0014] As a further improvement to this application, the steps for training a visual language model based on a contrastive learning task include: inputting a unified visual representation or a single image feature of each sequence into an image feature encoder to extract an image feature matrix; inputting diagnostic text corresponding to the unified visual representation or the single image feature of each sequence into a text encoder to extract a text feature matrix; calculating the second cosine similarity between all text pairs in the text feature matrix, and constructing a soft label matrix based on the second cosine similarity and a preset similarity threshold, wherein the soft label matrix is ​​represented as: ; in, This is a soft label matrix. and These represent different diagnostic texts. To preset the similarity threshold, For preset temperature parameters, To preset the maximum soft tag value, Sigmoid function The second cosine similarity is used; the image feature matrix, text feature matrix, and soft label matrix are used to calculate the image-text contrast loss corresponding to the unified visual representation or the single image feature of each sequence; the total image-text contrast loss is calculated based on the unified visual representation and the image-text contrast loss corresponding to the single image feature of each sequence, and the image feature encoder and text encoder are updated by backpropagation based on the total image-text contrast loss.

[0015] As a further improvement of this application, the steps of training a visual language model based on a text generation task include: performing feature enhancement processing on a unified visual representation to construct an image memory, wherein the feature enhancement processing includes extracting global image memory and fusing multi-sequence local features; converting the acquired real diagnostic text into a target token sequence and constructing a decoder input sequence based on the target token sequence; inputting the decoder input sequence and the image memory into a text generation module and performing training in an autoregressive manner, wherein at each time step the text generation module predicts the probability distribution of the next token based on the image memory and the generated preceding tokens; calculating the cross-entropy loss between the probability distribution and the real target token sequence, and updating the parameters of the text generation module based on the cross-entropy loss.

[0016] As a further improvement of this application, after the visual language model is pre-trained, a downstream task fine-tuning step is also included. The downstream task includes at least one of the following: brain tumor classification task, brain tumor segmentation task, and prognostic analysis task. The brain tumor classification task is fine-tuned by adding a classification head and using a progressive thawing strategy. The brain tumor segmentation task is fine-tuned by adding a decoder that includes multi-scale upsampling and skip connection mechanisms. The prognostic analysis task is fine-tuned by constructing a risk regression branch that includes multi-sequence feature fusion and time dimension modeling.

[0017] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a brain tumor diagnosis method based on a large-scale brain tumor imaging diagnostic model, comprising: acquiring image data of the object to be diagnosed, the image data including at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; inputting the image data into a pre-trained unified visual model of one of the above to obtain a unified visual representation; and inputting the unified visual representation into a pre-trained visual language model of one of the above to obtain a brain tumor diagnosis result.

[0018] As a further improvement to this application, the brain tumor diagnostic results include: brain tumor classification results, including primary and / or secondary classifications of the tumor; brain tumor segmentation results, including the tumor core, edematous regions, and segmentation masks that enhance the tumor; prognostic analysis results, including survival prediction or recurrence risk assessment; and a diagnostic text report containing a description of the lesion and a diagnostic conclusion.

[0019] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a pre-training system for a large-scale model of brain tumor imaging diagnosis, comprising: a first acquisition module for acquiring image data of brain tumor patients and corresponding diagnostic text, wherein the image data includes T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; a first training module for constructing a visual unification model, using the image data to perform full-sequence standard training and missing sequence distillation training on the visual unification model, learning the unified visual representation of image data with arbitrary sequence combinations, wherein the missing sequence distillation training uses the fusion features extracted from the full-sequence image data as teacher features and the features extracted from the missing sequence image data as student features, and constrains the student features and teacher features to align their distribution in the feature space based on the distillation loss function; and a second training module for constructing a visual language model, based on a multi-task objective that includes at least image-text alignment learning and text generation, using the unified visual representation output by the trained visual unification model as input, and combining it with diagnostic text to train the visual language model.

[0020] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a brain tumor diagnostic system based on a large-scale brain tumor imaging diagnostic model, comprising: a second acquisition module for acquiring image data of the object to be diagnosed, the image data including at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; a feature fusion module for inputting the image data into a pre-trained visual unified model to obtain a unified visual representation; and a result prediction module for inputting the unified visual representation into a pre-trained visual language model to obtain a brain tumor diagnostic result.

[0021] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, the computer device including a processor and a memory coupled to the processor, the memory storing program instructions, when the program instructions are executed by the processor, causing the processor to perform the steps of the pre-training method of the large model for brain tumor imaging diagnosis as described in any of the above claims or to perform the steps of the brain tumor diagnosis method based on the large model for brain tumor imaging diagnosis as described in any of the above claims.

[0022] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a storage medium storing program instructions capable of implementing the brain tumor image diagnosis large model pre-training method as described above or executing the brain tumor diagnosis method based on the brain tumor image diagnosis large model as described above.

[0023] The beneficial effects of this application are: This application's pre-training method for a large-scale model for brain tumor imaging diagnosis provides a comprehensive learning foundation for the model by acquiring multi-sequence magnetic resonance imaging data and corresponding diagnostic texts. It then innovatively combines full-sequence standard training with missing-sequence distillation training. In the missing-sequence distillation training, fused features extracted from full-sequence image data serve as teacher features, while features extracted from missing-sequence image data serve as student features. A distillation loss function constrains the alignment of student and teacher features in the feature space. This design enables the model to handle arbitrary sequence combinations. Finally, through multi-task training involving image-text alignment learning and text generation, using unified visual representations as input and combining them with diagnostic text, the model masters the complete mapping ability from images to diagnostic reports. This allows the trained model to extract effective visual representations from arbitrary sequence combinations, ensuring diagnostic stability even with missing sequences. The established unified feature space provides a high-quality visual foundation for multimodal diagnosis, and the achieved end-to-end diagnostic capability enables the model to output complete diagnostic reports. Attached Figure Description

[0024] Figure 1 This is a schematic flowchart of a large-scale model pre-training method for brain tumor imaging diagnosis according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the functional modules of the large model pre-training system for brain tumor imaging diagnosis according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a brain tumor diagnosis method based on a large model of brain tumor imaging diagnosis according to an embodiment of the present invention. Figure 4 This is a schematic diagram of the functional modules of a brain tumor diagnostic system based on a large model of brain tumor imaging diagnosis according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative spatial positions and movements of components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] Figure 1 This document illustrates a flowchart of an embodiment of the brain tumor imaging diagnosis large model pre-training method of the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the pre-training method for this large-scale brain tumor imaging diagnostic model includes: Step S100: Obtain the imaging data and corresponding diagnostic text of the brain tumor patient. The imaging data includes T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence.

[0029] Specifically, before model training, sample data is prepared in advance. In this invention, the model is used to identify whether a patient has a brain tumor through magnetic resonance imaging (MRI) data of the brain. Therefore, the prepared sample data includes multi-sequence MRI image data of brain tumor patients and corresponding professional diagnostic texts. The image data covers four core sequences required for clinical diagnosis of brain tumors: T1 sequence (T1-weighted imaging), T2 sequence (T2-weighted imaging), T1CE sequence (T1-enhanced scanning), and T2 FLAIR sequence (T2 fluid attenuation inversion recovery sequence). These sequences provide complementary information on brain anatomy, tissue water content, blood-brain barrier integrity, and the extent of edema.

[0030] Furthermore, after step S100, the following steps are also included: 1. Preprocess the image data.

[0031] Specifically, the preprocessing steps include: (1) DICOM to NIFTI format conversion. First, the original DICOM image data needs to be converted to NIFTI format for subsequent processing and analysis. In this embodiment, the dcm2niix tool can be used for conversion. This tool can efficiently convert DICOM files to NIFTI format in batches and can retain the key information and metadata of the original image data.

[0032] (2) Field strength correction. Due to factors such as magnetic field inhomogeneity, multi-sequence magnetic resonance imaging data may exhibit inconsistent field strength, which can affect the accuracy of subsequent analysis. The purpose of field strength correction is to eliminate this inhomogeneity and make the intensity distribution of the image more uniform. In this embodiment, the N4BiasFieldCorrection algorithm in the SimpleITK library can be used to perform field strength correction on images converted to NIFTI format to eliminate intensity deviations caused by magnetic field inhomogeneity.

[0033] (3) Removal of the skull. In order to focus on brain tissue, it is necessary to remove non-brain structures such as the skull. This step can be achieved using tools such as HD-BET (Brain Region Extraction Tool). By setting appropriate parameters, such as threshold and surface sampling interval, HD-BET can automatically extract brain tissue and remove non-brain regions such as the skull and scalp, thereby improving the purity of the image and reducing interference from other structures.

[0034] (4) Sequence registration. Due to differences in imaging principles, the grayscale and contrast of the same anatomical structure vary greatly in different sequences of multi-sequence imaging data. For example, T1 sequence focuses on anatomical details, T2 sequence is sensitive to tissue edema, T2FLAIR can suppress cerebrospinal fluid interference, and T1CE highlights vascular enhancement features. These complementary information are crucial for accurate diagnosis, but spatial misalignment can occur due to factors such as changes in patient position and magnetic field inhomogeneity, leading to lesion localization deviations or feature fragmentation. Therefore, spatial alignment must be achieved through registration techniques.

[0035] (5) 1×1×1 voxel normalization. The purpose of voxel normalization is to unify the spatial resolution of different images, making the size of each voxel consistent. The image is resampled to... The voxel size ensures that spatial resolution differences between different images do not affect the results in subsequent analysis. This process is usually performed after sequence registration, adjusting the voxel size of the images using methods such as interpolation to meet standardization requirements.

[0036] (6) Cropping the foreground brain regions. After removing the skull, the image may still contain some background areas that are not brain tissue. The step of cropping the foreground brain regions is to further focus on the brain tissue and remove these background parts. The boundaries of the brain regions can be determined manually or automatically, and then the regions containing complete brain tissue can be cropped. This helps to reduce the amount of data, improve computational efficiency, and allow subsequent analysis to focus more on brain structures that are meaningful for the diagnosis of brain tumors.

[0037] (7) Size Standardization. Finally, to ensure that the images of different patients are consistent in spatial size and to facilitate model input and comparative analysis, size standardization is necessary. Scaling the images to a specific size, such as 256×256×256 or 196×196×196, ensures consistency in spatial resolution and size across all images. This step is usually performed after voxel standardization and cropping of the foreground brain regions, adjusting the image size using interpolation or other methods to meet predetermined standard size requirements.

[0038] 2. Extract brain tumor diagnostic information and lesion region description from the diagnostic text, and construct a structured diagnostic text using the brain tumor diagnostic information and lesion region description.

[0039] Specifically, this embodiment utilizes Large Language Modeling (LLM) to extract information from diagnostic text. Pathology and imaging reports typically contain a large amount of technical terminology and complex medical information; therefore, precise prompts need to be designed for specific tasks and requirements. Through batch and streamlined code, LLM (such as Deepseek) interfaces are called to efficiently extract the required textual information, such as lesion region descriptions and brain tumor diagnostic categories. This invention designs the prompts according to the high-quality paradigm of prompt engineering, proposing specific requirements from four aspects: "role setting," "input instructions," "task requirements," and "output format." Targeted testing and fine-tuning are required to ensure the effectiveness of the output information. Since the brain tumor diagnosis task is essentially a classification task of several brain tumor subtypes, high-quality natural language text data providing weakly supervised information is crucial. For processing this type of information, this embodiment prioritizes extraction from pathology reports. However, in actual clinical practice, not all patients undergo both imaging and pathology examinations simultaneously. Therefore, to fully and rationally utilize available data, for patients lacking pathology reports, this embodiment extracts the required tumor classification information from imaging diagnostic reports. It is worth noting that diagnostic conclusions from imaging reports often serve only as a reference and may even contain multiple ambiguous labels, such as descriptions like "meningioma? pituitary adenoma? glioma?" in the report. For this type of weakly supervised labeled text from imaging reports, words like "maybe," "perhaps," and "or" need to be introduced when constructing template text to provide additional supplementary information, thereby optimizing the probability distribution when the model makes diagnostic decisions and improving diagnostic accuracy. For text describing lesion regions, these often appear alongside terms like "space-occupying lesion," "abnormal," and some special symbols. This information can be easily and effectively extracted by constructing appropriate string formats and regularization expressions. Finally, constructing the extracted brain tumor diagnostic information and lesion region descriptions into a custom text template is an important step in achieving information structuring and standardization. This embodiment of the invention adopts a "lesion region + diagnostic category" construction format, such as "right frontotemporal insula lesion, patient has glioblastoma." This template format is concise and clear, clearly expressing key information and facilitating quick understanding and application by doctors. In this way, a large amount of unstructured text data can be transformed into structured information, providing strong support for medical decision-making.

[0040] Step S101: Construct a visual unification model. Use image data to perform full-sequence standard training and missing-sequence distillation training on the visual unification model to learn the unified visual representation of image data with arbitrary sequence combinations. For missing-sequence distillation training, use the fusion features extracted from the full-sequence image data as teacher features and the features extracted from the missing-sequence image data as student features. Based on the distillation loss function, constrain the student features and teacher features to align their distribution in the feature space.

[0041] It should be noted that the Visual Unified Model is an advanced model for multimodal visual data processing, particularly suitable for scenarios such as medical imaging that contain multiple image sequences (e.g., T1, T2, T1CE, T2 FLAIR, etc.). In applications such as medical diagnosis, different image sequences can provide information about human tissues and lesions from different perspectives. However, in practice, some image sequences may be missing, posing challenges to subsequent image analysis and diagnosis. This model aims to achieve accurate reconstruction of the entire image sequence and robust handling of missing sequences through effective feature extraction, fusion, and generation mechanisms.

[0042] Specifically, in this embodiment, the model has two branch modes: full-sequence standard training and missing-sequence distillation training. Full-sequence standard training, given a full-sequence image input, is responsible for fusing features from different sequences to uncover their potential relationships and complementary information. Missing-sequence distillation training, in the presence of missing sequences, provides representational information for image reconstruction of missing sequences through feature interaction with other non-missing sequences and distillation learning with the fused full-sequence features. In this embodiment, full-sequence image data refers to image data including four sequences: T1, T2, T1CE, and T2FLAIR. Missing-sequence image data refers to image data missing at least one of the following sequences: T1, T2, T1CE, or T2FLAIR.

[0043] Furthermore, the visual unified model includes a shared encoder, an adapter layer, a feature interaction layer, and a decoder layer. The feature interaction layer includes a first fusion module for processing full-sequence image data and a second fusion module for processing missing sequence image data. The first fusion module is built based on a multilayer perceptron structure, and the second fusion module is built based on a multi-head attention mechanism.

[0044] Specifically, the shared encoder employs a 3D ViT encoder structure, which extracts common visual features from the input multi-sequence image data. By capturing local and global features in the image, it provides foundational feature data for subsequent processing. The core idea of ​​the 3D ViT-based encoder is to segment the 3D image into multiple patches, map these patches to a low-dimensional embedding space, and then extract features through a multi-layer Transformer module. The model uses PatchEmbeddingBlock (patch embedding module) implemented based on the MONAI (Medical Artificial Intelligence Open Network) library to segment the input 3D image into multiple patches and convert each patch into a fixed-length embedding vector. Simultaneously, convolutional positional encoding is added to help the model capture the spatial relationships between patches. Using 3D ViT as the visual encoder backbone network effectively captures global information of the image, avoiding the limitations of traditional convolutional neural networks in handling long-distance dependencies. Furthermore, the model introduces a learnable classification token, CLS Token, which is added before the embedding sequence of each sample, making it more suitable for subsequent diagnostic classification or feature representation.

[0045] The adapter layer, located after the shared encoder, adapts features from different sequences. Since the features of different image sequences have different distributions and characteristics, the adapter layer adjusts specific parameters to enable better fusion and interaction of features from different sequences. The adapter layer uses multi-layered stacked adapter modules to enhance feature representation capabilities. Each module contains six sub-modules: input normalization, self-attention mechanism, dimensionality reduction projection, nonlinear activation, dimensionality increase projection, and residual connections, working together collaboratively.

[0046] The feature interaction layer is the core of the model, consisting of a first fusion module, a feature masking module, a second fusion module, and a feature generator. The first fusion module, given a full-sequence image input, is responsible for fusing features from different sequences to uncover their potential relationships and complementary information. The feature masking module masks the full-sequence feature embeddings based on all sequence combinations. The feature generator, based on the fusion, generates representative sequence features. The second fusion module, in the case of missing sequences, provides representational information for image reconstruction of missing sequences through feature interaction with other non-missing sequences and distillation learning with the fused full-sequence features.

[0047] The decoder layer provides feature-to-image decoding and reconstruction capabilities, consisting of a shared decoder and four sequence-specific regression heads. During forward propagation, for each sequence feature, it sequentially undergoes feature processing, attention pooling, mapping to volume, reshaping into a 3D volume, passing through an upsampling network and an output head. The output is then resized and numerically constrained. Finally, the reconstructed images of all sequences are stitched together in a specific order and output, which is then used to calculate the reconstruction loss with the original image.

[0048] The first fusion module is based on a multilayer perceptron (MLP) structure, consisting of cascaded linear transformations, normalization, activation functions, and dropout layers. The second fusion module introduces a multi-head attention mechanism to compensate for insufficient information due to missing sequences. By specifying the number of input sequences and adaptively adjusting the number of attention heads, this module can capture the contextual relationships between different sequences under limited information conditions, improving the sequence feature fusion effect under incomplete input. The attention mechanism is applied after input concatenation, further alleviating the problems of feature sparsity and limited semantic representation.

[0049] Furthermore, the visual unification model is trained using image data on a full-sequence standard, including: Step 1: Use a shared encoder to extract the first feature representation corresponding to each sequence from the first image data including all sequences.

[0050] Specifically, in the full-sequence standard training phase, the input is a complete multi-sequence image, covering four sequences: T1, T2, T1CE, and T2 FLAIR. Taking the 3D ViT visual encoder as an example, the image is first segmented into multiple patches, which are then linearly projected into feature embeddings for each patch. Subsequently, in the Transformer layer, a self-attention mechanism is used to capture the global relationships between different patches, while a multilayer perceptron (MLP) is used for feature transformation. After multi-layer processing, the shared encoder outputs the first feature representation corresponding to each sequence.

[0051] Step 2: Use the adapter layer to adjust the feature dimension and distribution of the first feature representation of each sequence.

[0052] Specifically, the first feature representations of each sequence output by the shared encoder are fed into the adapter layer. The adapter layer processes the first feature representations of each sequence individually, adjusting the feature dimensions and distribution through convolution or fully connected operations. This ensures that the first feature representations of different sequences retain their unique information, allowing for better fusion in subsequent feature interaction layers. For example, for T1 sequence images of the brain, the corresponding adapter module might extract anatomical features such as gray and white matter boundaries; for T2 sequence images, it might extract features related to tissue water content.

[0053] Step 3: Use the first fusion module to integrate the first feature representations with adjusted feature dimensions and distribution to obtain the first fused feature.

[0054] Specifically, this first fusion module is based on a multilayer perceptron (MLP) structure, which deeply fuses features from different modalities through cascaded linear transformations, normalization, activation functions, and dropout layers. The input first feature representation is first concatenated along the sequence dimension, making the representation dimension of each sample hidden_size × num_sequences. Then, a four-layer MLP structure maps the concatenated high-dimensional features back to the original dimension hidden_size. This process not only integrates semantic information between different modalities but also effectively improves representation capabilities and model generalization performance through non-linear activation and layer normalization. Compared with general attention fusion mechanisms, this first fusion module is more lightweight and suitable for application scenarios with complete modal data that can fully model the interactions between different modalities.

[0055] Furthermore, to enhance the model's reconstruction and cross-modal generation capabilities, this embodiment of the invention introduces a set of independent sequence feature generation modules, each used to reconstruct the features of each sequence from the fused first fused feature. This part is called the "sequence feature generator," and its core consists of an independent MLP network for each sequence. Each generator accepts fused features of the hidden_size dimension as input, and after processing through multiple linear transformations and the GELU activation function, outputs reconstructed features of the same dimension as the original modal features. The key to this design is to introduce a specific parameter space for each sequence feature, enabling it to more effectively extract the parts most relevant to its modality-specific information from the shared semantic space, thereby improving the ability of feature completion and cross-modal mapping.

[0056] Step 4: Reconstruct the image using the decoder based on the first fusion feature.

[0057] Specifically, this image reconstruction process aims to effectively evaluate and reverse optimize the generated image in the multimodal feature fusion and modal completion task, ensuring that the generated result is as close as possible to the real image in terms of structure, content and edge details.

[0058] Step 5: Update the parameters of the shared encoder, adapter layer, first fusion module, and decoder layer through backpropagation based on the reconstructed image, first image data, and first preset loss function.

[0059] Repeat steps one through five until the preset number of training iterations is reached.

[0060] Furthermore, the first preset loss function is obtained by weighted summation of L1 loss, L2 loss and structural similarity loss.

[0061] It should be noted that this first loss function integrates L1 loss, L2 loss, and structural similarity loss (SSIMLoss), and includes L1 & L2 regularization constraints on the model parameters. Specifically, given a reconstructed image... With real image data The image reconstruction loss is defined as the sum of the following three parts: L1 loss (Mean Absolute Error); ; L2 loss (Mean Squared Error); ; Structural similarity loss (SSIM); ; in, This refers to a structural similarity index that measures local brightness, contrast, and structural consistency. A value closer to 1 indicates a higher similarity. Its formula is defined as: ; Ultimately, the total loss for image reconstruction is: ; in, It is L1 loss. It is L2 loss. It is structural similarity loss. It represents the number of reconstructed images. and These are the local mean and variance, respectively. and It is a stable term. These are the weighting coefficients for L1 loss, L2 loss, and SSIM loss, respectively.

[0062] Furthermore, the visual unification model is trained using image data through missing sequence distillation, including: Step 6: Construct second image data with missing sequences using the first image data, and determine the number of sequences in the second image data.

[0063] Specifically, during the model initialization phase, this embodiment of the invention predefines a set of sequence types M = {T1, T1CE, T2, T2FLAIR}, representing four available MRI image sequences. Based on this set, the model automatically constructs all possible subset combinations other than the complete set of sequence combinations, referred to as "missing sequence combinations". The specific process involves enumerating all non-empty proper subsets of M. When |S|<|M| is completed, the total number of combinations generated is 2^(|M|)–2 = 14. These combination types represent sequence missing situations that may occur in real clinical diagnostic scenarios.

[0064] Step 7: Use a shared encoder to extract the second feature representations corresponding to each sequence from the second image data.

[0065] Specifically, the operation in step seven is the same as that in step one. Please refer to step one for details, which will not be repeated here.

[0066] Step 8: Use the adapter layer to adjust the feature dimension and distribution of the second feature representation of each sequence.

[0067] Specifically, step eight is the same as step two. Please refer to step two for details, which will not be repeated here.

[0068] Step 9: Control the second fusion module to adaptively adjust the number of attention heads according to the number of sequences, and then integrate the second feature representation with adjusted feature dimensions and distribution to obtain the second fused feature.

[0069] Specifically, during model initialization, a dedicated `AttentionFusionModule` instance is pre-created for each of the 15 possible combinations of missing sequences (2^4-2, minus the complete sequence and the empty sequence). All these instances are part of the model, but their parameters are independent or partially shared. When a batch of data is input into the model, it first identifies the specific sequence combination for each sample (e.g., sample A is [T1, T2], sample B is [T1CE, FLAIR]). Then, based on this combination "label," the sample is routed to the dedicated `AttentionFusionModule` instance for processing. In this embodiment, the number of attention heads (`num_heads`) is set equal to the number of input sequences. For example, a module with two input sequences uses two attention heads, and a module with three input sequences uses three attention heads. With this setup, each attention head can "focus" on understanding and processing the features of one of the input sequences. Through the attention mechanism, these heads can calculate the interaction weights between the sequence and other sequences, thus capturing the contextual relationships between sequences "in the context of specific missing information." During the attention mechanism calculation process, the model dynamically generates attention weights. These weights determine the proportion of each sequence's features when fusing features. These weights are calculated in real time based on the current input feature content. Therefore, for different input samples (even the same missing combination), the fusion weights also change dynamically, thus achieving an adaptive, content-aware feature fusion.

[0070] Step 10: Using the first fusion feature as the teacher feature and the second fusion feature as the student feature, the second fusion module is trained by distillation using a preset distillation loss function.

[0071] Specifically, to address the issues of incomplete information and insufficient representational capacity in the case of missing sequences, this embodiment proposes a knowledge distillation mechanism. Combined with the previously defined missing sequence fusion module, this mechanism allows the model to simulate multiple combinations of incomplete sequences in advance during the training phase, learning fusion capabilities and thus achieving strong robustness during the inference phase. This strategy introduces a "teacher-student model" framework, achieving knowledge transfer and representation alignment based on shared parameters.

[0072] During the distillation training phase, the model first extracts teacher features based on the complete sequence set M. These features, generated by the first fusion module, exhibit the strongest semantic consistency and contextual completeness when all sequences are complete, and are therefore considered the upper bound of knowledge for the current task. Subsequently, the model simulates various combinations of missing sequences, invoking the second fusion module corresponding to each combination to fuse features containing only partial sequences, generating student feature representations. In this way, the model is exposed to various sequences with missing information that might be encountered in real-world scenarios during training, allowing it to gradually learn how to recover semantic information as much as possible even with incomplete information.

[0073] Furthermore, the distillation mechanism is not merely used to train the second fusion module; its other important function is to guide the optimization of the feature generation module. In the shared semantic space after fusion, each sequence is configured with an independent generator network to reconstruct the feature representation of that sequence and ultimately generate a reconstructed image of the entire sequence. Through the distillation process, the model not only learns how to fuse and utilize incomplete sequence combination information but also gradually develops the ability to recover local details from global semantic representations.

[0074] Furthermore, prior to step nine, the method includes assigning an independent embedding vector to the second feature representation corresponding to each sequence.

[0075] Specifically, to complement the fusion module, the model also introduces a sequence embedding mechanism. Each sequence is assigned an independent embedding vector, which is added to its corresponding sequence features before feature fusion to inject sequence recognition information. This design allows the model to encode the structural and semantic differences between different sequences in a unified space, effectively improving fusion quality and enhancing the sequence discriminability of feature representations.

[0076] Furthermore, the distillation loss function is obtained by weighting the L1 loss, L2 loss, first cosine similarity loss, and KL divergence loss.

[0077] Specifically, to improve the quality of knowledge transfer, this embodiment designs a composite distillation loss function, comprising four parts: L1 loss, L2 loss, cosine similarity loss, and KL divergence loss. L1 and L2 losses jointly constrain the similarity between student and teacher features in the numerical space, while the cosine loss enhances semantic consistency in terms of direction. To further capture subtle differences in feature distribution, KL divergence is introduced as an alignment mechanism for the probability space; this operation is implemented after feature normalization and Softmax transformation. Finally, the weighted sum of all loss terms constitutes the total distillation loss. It is noteworthy that the distillation loss of each second fusion module is weighted according to the number of missing sequences; the more missing sequences, the greater the weight, reflecting the difficulty and importance of the reconstruction task.

[0078] Step S102: Construct a visual language model based on a multi-task objective that includes at least image-text alignment learning and text generation. Use the unified visual representation output by the trained visual unified model as input and combine it with diagnostic text to train the visual language model.

[0079] Specifically, the visual language model is a multimodal model for multi-task learning. The training of the visual language model aims to achieve more accurate brain tumor identification and diagnosis and diagnostic text generation tasks by aligning and interacting visual and textual information at multiple granularities.

[0080] Furthermore, the visual language model includes an image feature encoder, a text encoder, and a text generation module.

[0081] Specifically, the visual language model mainly includes an image feature encoder, a text encoder, and a text generation module. Through the collaborative work of these modules, deep fusion and interaction of visual and text information are achieved, and it is used to perform diagnostic auxiliary functions such as text description generation and lesion localization heatmap generation.

[0082] The image encoder takes a unified visual representation as input, obtained by fusing image representations and features using a pre-trained model with visual uniformity. This representation then undergoes further encoding in this module, aiming to transform it into a representation more suitable for alignment with text features. The encoder employs a Transformer architecture, using a multi-head self-attention mechanism and a feedforward neural network to learn the relationships between image features. During encoding, the image encoder captures both global and local information of the image, generating a semantically rich image feature representation. This representation better matches text features, providing a foundation for subsequent multimodal tasks.

[0083] The primary function of the text encoder is to encode features from the input diagnostic text, converting it into a vector representation. This text encoder is based on pre-trained language models, such as BERT and CONCH models, and generates feature representations of the diagnostic text through word segmentation and encoding operations. In its design, the text encoder considers the semantic and contextual information of the text, learning the dependencies between text elements through multi-layered Transformer blocks. The generated text feature representations accurately reflect the semantic information of the text, providing crucial input for subsequent multimodal alignment and task execution.

[0084] In multimodal image-text alignment and generation tasks, the text generation module (Caption Generation Module) plays a core role, responsible for extracting semantic representations from visual input and transforming them into natural language descriptions. The text generation module used in this embodiment is built on a Transformer architecture, with its input derived from deep encoding of image features. Combined with a cross-modal attention mechanism, it effectively achieves semantic mapping from image to text. During training, this module progressively generates target description sequences through autoregression, using standard cross-entropy loss as the optimization objective to ensure that the generated results are as semantically and structurally close as possible to manually annotated text descriptions. Specifically, the text generation module consists of three parts: an embedding layer, a Transformer decoder, and an output head. First, the original text is converted into token IDs and mapped to a fixed-dimensional vector representation through the embedding layer. This representation is fed into the decoder as input to the target sequence. Simultaneously, image features are enhanced through feedforward and self-attention encoding to construct image memory, serving as the context input to the decoder. To ensure the generation process conforms to the natural order of the language, the model introduces a causal mask, ensuring that each generation step depends only on tokens preceding the current position. This autoregressive mechanism is a classic sequence generation strategy that helps maintain the coherence of the grammatical structure.

[0085] Furthermore, the goal of the contrastive learning task is to achieve image-text alignment. In practical applications, such as image retrieval and image caption generation, the ability to align images and text is crucial. This capability is achieved through contrastive learning, which aims to maximize the similarity between positive sample pairs while minimizing the similarity between negative sample pairs. Here, positive samples are corresponding image-text pairs, and negative samples are combinations of images with other text. Therefore, the steps for training a visual language model based on the contrastive learning task include: 1. Input the unified visual representation or the single image feature of each sequence into the image feature encoder to extract the image feature matrix.

[0086] Specifically, the unified visual representation is a comprehensive feature derived from the output of the unified visual model, which integrates information from all available sequences and represents the patient's global visual information. Individual image features of each sequence, such as independent T1 features, T2 features, etc., represent local, sequence-specific visual information.

[0087] 2. Input the diagnostic text corresponding to the unified visual representation or the single image feature of each sequence into the text encoder to extract the text feature matrix.

[0088] 3. Calculate the second cosine similarity between all text pairs in the text feature matrix. Construct a soft label matrix based on the second cosine similarity and a preset similarity threshold. The soft label matrix is ​​represented as follows: ; in, This is a soft label matrix. and These represent different diagnostic texts. To preset the similarity threshold, For preset temperature parameters, To preset the maximum soft tag value, Sigmoid function This represents the second cosine similarity.

[0089] 4. Calculate the image-text contrast loss corresponding to the unified visual representation or the single image feature of each sequence using the image feature matrix, text feature matrix, and soft label matrix.

[0090] 5. Then, calculate the total image-text contrast loss based on the unified visual representation and the image-text contrast loss corresponding to the single image feature of each sequence, and perform backpropagation update on the image feature encoder and text encoder based on the total image-text contrast loss.

[0091] In this embodiment, an image-text alignment loss is designed to enable the model to learn the association between images and text, ensuring that the model can understand that the image and the corresponding text description are semantically matched. By minimizing the image-text alignment loss, the model can find a suitable mapping relationship in the feature space of images and text, so that related images and texts are closer in the feature space, while unrelated ones are farther apart.

[0092] Specifically, image features are defined as matrices. The text features are matrices Each row represents the embedding vector of a sample. It is the batch size. It is the embedded dimension. The comparison temperature parameter (calculated from logit_scale) is used to control the "smoothness" of the distribution. The formula for calculating logits is as follows: ; The standard InfoNCE loss form is as follows (image to text): ; Similarly, text-to-image conversion works similarly: ; The final comparative loss is: .

[0093] To overcome the "hard boundary" limitation of positive and negative sample partitioning, this embodiment of the invention employs a semantically aware soft labeling mechanism. This mechanism utilizes a pre-trained text encoder (such as Bio_ClinicalBERT or MiniLM) to calculate a semantic similarity matrix between texts, whose elements are cosine similarities: ; in, These represent two separate texts; when Samples with similarity greater than the set similarity threshold are considered "semantically similar," and the corresponding soft-label matrix is ​​as follows: ; in, For similarity threshold, For temperature parameters, The maximum soft tag value, This is the Sigmoid function.

[0094] Finally, cross-entropy loss is used in conjunction with soft labels to calculate the semantically aware image-text contrast loss: .

[0095] This approach allows the model to assign intermediate weights to some "semantically similar" negative samples, thereby avoiding overly extreme training signals.

[0096] Furthermore, considering the objective situation of multiple sequences in MRI images, in order to achieve more granular and fine-grained image-text alignment, in addition to calculating the contrast loss for the multi-sequence fusion image features and the corresponding text features, this embodiment of the invention will also apply the same loss calculation process to each sequence and the corresponding text, and add it with the multi-sequence fusion contrast loss in a weighted manner to jointly assist in multimodal image-text alignment.

[0097] Further steps in training a visual language model based on a text generation task include: 1. Perform feature enhancement processing on the unified visual representation to construct image memory. Feature enhancement processing includes extracting global image memory and fusing multi-sequence local features.

[0098] Specifically, a feedforward neural network is used to deeply encode the representation, transforming the unified visual representation into a global image memory capable of representing the overall semantics of the image. Simultaneously, independent features of each sequence (T1, T2, etc.) are extracted from the unified visual model. These sequence features are then nonlinearly transformed through independent adapter layers to extract their structured local representations. All these local features are then projected onto the same dimension and concatenated, and integrated through a multi-granularity feature fusion network to form a local detail memory. The global image memory and the local detail memory are then fused (e.g., added) to form the final image memory. This memory integrates global diagnostic information and local details from different sequences, laying a solid visual foundation for generating comprehensive and accurate reports.

[0099] 2. Convert the real diagnostic text into a target token sequence, and construct the decoder input sequence based on the target token.

[0100] Specifically, the real diagnostic text is segmented into words, converted into a token ID sequence, and a start symbol [SOS] and an end symbol [EOS] are added to the beginning and end, respectively, forming the target sequence Y = {[SOS], y1, y2, ..., yT, [EOS]}. The target sequence Y is shifted one position to the right, and the ending [EOS] is removed, forming the decoder input sequence X = {[SOS], y1, y2, ..., yT}. Thus, during training, the model's input at time t is [SOS], y1, ..., y_{t-1}, and the next word to be predicted is... .

[0101] 3. Input the decoder input sequence and image memory into the text generation module and perform training in an autoregressive manner. At each time step, the text generation module predicts the probability distribution of the next token based on the image memory and the generated preceding token.

[0102] Specifically, the decoder input sequence X and the image memory are simultaneously input into the text generation module (a Transformer decoder). At each step (time step t), the decoder processes the currently generated token sequence ([SOS],..., y_{t-1}) through the self-attention mechanism to understand the language context and syntactic structure. Through the cross-attention mechanism, it queries the image memory to retrieve the visual information most relevant to the current generation context. Finally, based on the understanding of the language context and visual information, the decoder outputs a probability distribution P(y_t|y_{<t}, image memory) over the entire vocabulary to predict the next most likely token.

[0103] 4. Calculate the cross-entropy loss between the probability distribution and the true target token sequence, and update the parameters of the text generation module based on this loss.

[0104] Specifically, for each position in the sequence, calculate the cross-entropy loss between the probability distribution P(y_t) predicted by the model and the true next token y_t. The calculation formula of this text generation loss function is as follows: ; where denotes summing over the positions of non-padding tokens to avoid the influence of invalid tokens on the gradient. This loss function not only encourages the model to accurately predict the correct vocabulary at each step but also prompts it to better learn the structural rules of language generation.

[0105] In this embodiment, the text generation process not only depends on the overall features of the fused image but also fully integrates the fine-grained information provided by each image sequence to achieve high-quality image-text generation ability. First, the model extracts the global and local features of the fused image through the main visual encoder. The global features capture the overall semantics of the entire image, while the local features retain the local details in the spatial structure. These features constitute the initial image memory representation required for text generation.

[0106] Meanwhile, the model does not ignore the importance of multi-sequence images. For each type of sequence feature, the model is equipped with an independent sequence feature adaptation module to normalize and non-linearly transform its features respectively, and extract structured local representations from them. These local features from different sequences are unified projected to the same dimension and then concatenated as an additional visual supplementary signal. To fully fuse these multi-source information, the model introduces a multi-granularity feature fusion network, which is constructed through a series of linear transformations and activation functions and can capture the complementary relationships between different sequence features to enhance the expressive ability of the overall image memory.

[0107] After obtaining the final image memory, text generation is accomplished by a set of attention-based decoder modules. The model first takes the initial marker as input and, at each time step, understands the linguistic history context through a self-attention mechanism based on the current generation sequence state. Then, it uses a cross-attention module to retrieve visual information related to the current semantics from the image memory. This mechanism ensures that the generation process not only follows the grammatical structure of the language but also closely relates to the actual lesions, structures, or functional manifestations in the image in terms of content.

[0108] Furthermore, the model employs temperature scaling during token generation to control the entropy of the prediction distribution, thereby affecting the diversity of generated tokens. To further improve generation quality, a top-k and top-p (nucleus sampling) strategy is introduced during inference. Low-probability terms are removed during token sampling, and the next token is selected only from words with high cumulative probability. This strategy effectively suppresses redundancy and uncertainty in language generation while maintaining content consistency.

[0109] Furthermore, after the visual language model pre-training is completed, a downstream task fine-tuning step is included. The downstream tasks include at least one of brain tumor classification, brain tumor segmentation, and prognostic analysis. Specifically, the brain tumor classification task is fine-tuned by adding a classification head and employing a progressive thawing strategy; the brain tumor segmentation task is fine-tuned by adding a decoder that includes multi-scale upsampling and skip connection mechanisms; and the prognostic analysis task is fine-tuned by constructing a risk regression branch that includes multi-sequence feature fusion and temporal dimension modeling.

[0110] Specifically, within the framework of multimodal medical image understanding, fine-tuning strategies for specific downstream tasks are crucial, significantly improving the practicality and accuracy of the model in clinical applications such as brain tumor classification, segmentation, and prognostic analysis. This stage builds upon the pre-trained model, fully leveraging its cross-modal semantic alignment capabilities acquired in large-scale image-text alignment learning, and through structural optimization, adapting it to the specific needs of different tasks.

[0111] In brain tumor classification tasks, models typically use a global representation of fused images as input features. This representation already possesses strong semantic generalization capabilities during the pre-training phase. During fine-tuning, this global feature is input into a lightweight classification head, which is usually a combination of several layers of linear transformation and normalization modules. To prevent overfitting, parameters of the pre-training layers can be frozen in the early stages of training, updating only the classification head parameters. Subsequently, the deep backbone network is gradually unfrozen to achieve stable and efficient transfer learning. This strategy not only maintains the general image understanding capabilities acquired during pre-training but also improves the model's accuracy in identifying specific pathological types.

[0112] In brain tumor segmentation, the model needs to focus on spatially distributed details in the image, thus relying primarily on local representations and image patch features from the pre-trained model. During fine-tuning, a decoder structure is added to the original encoder output to restore the spatial resolution layer by layer. This decoder typically includes multi-scale upsampling modules and skip connections to enhance edge and structure preservation. Furthermore, to further enhance the localization of the segmentation target, the model can incorporate text guidance mechanisms, introducing linguistic cues about lesion type or location to guide the attention module to focus on specific regions, thereby improving segmentation accuracy and robustness.

[0113] The prognostic analysis task emphasizes the extraction and comprehensive understanding of long-term survival-related features, requiring the model to not only capture the morphological and distribution characteristics of lesions but also integrate semantic consistency and contextual information among multimodal images. In this task, the fine-tuning strategy focuses more on multi-sequence feature fusion and temporal dimension modeling. The model projects image features from different modalities into a unified representation space and extracts latent state features that evolve over time using a fully connected network or Transformer temporal modeling module. Finally, the model outputs survival probability or temporal stratification results through a risk regression branch. To alleviate the problem of imbalanced samples, a weighted loss function or a ranking-based optimization objective, such as Cox regression loss, can be introduced to improve the medical interpretability and generalization ability of the prediction.

[0114] This embodiment acquires multi-sequence magnetic resonance imaging data and corresponding diagnostic texts, providing a comprehensive learning foundation for the model. It then innovatively combines full-sequence standard training with missing-sequence distillation training. In the missing-sequence distillation training, fused features extracted from the full-sequence image data serve as teacher features, while features extracted from the missing-sequence image data serve as student features. A distillation loss function constrains the alignment of student and teacher features in the feature space. This design enables the model to handle arbitrary sequence combinations. Finally, through multi-task training involving image-text alignment learning and text generation, using unified visual representations as input and combining them with diagnostic texts, the model masters the complete mapping capability from images to diagnostic reports. This allows the trained model to extract effective visual representations from arbitrary sequence combinations, ensuring diagnostic stability even with missing sequences. The established unified feature space provides a high-quality visual foundation for multimodal diagnosis, and the achieved end-to-end diagnostic capability enables the model to output complete diagnostic reports.

[0115] Figure 2 This diagram illustrates the functional modules of an embodiment of the large-scale model pre-training system for brain tumor imaging diagnosis according to the present invention. Figure 2 As shown, the large-scale brain tumor imaging diagnostic model pre-training system includes: a first acquisition module 10, a first training module 11, and a second training module 12.

[0116] The first acquisition module 10 is used to acquire the image data and corresponding diagnostic text of brain tumor patients. The image data includes T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence.

[0117] The first training module 11 is used to construct a visual unification model. It uses image data to perform full-sequence standard training and missing-sequence distillation training on the visual unification model to learn the unified visual representation of image data with arbitrary sequence combinations. The missing-sequence distillation training uses the fusion features extracted from the full-sequence image data as teacher features and the features extracted from the missing-sequence image data as student features. The student features and teacher features are aligned in the feature space based on the distillation loss function.

[0118] The second training module 12 is used to construct a visual language model. It is based on a multi-task objective that includes at least image-text alignment learning and text generation. It uses the unified visual representation output by the trained visual unified model as input and combines diagnostic text to train the visual language model.

[0119] Optionally, after the first acquisition module 10 performs the operation of acquiring the image data and corresponding diagnostic text of the brain tumor patient, it is further used to: preprocess the image data; extract brain tumor diagnostic information and lesion area description from the diagnostic text, and construct a structured diagnostic text using the brain tumor diagnostic information and lesion area description.

[0120] Optionally, the visual unified model includes a shared encoder, an adapter layer, a feature interaction layer, and a decoder layer. The feature interaction layer includes a first fusion module for processing full sequence image data and a second fusion module for missing sequence image data. The first fusion module is built based on a multilayer perceptron structure, and the second fusion module is built based on a multi-head attention mechanism.

[0121] Optionally, the first training module 11 performs full-sequence standard training on the visual unified model using image data, specifically including: Step 1: using a shared encoder to extract the first feature representation corresponding to each sequence from the first image data including all sequences; Step 2: using an adapter layer to adjust the feature dimension and distribution of the first feature representation of each sequence; Step 3: using a first fusion module to integrate the first feature representations with adjusted feature dimension and distribution to obtain the first fused feature; Step 4: using a decoder to reconstruct the image based on the first fused feature; Step 5: using backpropagation to update the parameters of the shared encoder, adapter layer, first fusion module, and decoder layer based on the reconstructed image, the first image data, and the first preset loss function; Steps 1 to 5 are executed repeatedly until the preset number of training iterations is reached.

[0122] Optionally, the first preset loss function is obtained by weighted summation of L1 loss, L2 loss and structural similarity loss.

[0123] Optionally, the first training module 11 performs the operation of distilling missing sequences into the visual unification model using image data, specifically including: Step 6: Constructing second image data with missing sequences using the first image data and determining the number of sequences in the second image data; Step 7: Extracting the second feature representation corresponding to each sequence from the second image data using a shared encoder; Step 8: Adjusting the feature dimension and distribution of the second feature representation of each sequence using an adapter layer; Step 9: Controlling the second fusion module to adaptively adjust the number of attention heads according to the number of sequences, and then integrating the second feature representations with adjusted feature dimensions and distribution to obtain the second fused feature; Step 10: Using the first fused feature as the teacher feature and the second fused feature as the student feature, and combining it with a preset distillation loss function, performing distillation training on the second fusion module.

[0124] Optionally, before the first training module 11 performs the operation of controlling the second fusion module to adaptively adjust the number of attention heads according to the number of sequences, it is also used to: assign an independent embedding vector to the second feature representation corresponding to each sequence.

[0125] Optionally, the distillation loss function is obtained based on a weighted sum of L1 loss, L2 loss, first cosine similarity loss, and KL divergence loss.

[0126] Optionally, the visual language model includes an image feature encoder, a text encoder, and a text generation module.

[0127] Optionally, the second training module 12 performs the operation of training a visual language model based on a contrastive learning task, specifically including: inputting a unified visual representation or a single image feature of each sequence into an image feature encoder to extract an image feature matrix; inputting diagnostic text corresponding to the unified visual representation or the single image feature of each sequence into a text encoder to extract a text feature matrix; calculating the second cosine similarity between all text pairs in the text feature matrix, and constructing a soft label matrix based on the second cosine similarity and a preset similarity threshold, wherein the soft label matrix is ​​represented as: ; in, This is a soft label matrix. and These represent different diagnostic texts. To preset the similarity threshold, For preset temperature parameters, To preset the maximum soft tag value, Sigmoid function The second cosine similarity is used; the image feature matrix, text feature matrix, and soft label matrix are used to calculate the image-text contrast loss corresponding to the unified visual representation or the single image feature of each sequence; then the total image-text contrast loss is calculated based on the unified visual representation and the image-text contrast loss corresponding to the single image feature of each sequence, and the image feature encoder and text encoder are updated by backpropagation based on the total image-text contrast loss.

[0128] Optionally, the second training module 12 performs operations to train a visual language model based on a text generation task, specifically including: performing feature enhancement processing on the unified visual representation to construct an image memory, the feature enhancement processing including extracting global image memory and fusing multi-sequence local features; converting real diagnostic text into a target token sequence and constructing a decoder input sequence based on the target token; inputting the decoder input sequence and the image memory into the text generation module, and performing training in an autoregressive manner, wherein at each time step the text generation module predicts the probability distribution of the next token based on the image memory and the generated preceding token; calculating the cross-entropy loss between the probability distribution and the real target token sequence, and updating the parameters of the text generation module based on the loss.

[0129] Optionally, after the visual language model is pre-trained, the second training module 12 is also used to perform downstream task fine-tuning operations. The downstream tasks include at least one of brain tumor classification, brain tumor segmentation, and prognostic analysis. The brain tumor classification task is fine-tuned by adding a classification head and using a progressive thawing strategy. The brain tumor segmentation task is fine-tuned by adding a decoder that includes multi-scale upsampling and skip connection mechanisms. The prognostic analysis task is fine-tuned by constructing a risk regression branch that includes multi-sequence feature fusion and time dimension modeling.

[0130] For further details regarding the implementation techniques of each module in the large-scale brain tumor imaging diagnostic model pre-training system described in the above embodiments, please refer to the description in the large-scale brain tumor imaging diagnostic model pre-training method described in the above embodiments, which will not be repeated here.

[0131] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0132] Figure 3 This document illustrates a flowchart of an embodiment of the brain tumor diagnosis method based on a large-scale brain tumor imaging diagnostic model according to the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 3 As shown, the pre-training method for this large-scale brain tumor imaging diagnostic model includes: Step S200: Obtain image data of the object to be diagnosed, including at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence.

[0133] Step S201: Input the image data into the pre-trained visual unified model of one of the above embodiments to obtain a unified visual representation.

[0134] Step S202: Input the unified visual representation into the pre-trained visual language model of one of the above embodiments to obtain the brain tumor diagnosis result.

[0135] Further diagnostic results for brain tumors include: Brain tumor classification results, including primary and / or secondary classifications of the tumor; Brain tumor segmentation results, including the tumor core, edematous regions, and segmentation masks that enhance the tumor; Prognostic analysis results, including survival prediction or recurrence risk assessment; The diagnostic report includes a description of the lesion and a diagnostic conclusion.

[0136] This embodiment of the brain tumor imaging diagnostic method utilizes a unified visual model trained with missing sequence distillation. This enables the invention to effectively handle common clinical sequence loss situations. Even if a patient only possesses partial sequences from T1, T2, T1CE, and T2 FLAIR sequences, robust unified visual representations can be generated. This directly addresses the core pain point of diagnostic difficulties caused by sequence loss, significantly improving the method's clinical applicability and generalization ability. Furthermore, processing is performed using a visual language model trained with multi-task objectives (especially image-text alignment learning incorporating semantically aware soft labels), ensuring that the final diagnostic results are not only accurate but also highly interpretable. Crucially, this process can simultaneously output diversified diagnostic results, including classification, segmentation, prognostic analysis, and text reports. This demonstrates the comprehensive ability of the model constructed in this invention to perform comprehensive, multi-faceted analysis of brain tumors, effectively overcoming the limitations of traditional methods with their single function and insufficient information integration. It provides physicians with more comprehensive and multi-dimensional decision support, significantly improving the efficiency and accuracy of brain tumor auxiliary diagnosis.

[0137] Figure 4 This diagram illustrates the functional characteristics of one embodiment of the brain tumor diagnostic system based on a large-scale brain tumor imaging diagnostic model according to the present invention. Figure 4 As shown, the brain tumor diagnostic system based on a large model for brain tumor imaging diagnosis includes: a second acquisition module 20, a feature fusion module 21, and a result prediction module 22.

[0138] The second acquisition module 20 is used to acquire image data of the object to be diagnosed, including at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence.

[0139] The feature fusion module 21 is used to input image data into a pre-trained visual unified model to obtain a unified visual representation.

[0140] The result prediction module 22 is used to input the unified visual representation into the pre-trained visual language model to obtain the brain tumor diagnosis result.

[0141] Optionally, the brain tumor diagnostic results include: brain tumor classification results, including primary and / or secondary classification of the tumor; brain tumor segmentation results, including the tumor core, edematous region, and segmentation mask that enhances the tumor; prognostic analysis results, including survival prediction or recurrence risk assessment; and a diagnostic text report containing a description of the lesion and a diagnostic conclusion.

[0142] For further details regarding the implementation of the technical solutions for each module in the brain tumor diagnostic system based on the large model of brain tumor imaging diagnosis in the above embodiments, please refer to the description in the brain tumor diagnostic method based on the large model of brain tumor imaging diagnosis in the above embodiments, which will not be repeated here.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0144] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 5 As shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. The memory 32 stores program instructions. When the program instructions are executed by the processor 31, the processor 31 performs the method steps of any of the above-described embodiments of the large-scale brain tumor imaging diagnostic model pre-training method or performs the steps of any of the above-described embodiments of the brain tumor diagnosis method based on the large-scale brain tumor imaging diagnostic model.

[0145] The processor 31 can also be referred to as a Central Processing Unit (CPU). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0146] See Figure 6 , Figure 6 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. The storage medium of this embodiment stores program instructions 61 capable of implementing the above-described pre-training method for a large-scale brain tumor imaging diagnostic model or executing the brain tumor diagnosis method based on the above-described large-scale brain tumor imaging diagnostic model. These program instructions 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or computer devices such as computers, servers, mobile phones, and tablets.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed computer devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0148] Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for pre-training a large-scale model for brain tumor imaging diagnosis, characterized in that, It includes: Acquire imaging data and corresponding diagnostic text of patients with brain tumors, wherein the imaging data includes T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; A visual unification model is constructed, and the image data is used to perform full-sequence standard training and missing sequence distillation training on the visual unification model to learn the unified visual representation of image data with arbitrary sequence combinations. The missing sequence distillation training uses the fusion features extracted from the full-sequence image data as teacher features and the features extracted from the missing sequence image data as student features. The student features and teacher features are aligned in the feature space based on the distillation loss function. A visual language model is constructed based on a multi-task objective that includes image-text alignment learning and text generation. The unified visual representation output by the trained visual unified model is used as input, and the visual language model is trained in combination with the diagnostic text. The visual language model includes an image feature encoder, a text encoder, and a text generation module. The steps for training the visual language model based on image-text alignment learning include: The unified visual representation is input into the image feature encoder to extract the first image feature matrix; the diagnostic text corresponding to the unified visual representation is input into the text encoder to extract the first text feature matrix. The single image feature of each sequence is input into the image feature encoder to extract the second image feature matrix; the diagnostic text corresponding to the single image feature of each sequence is input into the text encoder to extract the second text feature matrix. Calculate the second cosine similarity between all text pairs in the first text feature matrix, and construct a soft label matrix corresponding to the first text feature matrix based on the second cosine similarity corresponding to the first text feature matrix and a preset similarity threshold; Calculate the second cosine similarity between all text pairs in the second text feature matrix, and construct a soft label matrix corresponding to the second text feature matrix based on the second cosine similarity corresponding to the second text feature matrix and a preset similarity threshold; The image-text contrast loss corresponding to the unified visual representation is calculated using the first image feature matrix, the first text feature matrix, and the soft label matrix corresponding to the first text feature matrix. The image-text contrast loss corresponding to a single image feature of each sequence is calculated using the second image feature matrix, the second text feature matrix, and the soft label matrix corresponding to the second text feature matrix. The total image-text contrast loss is calculated based on the image-text contrast loss corresponding to the unified visual representation and the image-text contrast loss corresponding to the single image feature of each sequence, and the image feature encoder and the text encoder are updated by backpropagation based on the total image-text contrast loss.

2. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 1, characterized in that, After acquiring the imaging data and corresponding diagnostic text of the brain tumor patient, the process also includes: The image data is preprocessed; Brain tumor diagnostic information and lesion region description are extracted from the diagnostic text, and a structured diagnostic text is constructed using the brain tumor diagnostic information and lesion region description.

3. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 1, characterized in that, The visual unified model includes a shared encoder, an adapter layer, a feature interaction layer, and a decoder layer. The feature interaction layer includes a first fusion module for processing full-sequence image data and a second fusion module for processing missing sequence image data. The first fusion module is built based on a multilayer perceptron structure, and the second fusion module is built based on a multi-head attention mechanism.

4. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 3, characterized in that, The step of performing full-sequence standard training on the visual uniform model using the image data includes: Step 1: Use the shared encoder to extract the first feature representation corresponding to each sequence from the first image data including all sequences; Step 2: Use the adapter layer to adjust the feature dimension and distribution of the first feature representation of each sequence; Step 3: Use the first fusion module to integrate the first feature representations with adjusted feature dimensions and distribution to obtain the first fused feature; Step 4: Reconstruct the image using the decoder based on the first fusion feature; Step 5: Based on the reconstructed image, the first image data, and the first preset loss function, backpropagate to update the parameters of the shared encoder, the adapter layer, the first fusion module, and the decoder layer; Repeat steps one through five until the preset number of training iterations is reached.

5. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 4, characterized in that, The first preset loss function is obtained by weighted summation of L1 loss, L2 loss and structural similarity loss.

6. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 4, characterized in that, Training the visual unification model using the image data to perform missing sequence distillation includes: Step 6: Construct second image data containing missing sequences using the first image data, and determine the number of sequences in the second image data; Step 7: Use the shared encoder to extract the second feature representation corresponding to each sequence from the second image data; Step 8: Adjust the feature dimension and distribution of the second feature representation of each sequence using the adapter layer; Step 9: Control the second fusion module to adaptively adjust the number of attention heads according to the number of sequences, and then integrate the second feature representation with adjusted feature dimensions and distribution to obtain the second fused feature; Step 10: Using the first fusion feature as the teacher feature and the second fusion feature as the student feature, the second fusion module is trained by distillation using a preset distillation loss function.

7. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 6, characterized in that, Before step nine, the following are also included: Each sequence is assigned an independent embedding vector for its corresponding second feature representation.

8. The method for pre-training a large model for brain tumor imaging diagnosis according to claim 6, characterized in that, The distillation loss function is obtained by weighting the L1 loss, L2 loss, first cosine similarity loss, and KL divergence loss.

9. The method for pre-training a large-scale model for brain tumor imaging diagnosis according to claim 1, characterized in that, A soft-label matrix is ​​constructed based on the second cosine similarity and a preset similarity threshold. The soft-label matrix is ​​represented as follows: ； in, This is a soft label matrix. and These represent different diagnostic texts. To preset the similarity threshold, For preset temperature parameters, To preset the maximum soft tag value, For the Sigmoid function, This represents the second cosine similarity.

10. The method for pre-training a large model for brain tumor imaging diagnosis according to claim 8, characterized in that, The steps for training the visual language model based on the text generation task include: The unified visual representation is subjected to feature enhancement processing to construct image memory, the feature enhancement processing including extracting global image memory and fusing multi-sequence local features; The acquired real diagnostic text is converted into a target token sequence, and a decoder input sequence is constructed based on the target token sequence. The decoder input sequence and the image memory are input into the text generation module, and training is performed in an autoregressive manner, wherein the text generation module predicts the probability distribution of the next token at each time step based on the image memory and the generated preceding token; Calculate the cross-entropy loss between the probability distribution and the real target token sequence, and update the parameters of the text generation module based on the cross-entropy loss.

11. The method for pre-training a large model for brain tumor imaging diagnosis according to claim 1, characterized in that, After the visual language model is pre-trained, a downstream task fine-tuning step is also included, wherein the downstream task includes at least one of brain tumor classification task, brain tumor segmentation task, and prognostic analysis task. The brain tumor classification task was fine-tuned by adding a classification head and using a progressive thawing strategy. The brain tumor segmentation task was fine-tuned by adding a decoder that includes multi-scale upsampling and skip connection mechanisms. The prognostic analysis task is fine-tuned by constructing a risk regression branch that incorporates multi-sequence feature fusion and time-dimensional modeling.

12. A brain tumor diagnosis method based on a large-scale brain tumor imaging diagnostic model, wherein the large-scale brain tumor imaging diagnostic model is trained by the large-scale brain tumor imaging diagnostic model pre-training method according to any one of claims 1 to 11, characterized in that, It includes: Acquire imaging data of the object to be diagnosed, wherein the imaging data includes at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; The image data is input into a pre-trained unified visual model to obtain a unified visual representation; The unified visual representation is input into a pre-trained visual language model to obtain brain tumor diagnosis results.

13. The brain tumor diagnosis method based on a large-scale brain tumor imaging diagnostic model according to claim 12, characterized in that, The brain tumor diagnosis results include: Brain tumor classification results, including primary and / or secondary classifications of the tumor; Brain tumor segmentation results, including the tumor core, edematous regions, and segmentation masks that enhance the tumor; Prognostic analysis results, including survival prediction or recurrence risk assessment; The diagnostic report includes a description of the lesion and a diagnostic conclusion.

14. A large-scale model pre-training system for brain tumor imaging diagnosis, characterized in that, The system is used to perform the large model pre-training method for brain tumor imaging diagnosis according to any one of claims 1 to 11, the system comprising: The first acquisition module is used to acquire image data and corresponding diagnostic text of brain tumor patients. The image data includes T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence. The first training module is used to construct a visual unification model. The visual unification model is trained using the image data to perform full-sequence standard training and missing-sequence distillation training to learn the unified visual representation of image data with arbitrary sequence combinations. The missing-sequence distillation training uses the fusion features extracted from the full-sequence image data as teacher features and the features extracted from the missing-sequence image data as student features. The student features and teacher features are aligned in the feature space based on the distillation loss function. The second training module is used to construct a visual language model. Based on a multi-task objective that includes at least image-text alignment learning and text generation, the visual language model is trained by taking the unified visual representation output by the trained visual unified model as input and combining it with the diagnostic text.

15. A brain tumor diagnostic system based on a large-scale brain tumor imaging diagnostic model, characterized in that, The system is used to perform the brain tumor diagnosis method based on a large model of brain tumor imaging diagnosis as described in any one of claims 12 to 13, the system comprising: The second acquisition module is used to acquire image data of the object to be diagnosed, wherein the image data includes at least one of T1 sequence, T2 sequence, T1CE sequence, and T2 FLAIR sequence; The feature fusion module is used to input the image data into a pre-trained visual unified model to obtain a unified visual representation; The result prediction module is used to input the unified visual representation into a pre-trained visual language model to obtain brain tumor diagnosis results.

16. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor. The memory stores program instructions that, when executed by the processor, cause the processor to perform the steps of the pre-training method for a large model for brain tumor imaging diagnosis as described in any one of claims 1-11, or the steps of the brain tumor diagnosis method based on a large model for brain tumor imaging diagnosis as described in any one of claims 12-13.

17. A computer storage medium, characterized in that, The system stores program instructions capable of implementing the pre-training method for a large-scale brain tumor imaging diagnostic model as described in any one of claims 1-11, or executing the brain tumor diagnostic method based on a large-scale brain tumor imaging diagnostic model as described in any one of claims 12-13.

Images