Meningioma ki67 classification method based on multi-modal medical data

Abstract

The application discloses a meningioma Ki67 classification method based on multi-modal medical data, applied to the field of image processing, aiming at the image data, radiology data and radiomics data based on 3D MRI with multiple modes during meningioma Ki67 prediction, and the existing technology has the problem of too large calculation overhead during multi-modal fusion; the application significantly reduces the calculation complexity through iterative modal information fusion of structured latent space and cross attention mechanism. It effectively captures the complementary information between modes through dynamic weight distribution and iterative refinement, while suppressing redundant noise, avoiding the feature dilution problem caused by mode length difference or simple superposition in the traditional splicing method; the compact representation of the latent space forces different modes to interact efficiently in the shared dimension, not only retaining the key patterns of cross-modal correlation, but also adaptively filtering irrelevant content through the attention mechanism, thereby achieving a better balance in information density and semantic consistency, especially suitable for processing real scenes with mode imbalance or partial loss.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a Ki67 classification technique for meningiomas. Background Technology

[0002] Meningiomas are the most common primary intracranial tumors, with an annual incidence of 9.73 per 100,000 individuals. Magnetic resonance imaging (MRI) is a cornerstone of patient management as it plays a crucial role in tumor diagnosis, follow-up, and treatment planning. For symptomatic meningiomas, total surgical resection, including / without adjuvant radiotherapy, is recommended, as it often leads to a favorable prognosis. Conversely, small, asymptomatic meningiomas (maximum diameter less than 3 cm) are conservatively managed through radiological follow-up, with the decision to initiate active intervention based on the development of clinical symptoms or radiological evidence of tumor growth. Recognizing the importance of prognostic risk stratification and treatment decisions, there is a growing emphasis on achieving personalized patient management and prognostic assessment. Recent studies have provided accumulating evidence regarding the clinical significance of the KI-67 index in personalized patient care. As a classic biomarker of cell proliferation, it can be used to identify patients with small, asymptomatic meningiomas requiring early intervention. The gold standard for KI-67 assessment remains immunohistochemical staining of tumor specimens obtained from biopsy or surgery. However, the invasiveness of these methods imposes a significantly increased physical burden and risk of complications.

[0003] Previous studies have identified a range of radiographic features on conventional MRI that are independently associated with Ki-67 expression. Small amounts of data obtained from advanced MR techniques (e.g., the relatively obvious diffusion coefficient (RADC)) may also potentially serve as independent predictors. However, accurate prediction by humans remains challenging due to the controversial nature and imprecision of these indicators, their reliance on clinical specialization, and the routine implementation of advanced MR techniques in patient management (A multi-modal deep learning model for prediction of Ki-67 for meningiomas using pretreatment MR images). In recent years, the development of deep learning has facilitated advancements in multimodal learning, making the fusion of multiple modalities to enhance classification performance an effective approach. The advantage of multimodal learning over unimodal learning lies in its ability to mimic the human approach of comprehensively utilizing multiple senses to understand the world. By simultaneously processing different forms of data such as text, images, and sound, it captures the richness and relevance of information more comprehensively. For example, when analyzing a video, relying solely on images may not accurately identify dialogue within background noise, while combining audio signals can significantly improve comprehension. Similarly, in medical diagnosis, X-rays alone may struggle to determine the nature of lesions, but combining pathology reports and clinical records allows for more reliable judgments. Multimodal learning not only overcomes the limitations of single data sources but also uncovers deeper patterns from interactions between different modalities, thus exhibiting stronger adaptability and robustness in complex tasks. This fusion of perception makes AI systems closer to the flexibility and accuracy of human cognition. However, while multimodal data brings new opportunities, it also presents problems such as misaligned distribution of modal information and difficulties in effective interaction. For ki67 prediction of naturally multimodal 3D MRI image data, radiological data, and radiomics data, traditional methods mainly employ manual prediction or single-modal prediction methods based on MRI image data or image data including radiological and radiomics data. These methods either consume enormous human resources or can only utilize single-modal data, failing to fully explore the potential of joint learning across multiple modalities. The emergence of the Transformer architecture and attention has driven the development of deep learning and also provided a solution for efficient multimodal alignment and interaction.

[0004] Existing technology 1 employs a multimodal sentiment recognition method based on contrastive learning and a Transformer architecture. Its core is to extract image and text features using lightweight models (MobileViT and MobileBERT) respectively, and then utilize contrastive learning to achieve modality alignment. First, the multimodal dataset is preprocessed, with sentiment labels unified through a voting mechanism, and divided into training, validation, and test sets. Image data is processed by MobileViT to extract visual features, while text data is converted into word vectors using MobileBERT to extract semantic features. Next, the cosine similarity between image and text features is calculated, and the image-text contrastive loss (ITC) is used to maximize positive sample similarity and minimize negative sample similarity, thus achieving modality alignment. The aligned features are concatenated and input into a multi-layer Transformer encoder for fusion, generating an initial fused vector representation. In the sentiment classification stage, random dropout is used to process the fused vector, and sentiment probabilities are output through fully connected layers and Softmax, with cross-entropy loss calculated. Furthermore, positive and negative samples are generated through image data augmentation (flipping, noise reduction, etc.) and text random dropout, combined with contrastive loss based on angle edges to further optimize the model's discriminative ability. The total loss consists of image-text contrast loss, classification loss, and enhancement contrast loss, which are jointly optimized by the Adam optimizer. The system includes modules for data extraction, feature alignment, Transformer fusion, and loss calculation, and supports end-to-end training and testing.

[0005] Existing technique one employs a direct concatenation of aligned image and text features during the feature fusion stage. While simple and efficient, this method has significant limitations. Specifically, it fails to establish an effective mechanism to explicitly distinguish and utilize complementary information between different modalities, such as the semantic association between facial expression features in images and emotional keywords in text, and it also fails to effectively filter redundant information. The angle edge contrast loss relies on a preset angle bias (M), requiring parameter tuning for different datasets and limiting its cross-scene generalization. For deep fusion of multimodal information, technique one uses a stacked transformer layer structure to process the concatenated token sequence and employs standard attention computation, resulting in a model with high quadratic computational complexity.

[0006] In summary, research on Ki67 meningioma based on multimodal information remains lacking. Furthermore, common multimodal information fusion methods directly concatenate and align various modalities, failing to establish effective mechanisms to explicitly distinguish and utilize complementary and redundant information between different modalities. This significantly limits model performance. Moreover, these methods relying on standard transformer structures for deep modal information fusion typically depend on complex feature extractors and incur substantial computational overhead. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a classification method for meningiomas Ki67 based on multimodal medical data. Through the design of text and image encoder structures, it effectively mines radiological information and the global dependencies between radiological information.

[0008] The technical solution adopted in this invention is: a Ki67 classification method for meningiomas based on multimodal medical data, comprising:

[0009] S1. Collect multimodal medical data, which includes: image data, radiological data and corresponding label information;

[0010] S2. Construct the encoder and pre-train it using the multimodal medical data collected in step S1. The encoder specifically includes a text encoder and an image encoder.

[0011] The pre-training process is as follows:

[0012] Image data is input into a text encoder to obtain text features; image data is input into an image encoder to obtain image features.

[0013] The text feature, image feature, and label data encoding results are concatenated, trained through supervised contrastive learning, and the weight files of the trained text encoder and image encoder are saved.

[0014] S3. Construct a multimodal fusion network, specifically including multiple cascaded Transformer blocks. The input of the first Transformer block is the query vector Q, and the input of subsequent Transformer blocks is the output of the previous Transformer block and a modal feature extracted by the pre-trained encoder in step S2. The outputs of each Transformer block between the first and last Transformer block are used as auxiliary classification vectors, and the output of the last Transformer block is the fused classification vector. The number of cascaded Transformer blocks is the sum of the number of text feature and image feature categories extracted by the pre-trained encoder in step S2 plus 1.

[0015] S4. Construct a classification network; input the fused classification vector and each auxiliary classification vector into the classification network to train the classification network;

[0016] S5. The multimodal medical data of the object to be processed is processed sequentially through the encoding obtained by the pre-training in step S2 and the multimodal fusion network constructed in step S3. The fused classification vector corresponding to the object to be processed and each auxiliary classification vector are input into the classification network trained in step S4 to obtain the classification result.

[0017] The beneficial effects of this invention are as follows: The technical solution of this invention effectively mines radiological information and the global dependencies between image information through the designed text and image encoder structures. Significant technical breakthroughs have been achieved in the Ki67 classification task for meningiomas through a structured latent space and cross-attention mechanism. Its core innovation lies in the adoption of an iterative multimodal fusion strategy. Through dynamic weight allocation and feature refinement, it effectively distinguishes and strengthens complementary information between different modalities (such as local anatomical details of 3D MRI and global functional features of radiomics data), while automatically filtering redundant content (such as background noise shared by multiple modalities). This mechanism solves the feature dilution problem caused by traditional stitching fusion methods, especially for common modal imbalance scenarios in medical imaging (such as the fusion of high-dimensional MRI data and low-dimensional clinical text). The compact representation of the latent space forces cross-modal features to interact efficiently in a unified dimension, providing a more reliable automated tool for personalized clinical prognostic assessment. Simultaneously, the auxiliary classification of the multimodal fusion information results enhances the model's ability to capture modality-specific features, while improving the generalization of the main classification task. Attached Figure Description

[0018] Figure 1 A classification method for meningiomas based on multimodal medical data (KI67);

[0019] Figure 2 This is a schematic diagram of the text encoder described in this invention;

[0020] Figure 3 A schematic diagram of a 3D MRI image encoder;

[0021] Figure 4 This is a schematic diagram of the comparative learning described in this invention;

[0022] Figure 5 This is a schematic diagram of the multimodal feature fusion module described in this invention;

[0023] Figure 6 This is a schematic diagram of the multimodal classification module described in this invention;

[0024] Figure 7 This is a schematic diagram of a Transformer block. Detailed Implementation

[0025] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0026] This invention provides an innovative multimodal information fusion network that significantly reduces computational complexity through iterative modal information fusion using a structured latent space and a cross-attention mechanism. By dynamically assigning weights and iteratively refining, it effectively captures complementary information between modalities (such as local details in images and global semantics in text) while suppressing redundant noise (such as recurring background features), avoiding feature dilution problems caused by modal length differences or simple superposition in traditional stitching methods. The compact representation of the latent space forces different modalities to interact efficiently on shared dimensions, preserving key patterns of cross-modal association (such as synchronization between actions in video and keywords in speech) while adaptively filtering irrelevant content through the attention mechanism. This achieves a better balance between information density and semantic consistency, making it particularly suitable for handling real-world scenarios with modal imbalances or partial missing features. Furthermore, an image encoder is designed for 3D MRI image data to effectively extract local features and mine global information, resulting in better feature representation.

[0027] Step 1 involves preprocessing the raw input data, primarily including center cropping, flipping, scaling, and normalization of the 3D MRI data, and missing value removal and normalization of the imaging data. The imaging data specifically includes radiological data and radiomics data.

[0028] The experimental data in this embodiment includes data from 349 patients. Data from 296 patients was split into training and validation sets at an 8:2 ratio, and the remaining 53 patients' data served as the test set. Each patient's data includes radiological and radiomics data, as well as corresponding label information. Specifically, the label information indicates whether the tumor is benign or malignant. The collected data is used for the pre-training process in step 2 and the classification network training process in step 4.

[0029] To reduce the training burden, the ROI regions are cropped from the training set data based on tumor label information. To accelerate model convergence and enhance the consistent representation of 3D MRI data, intensity normalization based on the foreground region is employed. Only the mean and standard deviation of non-zero pixels within the volume are calculated, and Z-score normalization is performed on the non-zero regions. The calculation process is shown below:

[0030]

[0031] Where μ is the mean of the non-zero pixels, σ adj This is the adjusted standard deviation. The volume represents the input 3D MRI data.

[0032] To improve the model's generalization ability, prevent overfitting, and enhance robustness, the image data underwent cropping, scaling, and flipping. Similarly, to improve the model's training efficiency, stability, and generalization ability, Z-score normalization was also applied to the image data.

[0033] Step 2: Pre-train the contrastive learning representation model.

[0034] All modal data, after being processed in step 1, are used to construct a pre-trained encoder in subsequent supervised contrastive learning.

[0035] Step 2.1, Build a text encoder

[0036] The structure of a text encoder is as follows Figure 2 As shown, this network employs a Transformer-based embedding architecture, primarily used for feature extraction and representation learning of clinical text information. The core of the network consists of two fully connected layers. First, the first fully connected layer projects the input clinical features from the original dimensions to a 1024-dimensional latent space. This process introduces a non-linear transformation through the ReLU activation function to enhance feature representation. Subsequently, the second fully connected layer further compresses the 1024-dimensional intermediate features to the target dimension (typically set to 256). To effectively mine the interdependencies between textual information, we further mine global dependency information through a transformer layer, forming the final semantic embedding representation. To prevent overfitting, a Dropout layer is applied before the output for random deactivation.

[0037] Step 2.2 Constructing a 3D MRI image encoder

[0038] The 3D MRI image encoder employs a hierarchical structure, primarily designed for processing 3D medical image data. The network input is a 3D MRI image, which undergoes a carefully designed five-layer architecture to progressively extract and compress features. The first layer consists of two consecutive 3D convolutional modules. Spatial features are first extracted using a 3×3×3 depthwise separable convolution, enhancing local feature representation while maintaining input resolution. Subsequently, a 1×1×1 point convolution maps the channel dimensions to a 16-dimensional space, completing the initial feature transformation. The following four layers all use a unified convolutional block structure, with a 2×2×2 3D max-pooling layer at the beginning of each layer for spatial downsampling, halving the feature map size. Local augmentation is then performed through this four-layer convolutional block structure.

[0039] Each convolutional block structure contains two parallel branches: the left branch first enhances spatial feature interaction through a 3×3×3 depthwise convolution, and then adjusts the channel dimension through a 1×1×1 pointwise convolution; the right branch directly uses a 1×1×1 pointwise convolution for channel transformation. The outputs of the two branches are added element-wise to achieve feature fusion. In the left and right branches, the right branch adds a 1x1 convolution to the left branch to form a residual connection. This residual connection method effectively mines local features while mitigating the gradient vanishing problem. As the layer deepens, the number of feature channels increases exponentially layer by layer, expanding to 32, 64, 128, and 256 dimensions, forming a pyramid-shaped feature representation system.

[0040] The 3D MRI image encoder network also incorporates a special embedding transformation module. First, a 3D convolution divides the 256-channel feature map into spatial tokens of a preset block size. These tokens are then flattened and added to the learnable location encoding to inject spatial location information. To capture global long-range dependencies in the image and better represent image information, the image encoding after local feature mining is processed by a Transformer encoder for global context modeling. Finally, it is processed through an average pooling layer (i.e.,...) Figure 3 The AvgPooL layer shown) and the linear projection layer (i.e. Figure 3 The fully connected layer shown maps the feature dimensions uniformly to the specified hidden space. The entire network uses the LeakyReLU activation function (with a negative slope of 0.2) to enhance non-linear expressive power, and applies Dropout regularization with a probability of 0.2 before the output to prevent overfitting.

[0041] Step 2.3 Construct a contrastive learning framework and complete the training.

[0042] The encoded radiological data, radiomics data, and label data are concatenated to form imaging features. The concatenation of label information is primarily for constructing a supervised contrastive learning loss. The imaging features and radiological features are effectively trained using supervised contrastive learning, and the trained network weight file is saved. Specifically, the label data encoding results are mapped to a specified dimension through a fully connected layer. This specified dimension is the same dimension containing both the encoded radiological data and the radiomics data.

[0043] The concatenation here specifically involves: E.g., radiology data tensor([B,N1,C]), radiomics data tensor([B,N2,C]), and labels (B,1,C). A concat operation is performed along the second dimension to obtain the result (B,N1+N2+1,C). B represents the number of samples input for each training iteration, N1 and N2 represent the token length (specifically, the number of the smallest units processed in the text), and C represents the encoded dimension.

[0044] Image features To imaging features Comparative loss:

[0045]

[0046] The imaging features here are MRI features.

[0047] Contrast loss from radiographic feature to radiographic feature:

[0048]

[0049] Where τ is the temperature parameter, It represents the number of training batches.

[0050] like Figure 4 As shown, the loss for model training is calculated as follows:

[0051] L=λL i,t +(1-λ)L t,i

[0052] Where λ is a weighting parameter that balances the contrast loss between the two modes.

[0053] The pre-trained encoder is obtained through contrastive learning, which yields better initialization parameters. Subsequent classification is then performed based on the outputs of the text encoder and image encoder obtained in this pre-trained step.

[0054] Step 3: Construct a multimodal meningioma Ki67 classification model

[0055] like Figure 5 As shown, the multimodal fusion framework mainly consists of multiple cascaded Transformer blocks. Furthermore, to improve gradient flow, enhance feature representation, and regularize the model, this invention performs auxiliary classification on the query vector output by each modality. Multi-task constraints reduce the risk of overfitting and improve the performance and generalization ability of the main task.

[0056] like Figure 7 As shown, operations are performed within each Transformer block:

[0057] input2=input1+Attention(LayerNorm(input1))

[0058] input3=input2+MLP(LayerNorm(input2))

[0059] Here, input1 represents the input to the Transformer block, input2 represents the result of adding the output of Attention to input1, input3 represents the result of adding the output of the MLP to input2, LayerNorm represents layer normalization, and Attention represents attention operations, including self-attention and cross-attention. When self-attention is used, the input is a single variable, constructing its own interaction relationship; when cross-attention is used, the input is two variables, constructing the interaction relationship between them. MLP represents stacked linear layers, and residual operations help the model converge effectively.

[0060] Before the modal information is fused in each Transformer block, the modal encoder first performs self-attention (SA) calculation on each modal data X and the input query vector Q. Figure 2 , 3 All Transformer blocks marked with SA in section 5 must first undergo this process to ensure they have valid information. Specifically, this includes:

[0061]

[0062] in: All are learnable parameters. Different subscripts of W indicate different weights of the fully connected layer in the linear transformation. x1 represents the feature corresponding to the first modality data, x2 represents the feature corresponding to the second modality data, x3 represents the feature corresponding to the third modality data, q1 represents the weight of the fully connected layer corresponding to the linear transformation of the query vector Q when interacting with the first modality data, q2 represents the weight of the fully connected layer corresponding to the linear transformation of the query vector Q when interacting with the second modality data, q3 represents the weight of the fully connected layer corresponding to the linear transformation of the query vector Q when interacting with the third modality data, d represents the encoding dimension, and s X This represents the result of self-attention computation on the modal feature X (via the SA transformer), s Q This is the result of the query vector self-attention calculation.

[0063] After self-attention processing, cross-attention (CA) is iteratively performed on each modality information within the transformer block with the query vector (Latent Query). Figure 5 Transformer blocks marked with SA must be executed first to inject modal information into the query vector. This process includes:

[0064]

[0065] in: All of these are learnable parameters.

[0066] By gradually fusing information from multiple modalities into the query vector Q through a cascaded Transformer network, the final query vector Q is used as input to a classification network for classification prediction. This iterative interaction method employs latent space projection and cross-attention mechanisms to achieve finer information integration during multimodal fusion. Through a learnable interaction space, features from different modalities can achieve deep collaboration: fine-grained visual features and abstract linguistic concepts form a dynamic coupling in the compressed dimension, while attention weights automatically weaken repetitive or irrelevant content in each modality. This design solves the modal suppression phenomenon caused by the crude splicing of traditional methods, enabling complementary features (such as the motion of objects in video frames and the temporal changes of corresponding narration) to establish accurate associations, while eliminating redundant information, ultimately achieving a dual optimization of tight semantic coupling and noise filtering in cross-modal representations.

[0067] Step 4, train the classification network

[0068] like Figure 6 As shown, the classification network employs a simple two-layer fully connected structure to map the input deep features to the final classification result. The network first processes the input features through a dimension-preserving fully connected layer (fc1), where both input and output dimensions are predefined, and LeakyReLU is used as the activation function, with a negative slope coefficient set to 0.2 to mitigate neuron death. After feature transformation, a Dropout layer is applied for regularization to prevent overfitting. Subsequently, a second fully connected layer (fc2) reduces the feature dimension to 2, directly outputting the final classification logits. For weight initialization, both fully connected layers use a Xavier uniform initialization strategy, while the bias term is initialized using a normal distribution with a standard deviation of 1e-6. This initialization scheme facilitates stable gradient propagation in the early stages of training. The entire network structure is concise and efficient. Through a combination of nonlinear transformations and regularization operations, it achieves discriminative modeling of deep features, making it suitable for binary classification tasks. The specific formula is as follows:

[0069] h1 = LeakyReLU 0.2 (W1X+b1)

[0070] y = x²h₁ + b²

[0071] Where h1 is the hidden layer dimension, y represents the model's logical output, and W... i and b iAll of these are learnable parameters, with W1 being the weight and b1 and b2 being the biases.

[0072] Finally, the classification result is obtained using the softmax function.

[0073]

[0074] Among them, y i This represents the probability of predicting class i after passing through two fully connected layers (without softmax processing). Similarly, y... j This represents the predicted class j probability after passing through two fully connected layers (without softmax processing).

[0075] Step 5, Determine the loss function

[0076] The model is trained using cross-entropy loss, a common loss mechanism in classification tasks. The model output includes two auxiliary losses and a final classification result loss. The specific steps are as follows:

[0077]

[0078] in, The auxiliary classification loss represents the image modality. This represents the auxiliary classification loss that includes both image modalities and radiological modalities. This represents the fusion classification loss that includes three modalities. The auxiliary classification output is defined by image modality and the auxiliary classification output includes both image modality and radiological modality data, respectively. Figure 5 The two auxiliary classification vectors in the data. This represents the fusion classification output containing three modalities, i.e., the corresponding Figure 5 The final classification vector is obtained by minimizing the loss L during network training.

[0079] The entire training process employs a Stochastic Gradient Descent (SGD) optimizer for model parameter updates. This optimizer is configured with a base learning rate of 3e-5 and introduces a momentum coefficient of 0.96 to accelerate convergence and smooth the parameter update trajectory. To control model complexity and prevent overfitting, an L2 regularization term (weight_decay = 1e-3) is set in the optimizer. This mechanism effectively constrains the magnitude of the weight parameters by adding a parameter norm penalty term to the loss function. Training lasts for 100 epochs, during which the learning rate is scheduled using a cosine annealing (Cosine AnnealingLR) strategy with a epoch length T_max of 20 epochs. Within each epoch, the learning rate decays from the initial value of 3e-5 to a minimum value of 0.05 according to a cosine curve.

[0080] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A Ki67 classification method for meningiomas based on multimodal medical data, characterized in that, include: S1. Collect multimodal medical data, which includes: image data, radiological data and corresponding label information; S2. Construct an encoder and pre-train it using the multimodal medical data collected in step S1. The encoder specifically includes a text encoder and an image encoder. The image encoder has a five-layer structure, with a 3D max pooling layer between adjacent layers. This layer is used to spatially downsample the output of the previous layer, halving the size of the feature map, and then using it as the input of the next adjacent layer. The first layer consists of two 3D convolutional modules connected in series. The last four layers have the same structure, each including two parallel branches, left and right. The left branch includes a depthwise convolution and a first point convolution in sequence, and the right branch includes a second point convolution. The outputs of the two branches are added element-wise to achieve feature fusion. It also includes an embedded transformation module, which includes: a transformer block, an average pooling layer, and a fully connected layer; the output of the fifth layer is modeled globally using the transformer block, and then the feature dimensions are uniformly mapped to the specified hidden space through the average pooling layer and the fully connected layer; The pre-training process is as follows: Image data is input into a text encoder to obtain text features; image data is input into an image encoder to obtain image features. The text feature, image feature, and label data encoding results are concatenated, trained through supervised contrastive learning, and the weight files of the trained text encoder and image encoder are saved. S3. Construct a multimodal fusion network, specifically including multiple cascaded transformer blocks. The input of the first transformer block is the query vector Q, and the input of subsequent transformer blocks is the output of the previous transformer block and a modal feature extracted by the pre-trained encoder in step S2. The outputs of each transformer block between the first and last transformer block are used as auxiliary classification vectors, and the output of the last transformer block is the fused classification vector. The number of cascaded transformer blocks is the sum of the number of text feature and image feature categories extracted by the pre-trained encoder in step S2 plus 1. S4. Construct a classification network; input the fused classification vector and each auxiliary classification vector into the classification network to train the classification network; S5. The multimodal medical data of the object to be processed is processed sequentially through the encoding obtained by the pre-training in step S2 and the multimodal fusion network constructed in step S3. The fused classification vector corresponding to the object to be processed and each auxiliary classification vector are input into the classification network trained in step S4 to obtain the classification result.

2. The meningioma Ki67 classification method based on multimodal medical data according to claim 1, characterized in that, Imaging data includes radiological data and radiomics data.

3. The meningioma Ki67 classification method based on multimodal medical data according to claim 2, characterized in that, The text encoder consists of two structurally identical sub-encoders, which are used to extract text features corresponding to radiological data and radiomics data, respectively. The structure of the two sub-encoders specifically includes a first fully connected layer, a second fully connected layer, and a transformer block connected in series.

4. The meningioma Ki67 classification method based on multimodal medical data according to claim 3, characterized in that, The loss function used in the pre-training process described in step S2 is: ； in, Indicate image features To imaging features The comparative loss, Indicates imaging features To image features The contrast loss is λ, which is a weighting parameter that balances the contrast losses of the two modes.

5. The meningioma Ki67 classification method based on multimodal medical data according to claim 4, characterized in that, The transformer block in the text encoder, the transformer block in the embedding module, and the first transformer block in step S3 specifically involve performing self-attention calculation on each data point and the input query vector Q.

6. The meningioma Ki67 classification method based on multimodal medical data according to claim 4, characterized in that, In step S3, the modal information fusion in the subsequent transformer block is specifically performed by iteratively performing a cross-attention operation with the query vector for each data point in the transformer block.

Images