Pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images

Abstract

The present invention discloses a pituitary neuroendocrine tumor classification system and method based on multimodal fusion of text and multiscale images, comprising the following steps: collecting and preprocessing the multimodal data of patients with pituitary neuroendocrine tumors before surgery, surgery, and follow-up; based on the prediction model of pathological classification, dividing the preprocessed multimodal data into text data and image data, and extracting the features of each modal data respectively, interacting the extracted features of each modal data, and obtaining fused multimodal features; using the fused multimodal features, training the prediction model of pathological classification; inputting the multimodal data of the patient before surgery, surgery, and follow-up to the prediction model of pathological classification trained by training, and obtaining the corresponding classification information of pituitary neuroendocrine tumors. The present invention uses the patient's medical text information and the prediction model of preoperative multiscale imaging examination for pathological classification to achieve accurate classification of pituitary neuroendocrine tumors.

Description

Technical Field

[0001] The present invention belongs to the field of information science, and specifically relates to a pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images. Background Art

[0002] Pituitary neuroendocrine tumors [PitNETs] are the third most common central nervous system tumors. The new 2022 WHOPitNETs classification (referred to as the 2022 version) is a detailed histological classification based on tumor cell lineages [immunohistochemistry], cell types [pathological images], hormones produced [immunohistochemistry], and other auxiliary characteristics. There are four major categories, including the PIT-1 lineage, T-PIT lineage, SF-1 lineage, and PitNETs with no clear cell lineage. There are twelve PitNETs subtypes, among which growth hormone cells, lactotroph cells, and adrenocorticotroph cells have dense granulation and sparse granulation subtypes, while thyrotropin cells and gonadotropin cells PitNETs have no subtypes. PitNETs are a common intracranial primary tumor that can cause disease through excessive hormone secretion and local sellar growth, seriously affecting the patient's quality of life. Depending on the type and manifestation of the pituitary adenoma, conservative treatment or medication, surgery or radiotherapy, or a combination of these treatments, can be used. Pituitary adenomas are classified differently based on function, size, invasiveness, phenotype, and histology, with the aim of guiding treatment and prognosis. More than half of adenomas that cause clinically important health problems do not require surgical treatment. Therefore, pituitary tumors that do not require surgery cannot be histologically classified. So far, there is no good and comprehensive clinical classification scheme to guide PitNETs treatment and prognosis system, especially combining imaging features to guide pathological diagnosis and clinical prognosis determination. Therefore, it is necessary to design a lightweight PitNETs auxiliary classification that can combine data from multiple modalities to achieve accurate treatment and prognosis of tumors.

[0003] To date, there has been no multimodal study based on PitNETs. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images. It uses the patient's medical text information and preoperative multi-scale imaging examinations (such as magnetic resonance imaging MRI) to perform a predictive model for pathological classification, which can achieve accurate classification of pituitary neuroendocrine tumors.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images, including: an acquisition module, a construction module, an extraction module, a training module and a classification module;

[0007] The acquisition module is used to collect and pre-process the multimodal data of patients with pituitary neuroendocrine tumors before, during and after surgery.

[0008] The building module is used to build a prediction model for pathology classification;

[0009] The extraction module is used to divide the preprocessed multimodal data into text data and image data based on the prediction model of the pathology classification, and extract the features of each modal data respectively, and interact the extracted features of each modal data to obtain fused multimodal features;

[0010] The training module is used to train the prediction model of the pathology classification by using the fused multimodal features;

[0011] The classification module is used to input the multimodal data of preoperative, surgical and follow-up of the pituitary neuroendocrine tumor patients to be tested into the trained prediction model of the pathological classification to obtain the corresponding classification information of the pituitary neuroendocrine tumor.

[0012] Preferably, the text data includes: symptom description and medical report; the image data includes: MRI image, WSI image and molecular expression.

[0013] Preferably, in the extraction module, the process of extracting features from the MRI image in the image data includes:

[0014] Given a pixel image region X R , the image area X R Split into non-overlapping patches of the same size, defined as Where p×p is the resolution of the patch, n×n is the number of patches, and 3 is the number of channels;

[0015] Based on the regional attention module, from the image region X R Extracting region-level feature maps Where c is the number of channels of the feature map;

[0016] Through the channel pooling operation, the regional feature map M is converted into a feature weight map Represents X P the distribution of regional-level characteristics;

[0017] A method of uniform sampling of feature weights is used to sort the feature weight values ​​of the feature weight graph M′;

[0018] The sorting results are uniformly sampled at speed r to generate a binary sampling bitmap

[0019] According to the binary sampling bitmap M″, we can get P The expected representative subset M sampled from P ;

[0020] Using the pre-trained deep residual network, extract the desired representative subset M P The features of all patches in generate a block-level feature representation sequence Where d is the latitude of feature representation;

[0021] Extracting region-level feature representation from region-level feature map M And the region-level feature is represented as f R and block-level feature representation sequence F P As a multi-scale feature sequence

[0022] Introducing Transformer, the multi-scale feature sequence F ms Append a learnable class label Then add the 2D-aware sample position embedding E pos Generate multi-scale feature sequence Y ms , using the standard Transformer encoder to fuse the multi-scale feature sequence Y ms .

[0023] Preferably, in the extraction module, the process of extracting features from the WSI image in the image data includes:

[0024] In a given WSI image S, S is divided into valid image area blocks of the same size but non-overlapping, defined as Where q×q is the resolution of the image region and m is the number of regions;

[0025] The effective image area S R Further split into tiles Where p×p is the resolution of the image block, and m×(n×n) is the number of image blocks;

[0026] Use the pre-trained regional attention module to extract the effective image area S R Extracting region-level feature maps And take channel pooling operation to transform the regional feature map M into a feature weight map Where c is the number of feature map channels;

[0027] Using the feature weight uniform sampling method, a binary sampling bitmap is generated from the feature weight map M' And use the binary sampling bitmap M' from the block S P The representative subset S is obtained by sampling P′ ;

[0028] Through the pre-trained CNN from the representative subset S P′ Extract block-level features

[0029] Extract region-level features from the region-level feature map M Where d is the dimension of feature representation;

[0030] Using feature representation extracted from slice thumbnails of histopathology images Initialize the class label; then add the position embedding E′ consisting of the region position and the block position pos , block-level features Regional characteristics Generate multi-scale feature sequence Y ms ;

[0031] According to the multi-scale feature sequence Y through the transformer encoder ms Learn slice-level features.

[0032] Preferably, in the extraction module, the process of extracting features from molecular expressions in image data includes: extracting features through a channel matching module and a feature extraction module; wherein the channel matching module is composed of a convolutional layer and a ReLU layer; and the feature extraction module is composed of a convolutional layer, a ReLU layer, a maximum pooling layer, an LSTM layer, and a dropout layer.

[0033] Preferably, in the extraction module, the process of extracting features from text data includes:

[0034] Divide text data of various structures into basic units;

[0035] Convert each basic unit into a word vector and use the pre-trained word vector model to transform the word semantics into word vectors;

[0036] Use LSTM to combine word vectors to form sentence representation;

[0037] Adding position information to sentence representation;

[0038] A text encoder is used to extract key features in sentence representations that add position information.

[0039] Preferably, in the extraction module, the process of interacting the extracted features of each modal data to obtain a fused multimodal feature includes:

[0040] Normalize the features of each modality;

[0041] The attention weight between each modality and other modality features is calculated through the attention mechanism;

[0042] The attention weights are used to perform weighted fusion on the modal features to obtain a new modal feature representation, and finally the ITC loss is continuously calculated to obtain the final result.

[0043] The present invention also discloses a pituitary neuroendocrine tumor classification method based on multimodal fusion of text and multi-scale images, comprising the following steps:

[0044] Collect and preprocess multimodal data of patients with pituitary neuroendocrine tumors during preoperative, surgical, and follow-up periods;

[0045] Constructing predictive models for pathology classification;

[0046] Based on the prediction model of the pathology classification, the preprocessed multimodal data is divided into text data and image data, and features of each modal data are extracted respectively, and the extracted features of each modal data are interacted to obtain fused multimodal features;

[0047] Using the fused multimodal features, training the prediction model of the pathology classification;

[0048] The multimodal data of preoperative, surgical and follow-up of the patients with pituitary neuroendocrine tumors to be tested are input into the trained prediction model of the pathological classification to obtain the corresponding classification information of the pituitary neuroendocrine tumors.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] ① The present invention has richer data processing and deeper modal data interaction: in the data input stage, the model will automatically distinguish different types of data and make corresponding divisions to ensure more sufficient feature extraction, and can achieve better retention of low-dimensional features when extracting features from different types of data. In the data interaction stage, a complementary strategy is adopted to analyze different modal data, with visual data providing spatial information and text data providing voice information, so that data of different modalities can fully utilize their own advantages.

[0051] ② The present invention has a faster model classification speed: in the early stage of the model processing data, a shared weight strategy is adopted to reduce the number of parameters that need to be learned in the early stage. Only in the feature extraction stage will different extraction strategies be selected according to different modal data, which can improve the processing speed of the model while ensuring accuracy.

[0052] ③ The present invention has a more flexible and stable model architecture: when designing the model, the weight sharing strategy is adopted in the data processing part. In the feature extraction stage, the corresponding feature extraction module is selected according to the data type, and finally the classification is realized by calculating the ITC loss. The feature extraction part can be flexibly added and replaced. For different types of data input, the corresponding extraction part can be replaced to ensure that the data can fully provide the corresponding features. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 It is a block diagram of a pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images according to an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of the structure of the prediction model of pathological classification according to the embodiment of the present invention;

[0056] Figure 3 It is a flow chart of a method for classifying pituitary neuroendocrine tumors based on multimodal fusion of text and multi-scale images according to an embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the experimental results of the prediction model for pathological classification according to the embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Embodiment 1

[0061] like Figure 1 As shown, the present invention provides a pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images, including: an acquisition module, a construction module, an extraction module, a training module and a classification module;

[0062] The acquisition module is used to collect and preprocess the multimodal data of patients with pituitary neuroendocrine tumors before, during and after surgery.

[0063] The building module is used to construct a predictive model for pathology classification;

[0064] The extraction module is used for the prediction model based on pathological classification, which divides the pre-processed multimodal data into text data and image data, and extracts the features of each modal data respectively, interacts the extracted features of each modal data, and obtains fused multimodal features;

[0065] The training module is used to train the prediction model of pathology classification using the fused multimodal features;

[0066] The classification module is used to input the multimodal data of preoperative, surgical and follow-up of patients with pituitary neuroendocrine tumors to be tested into the trained pathological classification prediction model to obtain the corresponding classification information of pituitary neuroendocrine tumors.

[0067] In this embodiment, the process of collecting and preprocessing multimodal data of preoperative, surgical, and follow-up of patients with pituitary neuroendocrine tumors includes: collecting preoperative, surgical, and follow-up image and text data, and constructing a training data set, wherein the training data set includes a series of data such as magnetic resonance imaging (MRI) images, panoramic pathological images (WSI) images, molecular expressions, and texts obtained from patients with pituitary neuroendocrine tumors.

[0068] Design source: The resolution of WSI is usually high, while the resolution of MRI images is relatively small. The key to WSI image processing is how to extract representative features from thousands of patches in the entire slice image. Although CNN has excellent feature representation capabilities and can meet the feature processing of MRI images and molecular expression information, it is difficult to meet the processing requirements of WSI using only CNN. As for the architecture successfully used in natural language, Transformer has been able to perform computer vision tasks by converting images into patches, and has produced good performance. Although compared with CNN, Transformer can learn long-range relationship information well by establishing sequences, it is also impossible to directly process thousands of patches. For this problem, there is a way to learn visual feature representation by re-establishing the original image from a random subset of patches, which provides an idea for feature representation learning. However, due to the uneven distribution of features in pathological images, most patches may not have any features. If the same feature extraction operation is performed on all of them, a large part of the time will be spent processing invalid blank information. Because of the richness of the structure of text data, there are mainly unstructured, structured, semi-structured and other structures, which need to be unified in some way to facilitate model processing.

[0069] Therefore, a multi-head attention mechanism and a multi-scale Transformer structure are proposed to process and classify the multimodal input of the model, which can not only meet the needs of low-resolution MRI images, but also adapt to high-resolution WSI images, as well as text and molecular expression data.

[0070] In this embodiment, if Figure 2As shown in the figure, this figure describes the general architecture of the proposed model, which is mainly divided into three parts: data vectorization (multi-head self-attention, patch + position embedding), feature extraction (Text, Vision, GeneExpression) and multimodal interaction (multimodal attention, ITC loss calculation); the feature extraction of the text part is mainly composed of a single-layer encoder and a multi-layer attention mechanism, the feature extraction of the image part is mainly composed of a self-attention mechanism and a feedforward network, and the feature extraction of the gene expression part is mainly composed of channel matching and a block containing an attention mechanism; first, all data are vectorized to generate a vector representation of the corresponding data; then, data of different categories enter the corresponding feature extraction link, and the corresponding features are extracted as a high-dimensional representation; then, multimodal data interact, and different types of data provide corresponding information in different aspects to supplement or correct the missing or noise between data; finally, the obtained interactive information is input into the classification structure to obtain the final result. The input is text, MRI images, WSI and molecular expression data, which will pass through a multi-head attention mechanism module to identify which type of data it is, and then it will be divided / segmented according to different types of data, and the position information will also be added to it. Then it enters the feature extraction phase. Text data will enter the left part. Text data of different structures will be segmented into data of the same structure, and then the key features will be extracted by the text encoder. Image data will be divided into two parts. MRI and WSI will pass through the middle part, and molecular expression will pass through the right part. Since MRI has a lower resolution than WSI, the block division of WSI will be more detailed. WSI will go through more attention modules and feedforward networks than MRI images, while molecular expression data will go through channel matching and feature extraction on the right. After that, after the data from all aspects has been normalized, it will go through the multimodal attention module to calculate the weight of each feature, and then after the attention and ITC loss are calculated, the features will interact continuously to obtain new features, and finally the final result will be obtained through a classification head.

[0071] In this embodiment, for MRI and WSI processing, a feature weight uniform sampling method is used to sample subsets of WSI and MRI images, retaining patches with representative features in the image region. In addition, regional-level tissue features and block-level cell features are combined as multi-scale histopathological features to characterize WSI, and Transformer is introduced to fuse multi-scale features for WSI-level pathology detection.

[0072] Specifically, in the extraction module, the process of extracting features from MRI images in image data includes:

[0073] Given a pixel image region XR , split it into non-overlapping patches of the same size, defined as Where p×p is the resolution of the patch, n×n is the number of patches, and 3 is the number of channels. Then, the regional attention block can be used to obtain the R Extracting region-level feature maps Where c is the number of channels in the feature map. Through the channel pooling operation, the feature map M is converted into a feature weight map Represents X P The distribution of regional features of M′ and X is constructed based on the regional attention module. P This is the key basis for patch sampling.

[0074] In order to largely eliminate noise and redundant patches, a feature weight uniform sampling method is adopted. This method sorts the feature weight values ​​of M′ and then uniformly samples the sorted results at a speed r to generate a binary sampling bitmap Then we can get the bitmap M" from X P The expected representative subset M sampled from P :

[0075] M P =X P ⊙M″∈R s×(p×p×3) ,

[0076] Where ⊙ represents the patch-level sampling based on the bitmap, s = r × n 2 It is the rounded sampling number of the patch. By adjusting the sampling rate r, a balanced ratio is achieved, which can largely eliminate noise and redundant patches while retaining sufficient feature representation of the image area.

[0077] Then, the pre-trained deep residual network is used to extract M P The features of all patches in , to generate a block-level feature representation sequence Where d is the latitude of feature representation, except for the block-level feature representation F P In addition, the regional level feature representation is further extracted from the feature map M And f R and F P As a multi-scale feature sequence Then, Transformer is introduced to explore the deep correlation information between multiple block-level features and multi-scale features through a multi-head self-attention module.

[0078] Following the standard process of inputting to the Transformer, the multi-scale sequence F ms Append a learnable class label Then add the 2D-aware sample position embedding E pos To generate the sequence:

[0079] Y ms =[y cls , f R , F P ]+E pos ,

[0080] Finally, a standard Transformer encoder is used to fuse the multi-scale feature sequence Y ms After the interaction of the class label and all feature labels, it will be used as the image region X R The integrated multi-scale feature representation is used as the input of a multi-layer perceptron classifier to predict the pathological state.

[0081] In order to improve the performance of the model, an auxiliary classifier based on region-level features is introduced to improve the performance of the regional attention module. A dynamic objective function is used to balance the main classification loss and the auxiliary classification loss. The calculation formula is:

[0082] L = λL′(z ms , z)+(1-λ)L′(z r , z),

[0083] Where L' is the binary cross entropy classification loss function L'(y, p) = -[y*log p+(1-y)*log(1-p)], y is the true label value, usually 0 or 1, representing two categories; p is the predicted probability, indicating the probability of prediction being 1. λ∈(0,1] is the dynamic balance factor, z ms is the predicted label of the multi-scale feature, z r is the predicted label of the region-level feature, and z is the true label. Specifically, λ is set to a small value during the initial training to ensure that the model focuses on optimizing the regional attention module. As the number of training epochs increases, the λ value gradually increases to 1, shifting the optimization focus to the Transformer-based multi-scale feature fusion module.

[0084] Specifically, in the extraction module, the process of extracting features from the WSI image in the image data includes:

[0085] Due to the high-resolution characteristics of WSI, it is necessary to adopt a more detailed split. Based on the region level, the WSI-level attention and multi-scale transformer are improved to detect high-pixel WSI images. Using a structure similar to the region level, in a given WSI image S, it is divided into valid image area blocks of the same size but non-overlapping, defined as Where q×q is the resolution of the image region, m is the number of regions, and region S R Will be further split into tiles Where p×p is the resolution of the image patch and m×(n×n) is the number of image patches.

[0086] Then, the pre-trained regional attention module is used to extract R Extracting region-level feature maps And take the channel pooling operation to input it into the feature weight map Where c is the number of feature map channels. Then, a binary sampling map is generated from M' by using the feature weight uniform sampling method. And use the bitmap M" from S P The representative subset S is obtained by sampling P′ :

[0087]

[0088] Where ⊙ represents block-level sampling according to the bitmap, and s represents the number of samples of blocks in a single area.

[0089] Next, we use the pre-trained CNN to extract P′ Extract block-level features Extracting region-level features from M Where d is the dimension of the feature representation. In order to optimize the model using the slice-level features of histopathology images, the feature representation extracted from the slice thumbnails is used. To initialize the class label. Multi-scale feature sequence Y ms Add a position embedding E′ consisting of region position and block position pos As input to a standard Transformer encoder:

[0090]

[0091] Finally, the multi-layer perceptron classifier is trained based on the multi-scale feature sequence Y ms Slice-level features are learned to detect tumor status.

[0092] Specifically, in the extraction module, the process of extracting features of molecular expressions in image data includes:

[0093] Since this part of the data is not suitable for feature extraction using the idea of ​​block segmentation, the following two steps (channel matching and feature extraction) are used for the operation.

[0094] Among them, the module consists of a convolutional layer and a ReLU layer. Since DNA sequences and DNA shapes are heterogeneous data, a reasonable strategy is needed to integrate their information. The method used involves four shape arrangements, and the convolutional layer DNA shape features with a kernel size of 1 are used to match the number of channels of the hybrid network. Specifically, the input of the model needs to be in binary format, so one-hot encoding is used to convert the DNA sequence into an image-like matrix n×l, where n corresponds to the four nucleotides A, T, G, C, and l represents the length of the DNA sequence. For each convolutional network layer, the output of the layer is calculated by the following formula:

[0095]

[0096] Where X is the input, i is the index of the output position, k is the kernel index, W is the convolution weight tensor, and b is the bias term. M and N represent the size of the convolution kernel in two dimensions, respectively. is the weight tensor of the kth convolution kernel at position i, X i+m，n represents the pixel value and feature value of input X at position (i+m,n), b ik Represents the bias term corresponding to the kth convolution kernel at position i. The convolution output is transformed by the activation function f, which uses the nonlinear function ReLU, which can alleviate the gradient vanishing problem during back-propagation training and has better convergence performance. Definition:

[0097] The feature extraction module consists of a convolutional layer, a ReLU layer, a max pooling layer, an LSTM layer, and a dropout layer. The DNA shape information and the original DNA sequence data processed by CNN are input into a shared hybrid neural network to extract features for prediction. The advantage of applying a shared model is that it not only greatly reduces the parameters of the network but also can be trained in parallel. First, the convolutional layer is followed by a ReLU layer with a kernel of 16 to capture spatial information from the given DNA sequence and DNA shape features. One of the advantages of applying CNN to analyze omics data is that the input data type is diverse, features can be easily integrated, and effective features can be automatically discovered through representation learning. Then, a max pooling layer is used to select the maximum value based on the output of the convolutional layer, thereby reducing the dimensionality of the input and improving the computational efficiency of the model. Pooling operation definition: pooling(X) ik =max({X iM，k , X (iM+1)，k , ..., X (iM+M-1)，k}). Where pooling(X) ik represents the output after the pooling operation, at the i-th position of the output feature sequence, corresponding to the k-th feature; X iM，kRepresents the value of the kth feature at the i-th position in the input feature sequence X; M represents the length of the pooling window. The maximum pooling layer is followed by the LSTM layer, which is used to capture long-term dependencies between sequences as well as directional and spatial distances. RNN is an alternative to CNN for processing sequential data, and LSTM networks were first proposed to use special hidden units to remember inputs for a long time. The key to using LSTM is the cell state, which is regulated by structures called gates, including input gates, forget gates, and output gates. First, the forget gate decides which information to discard or save, secondly decides how much new information should be added to the cell state, and finally decides what value to output. The specific implementation steps of LSTM are as follows:

[0098] (Ⅰ)f t =sigmoid(W f *[h t-1 , x t ]+b f )

[0099] (Ⅱ)i t =sigmoid(W i *[h t-1 , x t ]+b i )

[0100] (III)C t =tanh(W G *[h t-1 , x t ]+b G )(Ⅳ)S t =f t ⊙S t-1 +i t ⊙C t

[0101] (V)O t =sigmoid(W0*[h t-1 , x t ]+b0)(VI)h t =O t ⊙tanh(S t ), where W represents the weight matrix, b represents the bias, and W f , W i , W G Represents the weight matrix of the forget gate, output gate, and candidate value, b f , b i , b G , b0 represent the bias items of forget gate, input gate, candidate value and output gate respectively, S t represents the state at time step t, f t 、i t , Ot Respectively represent the weight values ​​of the forget gate, input gate, and output gate, x t , C t 、h t denote the input vector, memory representation, and hidden layer state at time t, respectively, and ⊙ is element-wise multiplication. A dropout layer with a probability of 0.2 is added to avoid overfitting caused by ignoring half of the feature detectors, making the model more generalizable. Next, all dropout results of DNA sequence and DNA shape information are combined into a feature vector, and then the corresponding features are output.

[0102] In this embodiment, in the extraction module, the process of extracting features from text data includes:

[0103] The feature extraction of the text part uses the improved BERT architecture Bio_clinicalBERT, which is an improved structure of BERT in the biomedical field and is trained using a large amount of biomedical text. However, due to the richness of the input text types, the format of the text needs to be unified when the data is input to ensure the correct input of the model. For text data of various structures, it is first divided into basic units, such as words, phrases, or sentences. For Chinese text, it is segmented by words; for English text, it is segmented by spaces. Then each basic unit is converted into a word vector, and the pre-trained BERT model is used to convert the word semantics into a vector. Next, LSTM is used to combine the word vectors to form a sentence representation. Since the model uses position encoding to represent the position information of words in a sentence, the position information will be added to the sentence representation when constructing the sentence representation so that the model can better understand the relationship between words and thus better extract features. Since the input format of the BERT model is: [CLS] + sentence representation + [SEP], where [CLS] is a special symbol added in front of each input instance and [SEP] is a special separator token, CLS and SEP are also added when creating the sentence representation.

[0104] There are two main steps to adopt this framework: pre-training and fine-tuning. During pre-training, the model is trained using unlabeled data in different pre-training tasks. For fine-tuning, the model is first initialized with pre-trained parameters, and then all parameters are fine-tuned using labeled data from downstream tasks. Since the BERT model has a significant feature, that is, a unified architecture across different tasks, that is, the difference between the pre-trained architecture and the final downstream architecture is small. Since text-type data is still very small compared to image data, Bio_clinicalBERT is selected as the pre-trained model as the initialization of the model, while ensuring limited text data input, it can also have a good output effect.

[0105] In this embodiment, in the extraction module, the process of interacting the extracted features of each modal data to obtain a fused multimodal feature includes:

[0106] Since data from different modalities may have different scales and distributions, before multimodal feature fusion, the features of each modality are normalized to ensure that they have the same statistical characteristics to promote the effectiveness of feature fusion. Subsequently, the attention weight between each modality and other modal features is calculated through the attention mechanism, and then the modal features are weightedly fused using the attention weights to obtain a new modal feature representation, and finally the ITC loss is continuously calculated to obtain the final result. Specifically, the image information is converted into}v cls , v1, ..., v n}, the text information is transformed into {w cls , w1,...,w N},s=g v (v cls ) T g w (w cls ), where v cls 、w cls is the vector corresponding to the class label; v n 、w N represents the word count corresponding to the nth (N) element in the sequence; s represents the similarity score, which calculates the similarity or correlation between two vectors; g v , g w It is a linear layer for dimensionality reduction; it also passes through a dynamic single-mode encoder and becomes g′ v (v′ cls ), g′ w (w′ cls ), transform the original class label vector into a high-level feature representation; then define: s(I, T) = g v (v cls ) T g′ w (w′ cls ), s(T, I) = g W (w cls ) T g′ v (v′ cls ), where s(I,T) represents the similarity score between the input sequence I (or vector) and the target sequence T (or vector), and s(T,I) represents the similarity score between the target sequence T (or vector) and the input sequence I (or vector). T and I are used as the reference frame and the object to be compared respectively. For each image and text, the softmax normalized image-to-text and text-to-image similarity is calculated as: where τ is a learnable parameter; s(I, T m ), s(T,I m ) means calculating the similarity between the input I and the mth element T in the target sequence, and calculating the similarity between the target T and the mth element I in the input sequence. Then let the true value of the onehot similarity be y i2t (I), y t2i (T), the probability of the negative sample pair in the true value is 0, and the probability of the positive sample pair is 1, then the resulting ITC loss is the cross entropy of p and y: The E and H tables represent expectation and entropy respectively.

[0107] Embodiment 2

[0108] like Figure 3 As shown, the present invention also provides a pituitary neuroendocrine tumor classification method based on multimodal fusion of text and multi-scale images, comprising the following steps:

[0109] Collect and preprocess multimodal data of patients with pituitary neuroendocrine tumors during preoperative, surgical, and follow-up periods;

[0110] Constructing predictive models for pathology classification;

[0111] Based on the prediction model of the pathology classification, the preprocessed multimodal data is divided into text data and image data, and features of each modal data are extracted respectively, and the extracted features of each modal data are interacted to obtain fused multimodal features;

[0112] Using the fused multimodal features, training the prediction model for the pathology classification;

[0113] The multimodal data of preoperative, surgical and follow-up of the patients with pituitary neuroendocrine tumors to be tested are input into the trained prediction model of the pathological classification to obtain the corresponding classification information of the pituitary neuroendocrine tumors.

[0114] Specifically, a series of preprocessing is first performed on the collected preoperative data, intraoperative data, and follow-up data to eliminate the influence of data noise, data bias, data distribution, etc. Then the data is divided into two categories: text and image for feature extraction. Due to the diversity of text category data, the text data will first be processed in a unified manner and converted into structured data, such as text information such as main symptoms, preoperative hormone testing, and medical history, which will be uniformly structured and used as the input of the text data model. Image data, such as WSI images, MRI images, molecular expression (transcription factors), etc., will be evenly divided into blocks, and then blocks with representative features will be selected as the input of the model to extract the corresponding image features. Finally, the parameters of the model are trained through the feature interaction of the two modal data. When rapid intraoperative pathological diagnosis is required, the data collected before surgery and the data obtained during surgery are used as the input of the model. The model analyzes the features and obtains the corresponding tumor classification information to achieve the purpose of rapid diagnosis and treatment.

[0115] Embodiment 3

[0116] like Figure 4 As shown, Figure 4 (a) Only preoperative data are used for training, lr0.005, ep50, 4k training times, and finally tends to be around 0.3. Figure 4 (b) The training is performed using preoperative, surgical, and follow-up data, with lr0.005, ep50, and 12k training times, and eventually tends to be around 0.15. Figure 4 (c) is a comparison chart of the two. After experiments, we can see that as the input data increases, the model's loss is gradually decreasing, and it has been able to learn most of the features in the data, proving that the designed model is feasible to a certain extent.

[0117] The present invention can more fully reflect the various conditions of patients through multimodal data from three periods: preoperative, surgical and follow-up. However, due to the richness and diversity of data types, and to achieve the purpose of rapid diagnosis and treatment, the model parameters need to be as few as possible and the data types that can be processed should be as many as possible. When designing this model, except for the feature extraction stage, other parts all adopt the strategy of shared weights to meet the requirement of as few parameters as possible. In the feature extraction stage, the model adopts the idea of ​​different data processing in different ways to ensure that multimodal data can be fully utilized, and the structure of this part can be flexibly selected. When new types of data are input, feature extraction modules can be freely added / replaced. For example: if the input data contains data such as MRI and SPECT, it can be replaced with the corresponding feature extraction part to achieve effective feature extraction.

[0118] The present invention utilizes multi-period and multi-modal data, and through an end-to-end model architecture, realizes the rapid classification of pituitary neuroendocrine tumors, so as to achieve the purpose of rapid diagnosis and treatment during surgery. The pre-protection point is that for the input part of the model, there is no need to consider the form of data input, and the model will automatically select the corresponding module to process the data. And in the feature extraction stage, this part of the structure can be added or replaced according to the needs of the task to ensure that the data can be fully utilized. Moreover, different loss functions can also be replaced to meet the needs of different downstream tasks through data of different modalities. For example: replacing it with ITM loss can realize image-text retrieval tasks, input pathological images into the model, and query image-related text and irrelevant text. Replacing it with LM loss can realize generation tasks, and can generate new text descriptions or images based on the input text or image, provide more accurate description information or improve the quality of the image.

[0119] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images, characterized by: include: Acquisition module, construction module, extraction module, training module and classification module; The acquisition module is used to collect and pre-process the multimodal data of patients with pituitary neuroendocrine tumors before, during and after surgery. The building module is used to build a prediction model for pathology classification; The extraction module is used to divide the preprocessed multimodal data into text data and image data based on the prediction model of the pathology classification, and extract the features of each modal data respectively, and interact the extracted features of each modal data to obtain fused multimodal features; The training module is used to train the prediction model of the pathology classification by using the fused multimodal features; The classification module is used to input the multimodal data of preoperative, surgical and follow-up of the pituitary neuroendocrine tumor patients to be tested into the trained prediction model of the pathological classification to obtain the corresponding classification information of the pituitary neuroendocrine tumor; The image data include: MRI images, WSI images and molecular expressions; In the extraction module, the process of extracting features from the MRI image in the image data includes: Given a pixel image region , the image area Split into non-overlapping patches of the same size, defined as , where p×p is the resolution of the patch, n×n is the number of patches, and 3 is the number of channels; Based on the regional attention module, from the image area Extracting region-level feature maps , where c is the number of channels of the feature map; Through the channel pooling operation, the regional feature map M is converted into a feature weight map ,express the distribution of regional-level characteristics; A method of uniformly sampling feature weights is used to uniformly sample the feature weight map. Sort by feature weight values; The sorting results are uniformly sampled at speed r to generate a binary sampling bitmap ; According to the binary sampling bitmap M'', we can get The expected representative subset sampled from ; Use the pre-trained deep residual network to extract the desired representative subset The features of all patches in generate a block-level feature representation sequence , where d is the latitude of feature representation; Extracting region-level feature representation from region-level feature map M , and the regional level features are represented as and block-level feature representation sequence As a multi-scale feature sequence ; Introducing Transformer to transform multi-scale feature sequences Append a learnable class label , and then add the 2D-aware sampling position embedding Generating multi-scale feature sequences , using a standard Transformer encoder to fuse multi-scale feature sequences ; In the extraction module, the process of extracting features from the WSI image in the image data includes: In a given WSI image S, S is divided into valid image area blocks of the same size but non-overlapping, defined as , where q×q is the resolution of the image region and m is the number of regions; The effective image area Further split into tiles , where p×p is the resolution of the image block and m×(n×n) is the number of image blocks; Use the pre-trained regional attention module to extract the effective image area Extracting region-level feature maps , and use channel pooling to transform the regional feature map M into a feature weight map , where c is the number of feature map channels; Using the feature weight uniform sampling method, a binary sampling bitmap is generated from the feature weight map M' , and use the binary sampling bitmap M'' from the block Sampling to get representative subset ; From the representative subset through pre-trained CNN Extract block-level features ; Extract region-level features from the region-level feature map M , where d is the dimension of feature representation; Using feature representation extracted from slice thumbnails of histopathology images Initialize class labels; then add position embedding consisting of region position and block position , block-level features , Regional level characteristics Generating multi-scale feature sequences ; According to the multi-scale feature sequence from the transformer encoder Learn slice-level features.

2. The pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images according to claim 1 is characterized in that: The text data includes: symptom descriptions and medical reports.

3. The pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images according to claim 1 is characterized in that: In the extraction module, the process of extracting features of molecular expressions in image data includes: extracting features through a channel matching module and a feature extraction module; wherein the channel matching module is composed of a convolutional layer and a ReLU layer; and the feature extraction module is composed of a convolutional layer, a ReLU layer, a maximum pooling layer, an LSTM layer, and a dropout layer.

4. The pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images according to claim 1, characterized in that: In the extraction module, the process of extracting features from text data includes: Divide text data of various structures into basic units; Convert each basic unit into a word vector and use the pre-trained word vector model to transform the word semantics into word vectors; Use LSTM to combine word vectors to form sentence representation; Adding position information to sentence representation; A text encoder is used to extract key features in sentence representations that add position information.

5. The pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multi-scale images according to claim 1, characterized in that: In the extraction module, the process of interacting the extracted features of each modal data to obtain fused multimodal features includes: Normalize the features of each modality; The attention weight between each modality and other modality features is calculated through the attention mechanism; The attention weights are used to perform weighted fusion on the modal features to obtain a new modal feature representation, and finally the ITCloss is continuously calculated to obtain the final result.

6. A method for classifying pituitary neuroendocrine tumors based on multimodal fusion of text and multiscale images, which is implemented by applying the pituitary neuroendocrine tumor classification system based on multimodal fusion of text and multiscale images according to any one of claims 1 to 5, characterized in that: The following steps are involved: Collect and preprocess multimodal data of patients with pituitary neuroendocrine tumors during preoperative, surgical, and follow-up periods; Constructing predictive models for pathology classification; Based on the prediction model of the pathology classification, the preprocessed multimodal data is divided into text data and image data, and features of each modal data are extracted respectively, and the extracted features of each modal data are interacted to obtain fused multimodal features; Using the fused multimodal features, training the prediction model of the pathology classification; The multimodal data of preoperative, surgical and follow-up of the patients with pituitary neuroendocrine tumors to be tested are input into the trained prediction model of the pathological classification to obtain the corresponding classification information of the pituitary neuroendocrine tumors.

Images