A multi-modal classification method and system for digital pathology images

Abstract

The application discloses a kind of multi-modal classification method and system for digital pathological image, it is related to image processing field, the pathological whole field slice image is divided into image block to construct multi-instance learning " bag " in the present application, and other modal auxiliary data features are extracted.Image features are obtained by pre-training general feature extractor, after image features and other modal features are spliced, input multi-modal gate network, calculate the gate weight of each image block corresponding different superclass subnetwork;Then image features are input into multiple subnetworks to obtain specific feature vectors, after aggregation by gate weight, through attention mechanism, it is summarized as " bag " feature and classified output.The present application adopts multi-stage pre-training strategy, first trains general feature extractor, then pre-trains subnetwork by superclass.The method effectively fuses image and auxiliary information, simulates the actual diagnosis decision process of pathologist, significantly improves classification accuracy and model interpretability.

Description

Technical Field

[0001] This invention relates to image processing, specifically a multimodal classification method and system for digital pathological images. Background Technology

[0002] The development of machine learning algorithms and the availability of whole-slice images have greatly accelerated research in the field of digital pathology. Current advances primarily focus on class prediction of tissue samples based on hematoxylin and eosin (H&E) staining, a commonly used method for pathological diagnosis. This approach provides quantitative opinions during the diagnostic process, aiming to reduce pathologist costs and ensure more consistent diagnostic results. While current machine learning methods for digital pathology mainly utilize full-view pathological slide images, simply observing H&E-stained tissue samples is often insufficient for a definitive diagnosis. In actual diagnosis, pathologists consider various factors, including basic clinical information, interview results such as physical condition, other test results, and examination of the tissue sample. Furthermore, most cases require additional information, such as immunohistochemically stained tissue samples and genetic testing, before a final decision can be made. To improve the accuracy of diagnostic support tasks and provide interpretability for the pathologist's decision-making process, it is necessary to combine digital pathological images with relevant ancillary data.

[0003] Traditional multimodal analysis methods typically treat ancillary data as input features. Previous studies have largely combined image features with ancillary data, and most methods have used Transformer architectures to encode the relationship between images and ancillary data. However, the contribution of ancillary data to classification is highly dependent on its characteristics. Some ancillary data may be useful for classifying major categories, while others may only be useful for specific disease types. Therefore, it is essential to consider which information from these images and ancillary data is taken into account, and how this information influences the final decision. From an interpretability perspective, we aim to simulate the diagnostic process of a pathologist and consider how image and ancillary information influences the decision-making process in diagnostic tasks.

[0004] Superclass and subclass classification problems in pathology involve the complexity, standardization, and information delivery of diagnosis. This classification is crucial because it directly impacts patient prognosis, treatment decisions, disease research, and public health management. For example, in actual lymphoma diagnosis, flow cytometry is a commonly used ancillary data source, providing valuable information for classifying lymphomas into different subtypes (superclasses), such as B-cell or T-cell lymphomas. Therefore, replicating a pathologist's lymphoma diagnosis requires a model that clearly distinguishes the role of inputs, mimicking the actual diagnostic process. This might involve using only flow cytometry for superclass classification, while the final classification is based on image data. Therefore, we have developed a novel classification model that effectively combines multimodal inputs, analyzing superclass and subclass problems across multiple pathologies.

[0005] The hybrid expert network architecture employs a hierarchical structure as its network architecture, which is particularly effective for classification problems with hierarchical superclasses and classes. In the hybrid expert network architecture, the final output of the model is obtained by applying weights from the gating network to the outputs of multiple weak classifiers (i.e., experts). In this patent, our goal is to achieve high classification accuracy by pre-training experts (covering superclasses and classes) with knowledge of the hierarchical class structure.

[0006] Because full-view slide images of pathology are large digital images (e.g., 100,000 × 100,000 pixels), they cannot be directly input into classification models. Typically, image patches are extracted from small regions of tissue sections as input. Furthermore, in most pathology image datasets, labels are assigned to full-view slide images rather than individual image patches due to the high cost of annotation by pathologists. This problem in full-view slide image analysis can be addressed using multi-instance learning techniques, a weakly supervised learning method. Multi-instance learning techniques are widely recognized as valuable in digital pathology and are applied not only to classification tasks but also to segmentation and survival prediction tasks. In our proposed method, we incorporate multi-instance learning techniques into a hybrid expert network architecture to create a multimodal pathology classification model.

[0007] In the field of digital pathology, various machine learning algorithms have been developed to handle a wide range of tasks based on digital pathology images. These tasks include digital pathology slide classification, segmentation, detection, survival prediction, similar image retrieval, and pathological analysis. A full-view pathology slide image is a digital image representing a section of pathological tissue digitized by a full-view pathology slide image scanner; these images can reach approximately 100,000 × 100,000 pixels in size. Due to computational resource limitations, small image patches are typically extracted from full-view pathology slide images as input to classification models such as convolutional neural networks (CNNs). In most pathology image datasets, class labels are assigned to full-view pathology slide images rather than individual image regions because annotating tumor regions is prohibitively costly for pathologists. Multi-instance learning techniques have proven effective when predicting slide-level labels based solely on image regions without region-level labels.

[0008] In multi-instance learning (MIM) classification, the model receives a "bag" representing a set of instances, where class labels are assigned to the entire "bag" rather than to individual instances. In the simplest binary classification problem, a positive bag contains at least one positive instance, while a negative bag contains only negative instances. When the classification problem is extended to multi-class classification, each "bag" representing a class contains at least one instance specific to that class. Note that most MLM research in digital pathology focuses on binary classification, i.e., distinguishing between cancer and normal cases. Patents on multi-class classification are relatively few in this area. MLM classification models aggregate information from each instance to generate a final prediction for the input "bag". In machine learning for digital pathology, the classification problem of full-view pathology slide images can be formulated as a MLM problem by treating a single full-view pathology slide image (bag) as a collection of multiple image regions (instances).

[0009] In many medical fields, including pathology diagnosis, doctors rely not only on images for diagnosis but also on other auxiliary data such as clinical records. Therefore, incorporating auxiliary information into machine learning methods, in addition to images, is an effective approach. In the medical field, numerous studies have explored multimodal analysis methods that combine images with clinical records. Previous research has shown that utilizing dermoscopy images and clinical records has improved the accuracy of skin lesion classification and melanoma detection. Multi-input learning techniques combine tabular clinical data with histological images to predict lymph node metastasis, including attributes such as age, genes, and tumor location, as well as multi-scale histological images as input. Experts have also proposed a transformer-based multimodal model for survival prediction using image and genetic data. Their method visualizes the co-attention between image and genetic information. Another study uses a transformer mechanism to encode the relationship between images and clinical records, providing exploratory and interpretable attention in classification and achieving high interpretability. In pathology and other medical fields, in addition to images, using diagnostic information as auxiliary input helps improve task accuracy.

[0010] Hybrid expert network architectures are effective in handling classification problems involving superclasses and subclasses. A hybrid expert network architecture is a machine learning method where the final output is obtained by weighting the outputs of multiple networks (i.e., experts) using a gating network. The gating network can employ soft gating, which probabilistically weights the expert outputs, or hard gating, which selects the output of a single expert. Hybrid expert network architectures have been applied in digital pathology research. For example, there is a segmentation method for histopathological images that introduces experts corresponding to different magnifications; and a classification method for cytopathological images that introduces experts corresponding to different input modalities.

[0011] In hybrid expert network architectures, the gating network typically receives the same input as the expert. By using multimodal data as input to the gating network, we aim to construct a classification model that closely resembles the decision-making process of a pathologist using their multimodal data for superclass classification. We propose a hybrid expert network architecture classification model that combines image and other modal data with a robust gating method. Furthermore, we aim to develop a highly interpretable model by combining the hybrid expert network architecture with multi-instance learning techniques. Summary of the Invention

[0012] The purpose of this invention is to provide a multimodal classification method and system for digital pathological images to solve the problems mentioned in the background art.

[0013] To achieve the above objectives, the present invention provides the following technical solution: A multimodal classification method for digital pathology images includes the following steps: Obtain a patient dataset, which includes pathological full-field section image data. Auxiliary data for other modalities and the corresponding tag data The tag data contains a hierarchical structure of superclasses and subclasses, and each subclass uniquely belongs to a superclass. From the pathological full-view slice image data Extracting multiple small image patches The image patch set is considered as a "package", and the classification problem is constructed as a multi-instance learning problem; The image feature vector is obtained by extracting features from the image patch and other modal data using a feature extractor. Other modal feature vectors ; The image feature vector and the other modal feature vectors are concatenated and input into a multimodal gating network to calculate the gating weights of the subnetworks corresponding to different superclasses for each image patch. ; The common image feature vector is input into multiple sub-networks corresponding to different superclasses to obtain specific feature vectors for each superclass. These specific feature vectors are then weighted and aggregated according to the gate weights to obtain an aggregated feature vector. ; The attention weights for each image patch are calculated using a multi-instance learning technique based on an attention mechanism. The aggregated feature vectors are then weighted and summed to obtain the final feature vector z representing the entire "package". The final feature vector is input into the classifier, which outputs the predicted category. The model is trained by minimizing the cross-entropy loss function between the predicted values ​​and the true labels.

[0014] Auxiliary data for other modalities The data can be one or more of the following: flow cytometry data, clinical records, gene testing data, or immunohistochemical data; the other modality feature extractor is a multilayer perceptron.

[0015] The multimodal gating network uses a Softmax activation function with a temperature parameter to calculate the gating weights. The balance between soft gating and hard gating is controlled by adjusting the value of the temperature parameter. When the temperature parameter value decreases, the weight distribution becomes sparser to highlight the subnetworks that should be focused on in each image patch.

[0016] The sub-network is pre-trained to extract encoded features for accurately predicting subclasses within the corresponding superclass; the specific pre-training steps include: Phase 1: Train an attention-based multi-instance classification model using data from all categories to obtain a general image feature extractor capable of extracting common features from all categories. ; The second stage involves training a multi-instance classification model using only the full-view pathological slide images of that superclass to obtain the sub-network for that superclass. This allows it to distinguish between multiple subclasses under the superclass.

[0017] As a further aspect of the present invention, the present invention also provides a multimodal classification system for digital pathological images, comprising: The data acquisition module is used to acquire the patient's full-field pathological slide image data, auxiliary data of other modalities, and label data; The image patch extraction module is used to extract multiple small image patches from pathological full-view slice images and construct a "package" for multi-instance learning; The feature extraction module includes an image feature extractor and other modal feature extractors; The multimodal gating module is used to input the stitched image features and auxiliary data features into the gating network and output the gating weights. The hybrid expert network module contains multiple sub-networks corresponding to different superclasses, which are used to output specific feature vectors and perform weighted aggregation; The attention aggregation module is used to calculate attention weights and summarize the aggregated feature vectors into a single "bundle" feature; The classification module is used to output the final predicted classification result.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention uses multi-instance learning technology to solve the problems of high cost and inability to input whole images in digital pathology image annotation, and can complete model training using weak labels; through a hierarchical hybrid expert network architecture, it adapts to the hierarchical classification requirements of superclasses and subclasses in pathology, allowing dedicated subnetworks of different superclasses to process corresponding category features, which is more in line with the actual diagnostic logic of pathologists; at the same time, it introduces a multimodal gating network, which integrates information from images and auxiliary data, can adaptively adjust the contribution weights of different superclass subnetworks, and can also control the sparsity of gating through temperature parameters, thereby improving the interpretability of the model while ensuring classification accuracy, clearly demonstrating the impact of different modal information on the decision-making process, and solving the problem that existing methods cannot clearly distinguish the roles of images and auxiliary data in different hierarchical classifications. Attached Figure Description

[0019] Figure 1This is a schematic diagram of the multi-stage pre-training process in the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the second stage of pre-training in the method of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] In this embodiment of the invention, a multimodal classification method for digital pathological images includes the following steps: A dataset containing N patients is represented as follows: Each dataset contains three components. This is represented as a full-field pathological section image. For data in other modalities, This is the label data. Among them... Let L be a vector. It is a one-hot encoded vector representing a K-dimensional subclass. In a dataset containing C superclasses and K subclasses, the superclasses are represented as c1, C..., and the subclasses are represented as k1, K... Each of these K subclasses belongs to exactly one of the C superclasses. The objective of this patent is to provide pathological full-view slide image data for machine learning models. and other modal data As input, the specific classification of the nth patient is determined. The output of the model is a vector. Its purpose is to match the true one-hot encoded categories. Matching. Due to the full-view slice image. It is a large digital image that cannot be directly input into a classification model; therefore, the classification of the nth patient is done by... Extract A small image block This is determined by [the specific method used]. We define this problem as a multi-instance classification problem, and the mathematical expression of the model is defined as: .

[0023] Figure 1 shows an overview of the proposed classification model, which consists of four parts: Feature extractor; Multimodal gating networks; Multiple subnetworks; Attention-based multi-instance learning.

[0024] The feature extraction stage includes image feature extractors. Other modal feature extractors Where h and They are all a set of vectors of the same size M. As a feature extractor for a convolutional neural network, It is a feature extractor of a multilayer perceptron. It is pre-trained on images for all classifications. The feature extractor can obtain better common features, which are then fed into the sub-network mentioned later.

[0025] Multimodal Gating Stage: This stage uses other modalities as input to the gating network in the hybrid expert network architecture. This follows the diagnostic approach of pathologists, who typically use other modalities as a reference for superclass classification in diagnosis. Combining image patches with other modalities as input allows for the computation of robust gating weights to the variability of other modalities for each image patch. For each image patch, a feature vector from another modality... Each of them will produce a copy of a feature vector. For blocks The gating weights are calculated through a gating network with a multilayer perceptron structure, and their mathematical expression is as follows: .

[0026] in It is composed of image feature vectors and other modal feature vectors Obtained by merging and connecting. A C-dimensional vector. Each element (c) represents each image patch. The weights assigned to the C subnetworks, where To balance soft and hard gating, we introduce a temperature parameter T into the softmax activation function of the gating network. A smaller temperature parameter value results in sparser weights, which helps to highlight the subnetworks that should be prioritized for each image patch.

[0027] Multiple self-network stages: Each self-network Receive a common image feature vector It outputs a specific feature vector for each subclass of a different superclass. Each subnetwork corresponds to an expert in a standard hybrid expert network architecture and is pre-trained to extract encoded features for accurately predicting the subclass within the corresponding superclass. Each image feature vector... Mapping function of the c-th subnet This is mapped to the representation vector of the c-th superclass. Each image patch, after passing through C sub-networks to obtain C feature vectors, is then gated by weights. The aggregated feature vector is obtained by performing aggregation calculations, and its mathematical expression is as follows: .

[0028] In this way, the feature vectors of each image patch in a "packet" are aggregated, and due to the effect of multimodal gating, the prediction becomes more discriminative. By using a common feature extractor and a trainable subnetwork to pre-train a multi-instance classification model on each superclass, a specific subnetwork is obtained for each superclass.

[0029] Attention mechanism stage based on multi-instance learning: image feature vector It is a weighted sum of the output vectors of the sub-networks based on gating weights, which are then aggregated into a single "packet" of feature vectors using a multi-instance learning technique based on an attention mechanism. This is achieved through an attention network with a neural network structure. Calculate an attention weight , used to represent image feature vectors Contribution to classification. The aggregate feature vector representing an input "bag" is calculated using a weighted image feature vector and attention weights, mathematically expressed as: Next, a classifier is connected to classify... Classifier Using z as input, the output is the predicted value. The training objective of the entire classification model is to minimize the predicted value. The cross-entropy loss function between the correct label Y and the correct label Y.

[0030] Multi-stage pre-training phase: First, a feature extractor is pre-trained, which is crucial for extracting feature representations of all subclasses and superclasses. Before initializing the proposed model, the feature extractor and sub-network are initialized by pre-training a multi-instance classification model. Both model structures are basic attention-based multi-instance classification models. First, multi-instance learning is performed on all categories to enable the model to classify the input pathological full-view slice image into the target category. By training this classification model, we obtain a universal feature extractor capable of representing image features of all categories. Subsequently, a multi-instance classification model was trained for the superclass. For the training sub-network... Only pathological full-field slice images belonging to superclass c are used. The number of subclasses in the c-th superclass is represented as... At that time, the output of the superclass model is dimensional vector This represents the prediction results for subclasses within superclass c. We train each subnetwork so that the computed features can effectively distinguish multiple subclasses within the corresponding superclass. We then use these pre-trained feature extractors and subnetworks as initial parameters to train the proposed model.

[0031] All classification models employ two types of attention: gating weights and attention weights. Gating weights indicate which subnetwork outputs should be focused on, while attention weights indicate which image regions should be focused on.

[0032] The entire network architecture is as follows Figure 1 As shown: The multi-stage pre-training diagram is shown below: Pre-training includes a general feature extractor and multiple sub-networks, which are implemented by training various multi-instance classification models. In the first stage, data from all categories is used to train a multi-class classification model to obtain a model capable of extracting features from all categories. This model can generate feature vector representations for each category. For example... Figure 2 As shown.

[0033] The second stage, for pre-training the sub-network, involves classifying multiple instances of data belonging to each superclass using multi-class classification. The second stage is illustrated in the following diagram: Implementation details: We describe the model architecture and training process in detail. ResNet50 was used as a feature extractor for pathological images. After global average pooling, a 2048-dimensional vector is obtained as its output. The pre-trained feature extractor described above was fine-tuned and used with the learned image feature extractor. Used for all other methods. Feature extractor for multimodal data. It is a two-layer neural network that uses 128 output units and the ReLU activation function to transform a multidimensional, multimodal data vector T into a 128-dimensional feature vector. The increased dimensionality of the output vector compared to the input vector aims to handle the numerical data inherent in multimodal processing. We implement a general multimodal classification model for pathological images. Gated network. It is a three-layer multilayer perceptron with 512 hidden layers, employing the ReLU activation function and outputting a three-dimensional weight vector through the softmax activation function. Subnetwork It is also a three-layer multilayer perceptron with 1024 hidden units and a ReLU activation function, thus producing a 512-dimensional vector. Attention network It is a three-layer multilayer perceptron with 128 hidden layer units and a hyperbolic tangent activation function, outputting a scalar value. (Classifier) It is a three-layer multilayer perceptron with 128 hidden units and a ReLU activation function, outputting a multi-dimensional class prediction vector through a softmax activation function. For multi-instance learning classification of multiple categories, an attention network is used. and classifier The number of hidden units was set to 512 because the length of the input feature vector was 2048. The network parameters were initialized randomly, except for the ResNet50-based feature extractor. The subnetworks are trained for classification across multiple categories. In methods based on multimodal extended hybrid expert networks, the subnetworks... The parameters are initialized based on the parameters of the subnetworks trained in the multi-instance learning classification model of each superclass.

[0034] During image preprocessing, all full-view pathology slide images were segmented into 224×224 pixel image patches at 40x magnification. The average chromaticity within each image patch was thresholded using the Otsu binarization method to remove glassy areas from the entire slide, extracting only the tissue region. Through this processing, each WSI contains 100 or more image patches. An input “packet” consists of 100 randomly selected 224×224 pixel image patches from the entire tissue at 40x magnification, and a multidimensional, multimodal data vector. In the training step, up to 5000 image patches are randomly selected from each full-view pathology slide image per cycle, creating up to 50 “packets” for each patient. There is no image patch overlap between “packets,” and each image patch is used only once in a single training cycle. To enhance the model's robustness to the randomness of image patches within a pack, 5000 image patches are randomly selected from each full-view pathology slide image per cycle. The network parameters were optimized by training for 10 epochs using momentum derived from stochastic gradient descent. The learning rate was set to 0.001, and the momentum was set to 0.9. The learning rate was adjusted by decreasing by 0.1 every 5 epochs.

[0035] As a further embodiment of the present invention, the present invention also provides a multimodal classification system for digital pathological images, comprising: The data acquisition module is used to acquire the patient's full-field pathological slide image data, auxiliary data of other modalities, and label data; The image patch extraction module is used to extract multiple small image patches from pathological full-view slice images and construct a "package" for multi-instance learning; The feature extraction module includes an image feature extractor and other modal feature extractors; The multimodal gating module is used to input the stitched image features and auxiliary data features into the gating network and output the gating weights. The hybrid expert network module contains multiple sub-networks corresponding to different superclasses, which are used to output specific feature vectors and perform weighted aggregation; The attention aggregation module is used to calculate attention weights and summarize the aggregated feature vectors into a single "bundle" feature; The classification module is used to output the final predicted classification result.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0037] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal classification method for digital pathological images, characterized in that, Includes the following steps: Obtain a patient dataset, which includes pathological full-field section image data. Auxiliary data for other modalities and the corresponding tag data The tag data contains a hierarchical structure of superclasses and subclasses, and each subclass uniquely belongs to a superclass. From the pathological full-view slice image data Extracting multiple small image patches The image patch set is considered as a "package", and the classification problem is constructed as a multi-instance learning problem; The image feature vector is obtained by extracting features from the image patch and other modal data using a feature extractor. Other modal feature vectors ; The image feature vector and the other modal feature vectors are concatenated and input into a multimodal gating network to calculate the gating weights of the subnetworks corresponding to different superclasses for each image patch. ; The common image feature vector is input into multiple sub-networks corresponding to different superclasses to obtain specific feature vectors for each superclass. These specific feature vectors are then weighted and aggregated according to the gate weights to obtain an aggregated feature vector. ; The attention weights for each image patch are calculated using a multi-instance learning technique based on an attention mechanism. The aggregated feature vectors are then weighted and summed to obtain the final feature vector z representing the entire "package". The final feature vector is input into the classifier, which outputs the predicted category. The model is trained by minimizing the cross-entropy loss function between the predicted values ​​and the true labels.

2. The multimodal classification method for digital pathological images according to claim 1, characterized in that, Auxiliary data for other modalities The data can be one or more of the following: flow cytometry data, clinical records, gene testing data, or immunohistochemical data; the other modality feature extractor is a multilayer perceptron.

3. The multimodal classification method for digital pathological images according to claim 2, characterized in that, The multimodal gating network uses a Softmax activation function with a temperature parameter to calculate the gating weights. The balance between soft gating and hard gating is controlled by adjusting the value of the temperature parameter. When the temperature parameter value decreases, the weight distribution becomes sparser to highlight the subnetworks that should be focused on in each image patch.

4. The multimodal classification method for digital pathological images according to claim 2, characterized in that, The sub-network is pre-trained to extract encoded features for accurately predicting subclasses within the corresponding superclass; the specific pre-training steps include: Phase 1: Train an attention-based multi-instance classification model using data from all categories to obtain a general image feature extractor capable of extracting common features from all categories. ; The second stage involves training a multi-instance classification model using only the full-view pathological slide images of that superclass to obtain the sub-network for that superclass. This allows it to distinguish between multiple subclasses under the superclass.

5. A multimodal classification system for digital pathological images employing the method described in any one of claims 1-4, characterized in that, include: The data acquisition module is used to acquire the patient's full-field pathological slide image data, auxiliary data of other modalities, and label data; The image patch extraction module is used to extract multiple small image patches from pathological full-view slice images and construct a "package" for multi-instance learning; The feature extraction module includes an image feature extractor and other modal feature extractors; The multimodal gating module is used to input the stitched image features and auxiliary data features into the gating network and output the gating weights. The hybrid expert network module contains multiple sub-networks corresponding to different superclasses, which are used to output specific feature vectors and perform weighted aggregation; The attention aggregation module is used to calculate attention weights and summarize the aggregated feature vectors into a single "bundle" feature; The classification module is used to output the final predicted classification result.

Images