A multi-modal medical image classification method based on tensor decomposition subspace fusion

Abstract

The application relates to a multi-modal medical image classification method based on tensor decomposition subspace fusion. We first construct a multi-modal pleural fluid cell dataset and label the cell categories. In view of the common problems existing in the existing multi-modal fusion, a fusion framework based on the subspace level is designed. Based on a general image classification network, for the cell pictures of two modes, two independent convolutional neural networks are used to extract high-dimensional features, and the extracted high-dimensional features are decomposed to obtain low-dimensional feature subspaces. In the subspace, a self-attention mechanism is used to fully utilize the correlation of each part in the subspace, so as to change the spatial structure of the original features. Finally, the feature subspace is reconstructed to obtain the fully fused features for classification. The application is suitable for multi-modal medical image fusion classification tasks, such as cell category detection, fundus image age-related macular degeneration category detection and skin cancer melanoma detection.

Description

Technical Field

[0001] This invention relates to a multimodal medical image classification method based on tensor decomposition subspace fusion, belonging to the fields of computer vision and medical image processing. Background Technology

[0002] Multimodal fusion is widely used in computer vision, primarily to extract useful and complementary information from different modalities for subsequent tasks. In the medical field, deep learning-based methods play an irreplaceable role in disease diagnosis and prognosis. Before the widespread adoption of deep learning, image fusion had already been extensively studied. Early methods for image fusion involved manual analysis using relevant mathematical transformations and designing fusion rules based on the spatial or transform domains, known as traditional fusion methods. Typical traditional fusion methods include those based on multi-scale transformations, sparse representations, subspaces, and saliency. However, these methods have limitations. Traditional methods apply the same transformation method to different source images, failing to consider the differences in image features, resulting in poor feature representation. Furthermore, the feature fusion strategies of these traditional methods are too coarse, limiting fusion performance. Therefore, to overcome the limitations of traditional methods, deep learning has been introduced into image fusion algorithms. Deep learning-based methods can utilize different network branches to achieve differentiated feature extraction, thereby obtaining more targeted features. Secondly, guided by a carefully designed loss function, deep learning-based methods can learn more reasonable feature fusion strategies, achieving adaptive feature fusion. Thanks to these advantages, deep learning has facilitated tremendous progress in image fusion, with its performance far exceeding that of traditional methods.

[0003] Raw cell images are mostly transparent, so cytological analysis usually requires different staining reagents to induce color reactions in cells, causing different cells or cell structures to exhibit different colors. Different components within the cell tissue produce different refractive indices, facilitating observation and identification. Using differently stained cell images as different cell modalities, multimodal images, compared to single-modal images, help extract features from different views and provide complementary information, contributing to better data representation and network discrimination capabilities. Applying multimodal images can also reduce cell classification uncertainty and improve clinical diagnostic accuracy. However, as the feature dimension extracted by the network increases, multimodal data often introduces a large amount of noise, interfering with network training. Most fusion methods simply splice or add these high-dimensional features, a coarse fusion approach that does not consider the complementarity between modalities and increases the dimensionality of the fused features, making network training more difficult. Therefore, we decompose the high-dimensional features extracted by the network, separating redundant noise from truly useful information to obtain a lower-dimensional and cleaner feature subspace. We perform fusion at the subspace level, innovatively combining subspace learning with a self-attention mechanism. We introduce the self-attention mechanism from the Transformer into the fusion module, fully utilizing the correlation between different parts of the subspace to alter the spatial structure of the original features. Finally, we reconstruct the feature subspace using Tucker to obtain the fully fused features for classification. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal medical image classification method based on tensor decomposition subspace fusion, which classifies four types of cells by fusing cell images from two modalities.

[0005] To achieve the above objectives, the solution of the present invention is as follows:

[0006] Based on a general image classification network, a multimodal fusion classification framework was designed to remove redundant noise from high-dimensional features and incorporate a self-attention mechanism into the fusion module. The specific steps are as follows:

[0007] (1) Construct a multimodal medical cell image dataset, obtain the original cell images and corresponding HAL reagent stained images, and label the cells in four categories: cancer cells (A549), mesothelial cells (Met 5A), red blood cells (RBC), and white blood cells (WBC);

[0008] (2) Design a neural network to achieve multimodal image fusion and classification. Use a convolutional neural network to extract high-dimensional features from image data of different modalities;

[0009] (3) Using the idea of ​​Tucker decomposition, the high-dimensional features of the two modes are decomposed into core tensors and factor matrices of each dimension, and the high-dimensional features are denoised and reduced in dimensionality.

[0010] (4) The self-attention mechanism from Transformer is introduced into the fusion module to exchange feature information across various dimensions and explore the correlation between components in the subspace. Finally, the components in the subspace are reconstructed through Tucker reconstruction to obtain fully fused features.

[0011] (5) Conduct multiple experiments and record the average accuracy, F1-score, accuracy, precision, and F1-score of each experiment.

[0012] (6) Design an ablation experiment to investigate the contribution of each module to the classification performance.

[0013] The beneficial effects of this invention are as follows: This invention proposes a multimodal medical image classification method based on tensor decomposition subspace fusion. First, a multimodal medical image cell classification dataset is constructed. Then, a convolutional neural network is used for feature extraction. The Tucker decomposition approach is used to decompose high-dimensional features into low-dimensional feature subspaces. Noise reduction is achieved by adjusting the rank of each dimension. In the fusion module, the self-attention mechanism from the Transformer is introduced to exchange feature information across dimensions, allowing for full fusion of subspaces from different modalities and eliminating semantic gaps between them. Finally, the core tensor in the subspace and the feature matrices in the three dimensions are reconstructed to obtain the feature tensor used for classification. Attached Figure Description

[0014] Figure 1 This is a flowchart for constructing a multimodal cell dataset.

[0015] Figure 2 : This is a flowchart of the proposed multimodal fusion method. Detailed Implementation

[0016] We obtained two modalities of cell images: raw cell images and HAL-stained images. The two modalities corresponded one-to-one, and the cells were divided into four categories: cancer cells (A549), mesothelial cells (Met 5A), erythrocytes (RBCs), and leukocytes (WBCs). The dataset was collected using the following method: Figure 1 As shown in the figure. Each category has 100 paired samples, and the entire dataset contains 400 samples. 60% are used for training, 20% for validation, and 20% for testing.

[0017] Using original cell images and stained cell images, a multimodal fusion classification algorithm is implemented using Convolutional Neural Networks (CNN) and deep learning methods. Feature extraction is performed on cells from both modalities, and a multimodal feature fusion algorithm and loss function are designed.

[0018] like Figure 2 As shown in the diagram, this classification network mainly consists of several parts. First, two independent feature extraction networks extract high-dimensional features from two modalities. Then, our proposed Tucker decomposition module decomposes these high-dimensional features into feature subspaces. At the subspace level, a sub-attention mechanism is used to fully learn the correlations between each dimension, ultimately reconstructing the features for subsequent classification tasks. Finally, based on the classification results, accuracy, sensitivity, precision, F1-score, and the area under the receiver operating characteristic curve (AUROC) are calculated to evaluate the model's performance.

[0019] In our classification algorithm, we first use two independent residual networks, ResNet50, to extract features from cell images of two different modalities, enabling the network to learn the image features of different modalities.

[0020] After extracting image features from different modalities, the Tucker decomposition module is used to decompose the high-dimensional features, resulting in a core tensor g and factor matrices A, B, and C in the channel, length, and width dimensions. The core tensor g represents the correlation between the three dimensions, and the three factor matrices represent the principal components in each dimension.

[0021] X=((g×1A)×2B)×3C (1)

[0022] After feature decomposition, the self-attention mechanism from the Transformer is used to exchange the main information across the dimensions of the subspace, making the low-dimensional features in the subspace more correlated, learning complementary information from different modalities, and aligning features from different modalities. In our self-attention-based subspace feature fusion module, the factor matrix in each dimension serves as the Quary, and the three factor matrices are concatenated as the Key and Value.

[0023]

[0024] Among them, Q iQ1, Q2, and Q3 represent the factor matrices in the three dimensions, respectively; K and V represent the concatenated Key and Value; and d k This represents the scaling factor.

[0025] The core tensor g represents the correlation between the factor matrices in each dimension and is also the main component of the original high-dimensional feature tensor. Therefore, we still use the self-attention mechanism to fuse the high-frequency information in the core tensor with the three factor matrices. We reduce the number of channels in each dimension of the core tensor g to 1 through convolution operation to obtain the attention weights of the corresponding dimensions as Key and Value, and the factor matrices of the three dimensions as Query. The calculation formula is shown in formula (2).

[0026] Finally, we removed noise using Tucker decomposition, preserving the useful information of the high-dimensional features. We then used the self-attention mechanism in the Transformer to fuse the feature subspaces of the two modalities, obtaining a clean and fully fused feature subspace. We reconstructed the feature subspace, which is the inverse process of formula (1). The final fused features for classification were obtained, and the final prediction result was obtained through global average pooling and linear layer operations. Cross-entropy loss was used as the loss function, as shown in the following formula:

[0027]

[0028] Where N represents the number of samples, M represents the number of categories, and y ic This indicates whether the i-th sample belongs to the c-th class; a value of 1 indicates it belongs, and a value of 0 indicates it belongs. ic This represents the predicted probability that sample i belongs to category c.

[0029] The medical image classification algorithm based on multimodal fusion can be applied to real-world medical research and can be extended to various other medical image modalities, contributing to medical research in a modest way.

[0030] It should be noted that the above descriptions are merely embodiments of the present invention and are only intended to explain the invention, and are not intended to limit the scope of the patent. Modifications that are obvious from the inventive concept are also within the scope of protection of the present invention.

Claims

1. A multi-modal medical image classification algorithm based on tensor decomposition subspace fusion, characterized in that, Comprising the following steps: (1) Obtain the cell original image and the corresponding HAL reagent staining image, and mark four categories of cells; (2) Design a neural network framework, use a convolutional neural network to extract features from different modal image data, and use the idea of Tucker decomposition to decompose high-dimensional features into a low-dimensional feature subspace; wherein, using the idea of Tucker decomposition to decompose high-dimensional features into a low-dimensional feature subspace specifically includes: using a Tucker decomposition module to decompose high-dimensional features to obtain a core tensor and factor matrices in the channel dimension, length dimension and width dimension; wherein, the core tensor represents the correlation degree of the three dimensions, and the three factor matrices represent the main components in the three dimensions; (3) Introduce the self-attention mechanism in the Transformer into the fusion module to exchange feature information in each dimension and explore the correlation of each component in the subspace; (4) Finally, reconstruct each component in the feature subspace obtained by Tucker decomposition to obtain the fully fused features, and realize the multi-modal image fusion classification function; (5) The performance of the model is evaluated by calculating the average accuracy, F1-score, and the accuracy, precision, F1-score and area under the receiver operating characteristic curve (AUROC) of each class.

2. The multi-modal medical image classification algorithm based on tensor decomposition subspace fusion of claim 1, wherein, In step (1), the cell original image and the HAL reagent staining image correspond one by one; the four categories of cells are cancer cells A549, mesothelial cells Met 5A, red blood cells RBC and white blood cells WBC.

3. The multi-modal medical image classification algorithm based on tensor decomposition subspace fusion of claim 1, wherein, In step (4), the fully fused and denoised feature subspace is obtained, which is reconstructed into a fusion feature by Tucker to be used for final classification prediction.

Images