Multimodal image classification method based on hybrid model network

By constructing feature maps and performing multi-level roughening and edge weight updates, combining graph convolution and full-connection networks, the problem of neglected connection between features in traditional methods is solved, and the accuracy and generalization ability of multimodal image classification is improved. , especially in the molecular phenotype classification of brain gliomas.

CN114821157BActive Publication Date: 2024-11-08SHANDONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210338626.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-01
Publication Date
2024-11-08
Estimated Expiration
2042-04-01

AI Technical Summary

Technical Problem

Traditional multimodal image classification methods ignore the potential connection and complementarity between features, resulting in redundant vector input and low classification accuracy, especially in the molecular phenotype classification of brain gliomas.

Method used

Using a multimodal image classification method based on hybrid model network, the feature map is constructed, and the Graclus algorithm is used to perform multi-level roughening. Combined with the convolution operation on the graph structure and the Gaussian kernel weight function, the edge weights are adaptively updated and the features are extracted. potential relationships and use a fully connected network for classification prediction.

Benefits of technology

The effect of feature selection is improved, the hidden layer relationship between features is paid attention to, the classification accuracy and generalization ability are improved, and it is manifested as higher classification accuracy and better AUC values.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821157B_ABST
    Figure CN114821157B_ABST
Patent Text Reader

Abstract

The present invention proposes a method, device and computer-readable storage medium for multimodal image classification based on a hybrid model network. By performing feature-level fusion on medical images from multiple different modalities, the features are regarded as nodes to construct a feature graph, feature selection is completed through a multi-level coarsening module, and the hybrid model network is used to adaptively learn the graph structure to find the potential connection between features, thereby obtaining a more reliable and accurate method for lesion category attribute information. The BraTS2017 dataset was used for verification, and compared with the traditional feature-level fusion method, the multimodal image classification method based on the hybrid model network has better results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and analysis, and specifically relates to a multimodal image classification method, device and computer-readable storage medium based on a hybrid model network. Background Art

[0002] With the rapid development of high-tech and the gradual realization of digital medical imaging technology, various imaging examinations have provided valuable multimodal medical imaging data for disease research. Due to the complex application scenarios of medical diagnosis, single-modal imaging data often cannot explain all possible subtle changes or abnormalities caused by the disease. Therefore, effectively integrating lesion information from multiple imaging modalities and leveraging the complementary advantages of each single-modal data have important guiding significance for medical clinical diagnosis.

[0003] Gliomas are a type of central nervous system tumor with a high recurrence rate and high mortality rate, which seriously threatens the health and life of patients. Clinical studies in recent years have found that its molecular pathology information can not only improve the accuracy of diagnosis, but also play an important guiding role in the clinical treatment and prognosis evaluation of gliomas. Therefore, in the fourth edition of the WHO classification of central nervous system tumors, genetic molecular phenotypes are integrated, and how to use multimodal imaging to achieve molecular phenotype classification of gliomas has become a current hot issue.

[0004] At present, the common multimodal image fusion is divided into: data layer fusion, feature layer fusion, and decision layer fusion. Feature layer fusion means that after feature extraction, the features of these different modalities are combined according to specific rules to obtain the fused features, and then classified. Compared with the other two schemes, this scheme is less likely to lose the original information, and at the same time has good real-time performance and is relatively more flexible. However, traditional feature layer fusion schemes often use linear fusion strategies such as serial splicing, which ignore the complementarity between features and may lead to the input of redundant vectors. Summary of the invention

[0005] Traditional multimodal feature layer fusion schemes often adopt the strategy of feature splicing or weighted addition, and then obtain the final classification result through the classifier, ignoring the potential connection and complementary information between features. To address this problem, the present invention proposes a multimodal image classification method based on a hybrid model network, which regards features as nodes and constructs a feature graph, and uses the Graclus algorithm to complete the multi-level roughening of the feature graph to achieve feature selection; then, the convolution operation on the graph structure and the Gaussian kernel weight function are used to adaptively update the edge weights of the nodes to complete the extraction and update of the potential relationship of the features; finally, the fully connected network is used to map to the sample label space to complete the prediction and classification of the disease.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A multimodal image classification method based on a hybrid model network constructs a feature map of the features proposed by all modalities, roughens it with the help of the Graclus algorithm, and finally completes the prediction classification based on the hybrid model network. The specific steps include:

[0008] (I) Data preprocessing: label each classification object;

[0009] (ii) Feature extraction: First, the region of interest is extracted from the image data, and then the region of interest is used as a mask to extract features from each modality image, and texture features, intensity histogram features, and shape features are obtained respectively, and feature normalization is performed on them;

[0010] (III) Constructing a feature graph: Consider each feature proposed in the above single sample as a discrete node, and based on the Euclidean distance between nodes, use the K nearest neighbor algorithm to construct the initial feature graph structure G0 = (V0, E0), where V0 is the node set of the feature graph, and E0 is the set of all edges in the feature graph;

[0011] (IV) Feature graph roughening: The feature graph constructed above is used as input, and multi-level roughening is performed based on the Graclus multi-scale clustering algorithm. First, the partitioning target is set to the normalized cut, that is, the initial weight of the node is set to the degree of the node; then all nodes in the feature graph are randomly visited. If the visited node x is not marked and all nodes in its neighborhood are marked, then the node x is marked and the next node is visited; if there are unmarked nodes in the neighborhood of the node, then the unmarked vertex y in the neighborhood is found to maximize the following formula:

[0012]

[0013] Among them, e(x, y) refers to the edge weight between nodes, and w(x) and w(y) refer to the weights of nodes x and y respectively. After finding the maximized node y, the two matching nodes are merged into a new node. The weight of the new node is the sum of the weights of the original nodes. After all nodes are traversed, the size of the feature graph can be reduced by nearly half, that is, the first-level coarsening process is completed;

[0014] (V) Edge weight update: Using the graph convolution kernel based on the hybrid model network, the edge weights of the nodes in the coarsened feature graph are updated. First, the node x is represented in polar coordinates, and a pseudo-coordinate u(x, x′) of a d-dimensional vector is created through the node x′ in its neighborhood to determine the relative position between the node x and its neighbor x′:

[0015] u(x)=(ρ(x,x′),θ(x,x′))

[0016] In polar coordinates, ρ represents the polar radius and θ represents the polar angle. The relative position relationship between the above nodes is mapped to the relative weights between the nodes, and a set of Gaussian kernel functions w1(u),…,w with learnable parameters are used. J (u) Continuously perform convolution operations on the graph structure and adaptively learn weights to eventually complete the edge weight update of the node. The representation of the Gaussian kernel function is as follows:

[0017]

[0018] in and μ j They represent the learnable d×d and d×1 covariance matrices and the mean vector of the Gaussian kernel, respectively. J represents the dimension of the spatial convolution kernel during the convolution operation. exp() represents the exponential function with base e.

[0019] (VI) Fully connected network: The feature map after edge weight update is expanded into a one-dimensional vector with the help of global maximum pooling, and then mapped to the sample label space using a fully connected network to complete the explicit expression of the classification.

[0020] In particular, in the training of the model, it is necessary to shuffle the multimodal data in order, divide all the data into k parts equally, take one of them as the validation set, and use the remaining k-1 parts as the training set for k-fold cross-validation to obtain the final result.

[0021] In particular, when performing feature extraction in step (ii), for the multimodal magnetic resonance image, the tumor marker portion thereof can be used as the region of interest.

[0022] The present invention also provides an image processing device, which includes a data acquisition component, a memory and a processor, wherein the data preprocessing component can implement the above steps (1) to (2); the processor can be used to execute the above steps (3) to (6); the memory part is used to store the computer programs used in all steps, the processed data, and can store the final training model.

[0023] The present invention also includes a computer-readable storage medium having a computer program stored thereon, characterized in that the steps of the above method can be implemented when the program is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the multimodal image classification method based on the hybrid model network of the present invention.

[0025] Figure 2 It is a schematic block diagram of the image processing device of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the schematic diagram of the multimodal image classification method based on the hybrid model network of the present invention is as follows:

[0028] (I) Data preprocessing: label the sample data. For example, if it is a molecular phenotype classification, there are the following labels: mark the brain glioma IDH mutation and 1p / 19q joint deletion as 0, its IDH mutation and 1p / 19q non-joint deletion as 1, and its IDH wild type as 3.

[0029] (II) Feature extraction: First, the region of interest is extracted from the image data. For multimodal magnetic resonance images, the tumor marker part can be used as the region of interest. Then, the region of interest is used as a mask, and the Pyradiomics toolkit is used to extract features from each modality image to obtain texture features, intensity histogram features, and shape features, and then perform feature normalization processing on them.

[0030] (III) Constructing the feature graph: Consider each feature proposed by the single sample as a discrete node, and based on the Euclidean distance between nodes, use the K nearest neighbor algorithm to construct the initial feature graph structure G0 = (V0, E0), where V0 is the node set of the original graph, and E0 is the set of all edges in the feature graph.

[0031] (IV) Feature graph roughening: The feature graph constructed above is used as input, and multi-level roughening is performed based on the Graclus multi-scale clustering algorithm. First, the partitioning target is set to the normalized cut, that is, the initial weight of the node is set to the degree of the node; then all nodes in the feature graph are randomly visited. If the visited node x is not marked, and all nodes in its neighborhood are marked, then the node x is marked and the next node is visited; if there are unmarked nodes in the neighborhood of the node, then the unmarked vertex y in the neighborhood is found to maximize the following formula:

[0032]

[0033] Among them, e(x,y) refers to the edge weight between nodes, while w(x) and w(y) refer to the weights of nodes x and y, respectively. After finding the maximized node y, the two matching nodes are merged into a new node, and the weight of the new node is the sum of the weights of the original nodes. After all nodes are traversed, the size of the feature graph can be reduced by nearly half, that is, the first-level coarsening process is completed. Different levels of coarsening can be selected according to different application scenarios of the data set.

[0034] (V) Edge weight update: Use the graph convolution kernel based on the hybrid model network to update the edge weights of the nodes in the coarsened feature graph. First, put the node x into polar coordinates for representation, and create a d-dimensional vector pseudo-coordinate u(x,x′) through the node x′ in its neighborhood to determine the relative position between the node x and its neighbor x′:

[0035] u(x)=(ρ(x,x′),θ(x,x′))

[0036] In polar coordinates, ρ represents the polar radius and θ represents the polar angle. The relative position relationship between the above nodes is mapped to the relative weights between the nodes, and a set of Gaussian kernel functions w1(u),…,w with learnable parameters are used. J (u) Continuously perform convolution operations on the feature graph structure and adaptively learn weights to eventually complete the edge weight update of the node. The Gaussian kernel function is represented as follows:

[0037]

[0038] in and μ j They represent the learnable d×d dimensional covariance matrix and the mean vector of the d×1 dimensional Gaussian kernel respectively, J represents the dimension of the spatial convolution kernel during the convolution operation, and exp() represents an exponential function with base e.

[0039] (VI) Fully connected network: The feature map after edge weight update is expanded into a one-dimensional vector with the help of global maximum pooling, and then mapped to the sample label space using a fully connected network to complete the explicit expression of the classification.

[0040] Figure 2 A schematic block diagram of the image processing device of the present invention is given, which mainly includes three components: a data preprocessing component, a processor and a memory, wherein the data preprocessing component can implement the above steps (1) to (2); the processor can be used to execute the above steps (3) to (6); the memory part is used to store the computer programs used in all steps, the processed data, and can store the final training model.

[0041] Compared with the traditional method, the present invention not only completes the feature selection, but also pays attention to the hidden layer relationship of the graph structure between the features, so the classification accuracy is higher and the generalization ability is better.

[0042] The effect of the present invention can be further illustrated by the classification results:

[0043] The datasets used in the experiment are BraTS2017 and TCGA datasets, with a total of 167 multimodal data of objects. During the data preprocessing process of model training, the data were randomly shuffled and the dataset was divided into 5 parts, 4 of which were selected as training sets, and the remaining 1 was selected as a validation set. The features of the regions of interest in the data were extracted, and the proposed multimodal fusion method based on the hybrid model network was used for classification. Tables 1 and 2 respectively compare the classification accuracy, AUC value and overall standard deviation on KNN, RF and SVM classifiers after feature dimensionality reduction using VAR and ExtraTreesClassifier feature selection methods. In summary, the classification effect of the present invention is better than that of the classical method.

[0044] Table 1 Comparison of experimental results under VAR scheme

[0045] Experimental methods Accuracy ACC AUC VAR-KNN 65.45%±9.725% 0.8533±0.054113 VAR-RF 76.36%±7.545% 0.9051±0.048985 VAR-SVM 80.00%±9.959% 0.9250±0.038269 Ours 81.82%±4.285% 0.9409±0.038539

[0046] Table 2 Comparison of experimental results under ExtraTreesClassifier scheme

[0047] Experimental methods Accuracy ACC AUC ExtraTreesClassifier-KNN 71.52%±7.902% 0.8854±0.040820 ExtraTreesClassifier-RF 78.18%±9.191% 0.9150±0.052226 ExtraTreesClassifier-SVM 80.61%±9.958% 0.9325±0.044927 Ours 81.82%±4.285% 0.9409±0.038539

Claims

1. A multimodal image classification method based on a hybrid model network, which constructs a feature map of the features proposed by all modalities, roughens it with the help of the Graclus algorithm, and finally completes the prediction classification based on the hybrid model network. The specific steps include: (I) Data preprocessing: label each classification object; (ii) Feature extraction: First, the region of interest is extracted from the image data, and then the region of interest is used as a mask to extract features from each modality image, and texture features, intensity histogram features, and shape features are obtained respectively, and feature normalization is performed on them; (III) Constructing a feature graph: Consider each feature proposed in the above single sample as a discrete node, and based on the Euclidean distance between nodes, use the K nearest neighbor algorithm to construct the initial feature graph structure G0 = (V0, E0), where V0 is the node set of the feature graph, and E0 is the set of all edges in the feature graph; (IV) Feature graph roughening: The feature graph constructed above is used as input, and multi-level roughening is performed based on the Graclus multi-scale clustering algorithm. First, the partitioning target is set to the normalized cut, that is, the initial weight of the node is set to the degree of the node; then all nodes in the feature graph are randomly visited. If the visited node x is not marked and all nodes in its neighborhood are marked, then the node x is marked and the next node is visited; if there are unmarked nodes in the neighborhood of the node, then the unmarked vertex y in the neighborhood is found to maximize the following formula: Among them, e(x, y) refers to the edge weight between nodes, and w(x) and w(y) refer to the weights of nodes x and y respectively. After finding the maximized node y, the two matching nodes are merged into a new node. The weight of the new node is the sum of the weights of the original nodes. After all nodes are traversed, the size of the feature graph can be reduced by nearly half, that is, the first-level coarsening process is completed; (V) Edge weight update: Using the graph convolution kernel based on the hybrid model network, the edge weights of the nodes in the coarsened feature graph are updated. First, the node x is represented in polar coordinates, and a pseudo-coordinate u(x, x′) of a d-dimensional vector is created through the node x′ in its neighborhood to determine the relative position between the node x and its neighbor x′: u(x)=(ρ(x,x′),θ(x,x′)) In polar coordinates, ρ represents the polar radius and θ represents the polar angle. The relative position relationship between the above nodes is mapped to the relative weights between the nodes, and a set of Gaussian kernel functions w1(u),…,w with learnable parameters are used. J (u) Continuously perform convolution operations on the graph structure and adaptively learn weights to eventually complete the edge weight update of the node. The representation of the Gaussian kernel function is as follows: in and μ j They represent the learnable d×d and d×1 covariance matrices and the mean vector of the Gaussian kernel, respectively. J represents the dimension of the spatial convolution kernel during the convolution operation. exp() represents the exponential function with base e. (VI) Fully connected network: The feature map after edge weight update is expanded into a one-dimensional vector with the help of global maximum pooling, and then mapped to the sample label space using a fully connected network to complete the explicit expression of the classification.

2. The multimodal image classification method based on a hybrid model network according to claim 1, characterized in that: In the training of the model, the multimodal data needs to be shuffled in order, all the data needs to be divided into k parts on average, one of which is taken as the validation set, and the remaining k-1 parts are used as training sets for k-fold cross-validation to obtain the final result.

3. The multimodal image classification method based on a hybrid model network according to claim 1, characterized in that: When performing feature extraction in step (ii), for multimodal magnetic resonance images, the tumor marker portion can be used as the region of interest.

4. An image processing device, comprising a data preprocessing component, a memory and a processor, wherein the data preprocessing component can implement steps (1) to (2) described in claim 1; the processor can be used to execute steps (3) to (6) described in claim 1; the memory part is used to store the computer program used in each step described in claim 1, the processed data, and can store the final training model.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that When the program is executed, the steps of the method according to claim 1 can be implemented.