Brain tumor multi-mode magnetic resonance image joint classification method and system

Abstract

The invention provides a brain tumor multi-mode magnetic resonance image joint classification method and system. The method comprises the following steps: S1, acquiring and preprocessing brain tumor multi-mode magnetic resonance image data; s2, extracting a radiomics feature set Frad of the brain tumor multi-mode magnetic resonance image data; s3, training a depth feature extractor, and extracting a depth feature set Fdep of the brain tumor multi-mode magnetic resonance image data; s4, the depth feature set Fdep and the radiomics feature set Frad are combined, and a combined feature set Fhybrid is obtained through screening; and S5, based on the combined feature set Fhybrid, classifying the brain tumor multi-modal magnetic resonance image. According to the invention, depth separable convolution and expansion and compression layers are used in the convolutional network coding module and the feature map down-sampling module, so that the model can better fuse the cross-modal features of the multi-modal magnetic resonance image.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing technology, and in particular, relates to a method and system for joint classification of multimodal magnetic resonance images of brain tumors. More specifically, it is a method and system for joint classification of multimodal magnetic resonance images of brain tumors combining radiomics and CNN-Transformer alternating coding deep feature extractor. Background Art

[0002] Brain tumors are one of the most deadly types of tumors in the world today. Data show that the prevalence of brain tumors in my country is about 32 per 100,000 people, including gliomas, lymphomas, and metastatic tumors. With the rapid development of medical imaging technology, especially the widespread application of magnetic resonance imaging technology, the early diagnosis and treatment of brain tumors have been greatly promoted.

[0003] Magnetic resonance imaging has high soft tissue contrast and strong non-invasiveness, and plays an important role in the detection and analysis of brain tumors. However, the imaging manifestations of brain tumors are complex and diverse. Affected by factors such as tumor type, size, location, and individual differences, traditional image analysis methods are difficult to fully explore the deep features in multimodal magnetic resonance imaging. Therefore, how to efficiently and accurately extract and fuse the features of different modal images has become an important research direction in the current field of medical image analysis.

[0004] Radiomics, a technology that uses quantitative image features to predict and diagnose diseases, has been widely used in oncology research. However, radiomics methods usually rely on artificially designed features and may face problems such as improper feature selection and information loss when processing complex and high-dimensional imaging data.

[0005] To this end, deep learning methods, especially convolutional neural networks, or CNNs, have been proposed and widely used in the automated analysis of medical images. CNNs have achieved significant performance improvements by automatically learning local and global features in images.

[0006] In recent years, the introduction of the Transformer architecture, especially its ability to handle long-range dependencies and global information, has enabled deep learning models to achieve further breakthroughs in image analysis tasks. The joint model combining CNN and Transformer has become a research hotspot, because CNN can efficiently extract local features, while Transformer can capture global information. The combination of the two can further enhance the representation ability of image features.

[0007] Although existing methods have made progress in some aspects, how to effectively combine the advantages of radiomics features and deep learning models, especially the advantages of CNN and Transformer, to further improve the classification performance of multimodal magnetic resonance images of brain tumors is still a difficult problem to be solved. Therefore, the present invention proposes a method and system for joint classification of multimodal magnetic resonance images of brain tumors that combines radiomics with CNN-Transformer alternating coding deep feature extractor, in order to achieve efficient and accurate classification of brain tumors through innovative feature extraction and fusion methods.

[0008] Patent document CN118691624A discloses a multimodal brain tumor medical image segmentation method and a model construction method thereof. The scheme introduces a parallel encoder feature extraction framework, which enables neurons to adaptively learn the physical spatial position relationship of brain gliomas specific to different modal images, thereby facilitating the extraction of richer feature information. Through the application of the regional perception fusion module, the method can adaptively fuse features according to the sensitivity of different brain regions to multimodal images, thereby achieving accurate segmentation of brain tumor regions. This intelligent method not only takes into account the impact of different modalities on different regions, but also can flexibly adjust the segmentation strategy according to real-time changing image features and lesion morphology, thereby maximizing segmentation accuracy and stability. This scheme cannot enable the model to better integrate the cross-modal features of multimodal magnetic resonance images, and it is difficult to accurately capture the multi-level information of brain tumor lesion areas. This problem needs to be solved urgently. Summary of the invention

[0009] In view of the defects in the prior art, the object of the present invention is to provide a method and system for joint classification of multimodal magnetic resonance images of brain tumors.

[0010] A brain tumor multimodal magnetic resonance imaging joint classification method provided by the present invention comprises:

[0011] Step S1: acquiring and preprocessing brain tumor multimodal magnetic resonance imaging data;

[0012] Step S2: Extracting the imaging feature set F of the brain tumor multimodal magnetic resonance imaging data rad ;

[0013] Step S3: Train a deep feature extractor to extract a deep feature set F of brain tumor multimodal magnetic resonance imaging data deep ;

[0014] Step S4: Merge the deep feature set F deep and the radiomics feature set F rad , filter out the joint feature set F hybrid ;

[0015] Step S5: Based on the joint feature set F hybrid ,Classification of brain tumors using multimodal magnetic resonance imaging.

[0016] Preferably, in step S1, the brain tumor multimodal magnetic resonance imaging data includes: T1-CE sequence, T2-Flair sequence and ADC map;

[0017] In the step S1, the preprocessing, i.e., taking the spatial position of the T1-CE sequence as a reference, rigidly aligns the T2-Flair sequence and the ADC image to the image space of the T1-CE sequence, and then obtains unified brain tumor multimodal magnetic resonance imaging data through n4 bias field correction and normalization.

[0018] Preferably, the step S2 includes:

[0019] Step S2.1: Based on the brain tumor multimodal magnetic resonance imaging data, annotating a region of interest in the brain tumor imaging data;

[0020] Step S2.2: generating a mask label reflecting the position information according to the region of interest to obtain a labeling result;

[0021] Step S2.3: Extract radiomics features one by one according to all the annotation results of the multimodal magnetic resonance images of the brain tumor to obtain a radiomics feature set F rad .

[0022] Preferably, in step S3, the deep feature extractor, i.e., a CNN-Transformer alternating encoding deep feature extractor;

[0023] In the step S3, it includes:

[0024] Step S3.1: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature vector f of brain tumor multimodal magnetic resonance imaging data deep ;

[0025] Step S3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating encoding deep feature extractor, the deep feature vector f deep The elements on each vector dimension are used as the feature values, and then the deep feature set F of brain tumor multimodal magnetic resonance imaging data is obtained. deep ;

[0026] In the step S3.1, the construction process of the CNN-Transformer alternating encoding deep feature extractor includes:

[0027] Step S3.1.1: trimming the brain tumor multimodal magnetic resonance imaging data to obtain input data I m ;

[0028] Step S3.1.2: 1×1×1 convolution of the input data I m , and expand the input data I m The number of channels is increased to C, and the initial feature map is obtained

[0029] Step S3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through M convolutional network units The local features of

[0030] Step S3.1.4: Transform the first feature map Input the downsampling module to get the first downsampling feature map

[0031] Step S3.1.5: Subtract the first downsampled feature map Input to Transformer encoder Extract the first downsampled feature map through N multi-head attention layers The global features of

[0032] Step S3.1.6: The second feature map Input the downsampling module to get the second downsampling feature map Repeat steps S3.1.3 to S3.1.6 until the final feature map is obtained.

[0033] The final feature map That is, the Lth down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor;

[0034] Step S3.1.7: The final feature map Flatten to get the one-dimensional eigenvector f flatten , for the feature vector f flatten The deep feature vector f is further refined using the fully connected layer deep ;

[0035] Step S3.1.8: Transform the depth feature vector f deep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, obtain and back-propagate the optimal network parameters;

[0036] In step S3.1.2, the convergence of the training model is accelerated by group normalization; the mathematical expression of the group normalization is:

[0037]

[0038] Among them, X represents the sample input value after the input graph is grouped by channel, N(X) represents the normalized output, μ(X) represents the mean of the sample input, σ(X) represents the standard deviation of the sample input, γ represents an affine transformation parameter, β represents another affine transformation parameter, and ε represents a constant used to avoid the denominator being zero;

[0039] In step S3.1.2, for the expansion layer and the compression layer, a 1×1×1 convolution is used to expand the number of channels of the input feature map to twice the original input, and after activation by the GELU function, a 1×1×1 convolution is used to compress the number of channels of the input feature map to the number of channels of the original input.

[0040] Preferably, the feature screening method completes feature screening to obtain a joint feature set F of brain tumor multimodal magnetic resonance imaging data. hybrid ; The feature screening method includes LASSO regression algorithm and Pearson correlation coefficient;

[0041] In the step S4, it includes:

[0042] Step S4.1: Merge the deep feature set F deep and the radiomics feature set F rad , get the feature set;

[0043] Step S4.2: By using the Pearson correlation coefficient algorithm, calculate and obtain the correlation coefficient between the features in the feature set, and determine whether the correlation coefficient is greater than 0.9. If the result is yes, retain the feature with the strongest correlation with the dependent variable; if the result is no, do not process it;

[0044] Step S4.3: Determine the lambda value that can minimize the error value through the LASSO regression algorithm, and then select and obtain the feature corresponding to the lambda value to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ;

[0045] In step S5, based on the joint feature set F obtained by screening hybrid , multimodal magnetic resonance imaging data of brain tumors were classified through a machine learning network model; the classification categories included: brain glioma, brain metastasis and primary central nervous system lymphoma.

[0046] A brain tumor multimodal magnetic resonance imaging joint classification system provided by the present invention comprises:

[0047] Module M1: Acquire and preprocess multimodal magnetic resonance imaging data of brain tumors;

[0048] Module M2: Extracting the imaging feature set F of the multimodal magnetic resonance imaging data of the brain tumor rad ;

[0049] Module M3: Train a deep feature extractor to extract a deep feature set F from multimodal MRI data of brain tumors deep ;

[0050] Module M4: Merge the deep feature set F deep and the radiomics feature set F rad , filter out the joint feature set F hybrid ;

[0051] Module M5: Based on the joint feature set F hybrid ,Classification of brain tumors using multimodal magnetic resonance imaging.

[0052] Preferably, in the module M1, the brain tumor multimodal magnetic resonance imaging data includes: T1-CE sequence, T2-Flair sequence and ADC map;

[0053] In the module M1, the preprocessing, i.e., based on the spatial position of the T1-CE sequence, the T2-Flair sequence and the ADC image are rigidly registered to the image space of the T1-CE sequence, and then unified brain tumor multimodal magnetic resonance imaging data is obtained through n4 bias field correction and normalization.

[0054] Preferably, the module M2 includes:

[0055] Module M2.1: Based on the brain tumor multimodal magnetic resonance imaging data, annotating a region of interest in the brain tumor imaging data;

[0056] Module M2.2: generating a mask label reflecting the position information according to the region of interest to obtain a labeling result;

[0057] Module M2.3: Extract radiomics features one by one based on all the annotation results of the multimodal magnetic resonance images of the brain tumor to obtain the radiomics feature set F rad .

[0058] Preferably, in the module M3, the deep feature extractor, i.e., a CNN-Transformer alternating encoding deep feature extractor;

[0059] The module M3 includes:

[0060] Module M3.1: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature vector f of brain tumor multimodal magnetic resonance imaging data deep ;

[0061] Module M3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating encoding deep feature extractor, the deep feature vector f deep The elements on each vector dimension are used as the feature values, and then the deep feature set F of brain tumor multimodal magnetic resonance imaging data is obtained. deep ;

[0062] In the module M3.1, the construction process of the CNN-Transformer alternating encoding deep feature extractor includes:

[0063] Module M3.1.1: Trimming the brain tumor multimodal magnetic resonance imaging data to obtain input data I m ;

[0064] Module M3.1.2: 1×1×1 convolution of the input data I m , and expand the input data I m The number of channels is increased to C, and the initial feature map is obtained

[0065] Module M3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through M convolutional network units The local features of

[0066] Module M3.1.4: The first feature map Input the downsampling module to get the first downsampling feature map

[0067] Module M3.1.5: Downsample the first feature map Input to Transformer encoder Extract the first downsampled feature map through N multi-head attention layers The global features of the second feature map are obtained

[0068] Module M3.1.6: The second feature map Input the downsampling module to get the second downsampling feature map Repeatedly trigger modules M3.1.3 to M3.1.6 until the final feature map is obtained.

[0069] The final feature map That is, the Lth down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor;

[0070] Module M3.1.7: The final feature map Flatten to get the one-dimensional eigenvector f flatten , for the feature vector f flatten The deep feature vector f is further refined using the fully connected layer deep ;

[0071] Module M3.1.8: Transform the deep feature vector f drep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, obtain and back-propagate the optimal network parameters;

[0072] In the module M3.1.2, the convergence of the training model is accelerated by group normalization; the mathematical expression of the group normalization is:

[0073]

[0074] Among them, X represents the sample input value after the input graph is grouped by channel, N(X) represents the normalized output, μ(X) represents the mean of the sample input, σ(X) represents the standard deviation of the sample input, γ represents an affine transformation parameter, β represents another affine transformation parameter, and ε represents a constant used to avoid the denominator being zero;

[0075] In the module M3.1.2, for the expansion layer and the compression layer, a 1×1×1 convolution is used to expand the number of channels of the input feature map to twice the original input, and after activation by the GELU function, a 1×1×1 convolution is used to compress the number of channels of the input feature map to the number of channels of the original input.

[0076] Preferably, in the module M4, the feature screening method completes the feature screening to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ; The feature screening method includes LASSO regression algorithm and Pearson correlation coefficient;

[0077] The module M4 includes:

[0078] Module M4.1: Merge the deep feature set F deep and the radiomics feature set F rad , get the feature set;

[0079] Module M4.2: By using the Pearson correlation coefficient algorithm, calculate and obtain the correlation coefficient between the features in the feature set, and determine whether the correlation coefficient is greater than 0.9. If the result is yes, retain the feature with the strongest correlation with the dependent variable; if the result is no, do not process it;

[0080] Module M4.3: Determine the lambda value that minimizes the error value through the LASSO regression algorithm, and then select and obtain the features corresponding to the lambda value to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ;

[0081] In the module M5, based on the joint feature set F obtained by the screening hybrid , multimodal magnetic resonance imaging data of brain tumors were classified through a machine learning network model; the classification categories included: brain glioma, brain metastasis and primary central nervous system lymphoma.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] 1. The present invention uses deep learning and machine learning methods to construct a method for jointly extracting multimodal magnetic resonance imaging features and deep features of brain tumors based on imaging omics methods and CNN-Transformer alternating encoders. This method can more accurately capture the multi-level information of the brain tumor lesion area and improve the diagnostic accuracy of the model.

[0084] 2. The present invention proposes a deep feature extractor using a CNN and Transformer alternating coding structure, namely, a CNN-Transformer alternating coding deep feature extractor, which combines the advantages of both CNN and Transformer so that the model can simultaneously extract local features and global features; the local features include the tumor body and edge morphology; the global features include the distribution of edema around the tumor and the anatomical structure.

[0085] 3. The present invention uses deep separable convolution and expansion and compression layers in the convolutional network encoding module and the feature map downsampling module, so that the model can better integrate the cross-modal features of multimodal magnetic resonance images. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0087] Figure 1 A flowchart of the workflow provided by the present invention;

[0088] Figure 2A schematic diagram of deep feature extraction of multimodal magnetic resonance imaging data of brain tumors provided by the present invention. DETAILED DESCRIPTION

[0089] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0090] Example 1: The present invention is based on multimodal magnetic resonance imaging data, using a deep learning feature extractor based on CNN and Transformer alternating encoding and an imaging omics method to construct a method and system for classification and diagnosis of brain tumor diseases.

[0091] Specifically, the multimodal magnetic resonance imaging data used include gadolinium-enhanced T1 weighted sequence, namely contrast-enhanced T1 weighted imaging, referred to as T1-CE, T2 weighted fluid attenuated inversion recovery sequence, namely T2 weighted fluid attenuated inversion recovery imaging, referred to as T2-Flair, and apparent diffusion coefficient map, namely apparent diffusion coefficient map, referred to as ADC map.

[0092] like Figure 1 As shown, the present invention provides a brain tumor multimodal magnetic resonance imaging joint classification method combining radiomics and CNN-Transformer alternating coding deep feature extractor, the method comprising the following steps:

[0093] Step S1: acquiring brain tumor multimodal magnetic resonance imaging data, and preprocessing the multimodal magnetic resonance imaging data to obtain unified brain tumor multimodal magnetic resonance imaging data;

[0094] Step S1.1: Acquire multimodal magnetic resonance imaging data of brain tumors, using an 8-channel head coil on a magnetic resonance scanning device with a field strength of 3.0T to acquire multimodal magnetic resonance imaging data of brain tumor patients, including T1-CE sequence, T2-Flair sequence and ADC image;

[0095] Step S1.2: Preprocess the multimodal MRI data. Based on the spatial position of the T1-CE sequence images, rigidly align the T2-Flair sequence and ADC image to the space of the T1-CE sequence images. Perform n4 bias field correction, resample to 1 mm × 1 mm × 1 mm, and z-score normalization on the multimodal MRI data to obtain unified brain tumor multimodal MRI data.

[0096] Step S2: obtaining a region of interest (ROI) mask of the multimodal magnetic resonance imaging data, and extracting imaging omics features of the multimodal magnetic resonance imaging data of brain tumors;

[0097] Step S2.1: obtaining a mask of the brain tumor region on the brain tumor multimodal magnetic resonance imaging data and taking it as the region of interest;

[0098] Specifically, step S2.1: having two or more experienced radiologists annotate the region of interest (ROI) of the multimodal magnetic resonance imaging data, and the final ROI mask selects the overlapping parts of the annotated results of all the doctors;

[0099] Step S2.2: Based on the obtained region of interest, radiomics features are extracted using radiomics methods.

[0100] Specifically, step S2.2: extracting imaging omics features of brain tumor multimodal magnetic resonance imaging data, for each magnetic resonance sequence modality in the multimodal magnetic resonance imaging data, extracting imaging omics features in the ROI corresponding to the sequence on the original image and the image filtered by a specific filter, including morphological features, first-order statistical features, texture features, etc., to obtain an imaging omics feature set F of the brain tumor multimodal magnetic resonance imaging data. rad .

[0101] Step S3: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature set F of brain tumor multimodal magnetic resonance imaging data deep ; The CNN-Transformer alternately encodes a deep feature extractor, that is, the steps of convolutional network encoding, downsampling, Transformer encoding and downsampling are used in sequence, and the feature extractor of the input image is extracted through multiple cycles.

[0102] Step S3.1: training a CNN-Transformer alternating coding deep feature extractor, inputting the brain tumor multimodal magnetic resonance image into the CNN-Transformer alternating coding deep feature extractor for training and saving the optimal network parameters;

[0103] Step S3.1.1: Load the multimodal magnetic resonance image of the brain tumor. The magnetic resonance image contained in each modality of the multimodal magnetic resonance image occupies one channel. Cut off the background part and adjust the size of the multimodal magnetic resonance image to obtain the input data I m ;

[0104] Step S3.1.2: Input data I m Send it to the 1×1×1 convolution to expand the input data I m After the number of channels is increased to C, the initial feature map is obtained

[0105] Step S3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through M convolutional network units The local features of

[0106] Step S3.1.4: Transform the first feature map Input downsampling module, get the first downsampling feature map through stride convolution and residual structure

[0107] Step S3.1.5: Subtract the first downsampled feature map Input to Transformer encoder Extract the first downsampled feature map through N multi-head attention layers The global features of

[0108] Step S3.1.6: The second feature map Input to the downsampling module, and obtain the second downsampling feature map through stride convolution and residual structure

[0109] Step S3.1.7: Repeat steps S3.1.3 to S3.1.6 to perform feature extraction on the downsampled feature map until the final feature map is obtained.

[0110] The final feature map That is, the Lth down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor;

[0111] Step S3.1.8: Final feature map Input to the deep feature output module, the final feature map is transformed into The dimension is reduced to one-dimensional features, and the one-dimensional feature vector f is obtained flatten , for the one-dimensional eigenvector fflatten The deep feature vector f is further refined using the fully connected layer deep ;

[0112] Step S3.1.9: Transform the depth feature vector f deep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, and back propagate to optimize the network parameters;

[0113] Step S3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating coding deep feature extractor, the deep features of the multimodal MRI images of brain tumors are extracted to obtain the deep feature set F of the multimodal MRI images of brain tumors. deep .

[0114] Step S4: Merge the radiomics feature set F rad With the deep feature set F deep , the feature screening method is used to complete the feature screening, and the joint feature set F of brain tumor multimodal magnetic resonance imaging data is obtained. hybrid .

[0115] Step S5: Based on the selected joint feature set F hybrid , using machine learning methods to complete the classification of multimodal magnetic resonance images of brain tumors.

[0116] The present invention also provides a brain tumor multimodal magnetic resonance imaging joint classification system combining radiomics and CNN-Transformer alternating coding deep feature extractor, and the system includes the following modules:

[0117] Module M1: Brain tumor multimodal MRI data collection and preprocessing module, to obtain unified brain tumor multimodal MRI data;

[0118] Module M2: Radiomics feature extraction module, which obtains brain tumor mask as the region of interest on the multimodal magnetic resonance imaging data of brain tumors and uses radiomics methods to extract radiomics feature sets from the region of interest;

[0119] Module M3: CNN-Transformer alternating encoding deep feature extraction module, which adaptively extracts a deep feature set containing local and global information on brain tumor multimodal magnetic resonance imaging data;

[0120] Module M4: Feature screening module, which combines the radiomics feature set and the deep learning feature set, uses the feature screening method to complete feature selection, and obtains a joint feature set;

[0121] Module M5: Classification module, based on the joint feature set obtained by feature screening, establishes a machine learning network model to complete the classification of brain tumor multimodal magnetic resonance imaging data into category A, category B, and category C.

[0122] Specifically, the module M3 includes:

[0123] Module M3.1: Input module, loading the multimodal MRI image of brain tumor, cutting out the background part and adjusting the size of the multimodal MRI image to obtain input data;

[0124] Module M3.2: Convolutional network coding module, in this embodiment, that is, the A series of convolutional layers are used to extract convolutional features rich in local information of multimodal magnetic resonance images;

[0125] Module M3.3: Transformer encoding module, in this embodiment, that is, A series of Transformer modules are used to extract Transformer features rich in global information of multimodal magnetic resonance images;

[0126] Module M3.4: Feature map downsampling module, which uses a series of convolutional layers to reduce the size and number of channels of the input feature map;

[0127] Module M3.5: Deep feature output module, which flattens the feature map and outputs a deep feature set through a fully connected layer.

[0128] In other words, the module M3 includes:

[0129] Module M3.1: The input module loads the multimodal MRI image of the brain tumor, cuts off the background part and adjusts the size of the multimodal MRI image to obtain the input data I m ;

[0130] Module M3.2: Convolutional network coding module, which uses a series of convolutional layers to extract convolutional features rich in local information of multimodal magnetic resonance images;

[0131] Module M3.3: Transformer encoding module, which uses a series of Transformer modules to extract Transformer features rich in global information of multimodal magnetic resonance images;

[0132] Module M3.4: Feature map downsampling module, which uses a series of convolutional layers to reduce the size and number of channels of the input feature map;

[0133] Module M3.5: Deep feature output module, flattens the feature map and outputs the deep feature set F through the fully connected layer deep .

[0134] Specifically, the module M3.2, the module M3.3 and the module M3.4 together constitute a CNN-Transformer alternating coding layer, and the number of this combination in the module M3 is not unique;

[0135] Specifically, the module M3 may also include a module M3.6: a predicted label output module, which further refines the deep feature set to obtain the predicted label, and uses a fully connected layer with a softmax function activation to obtain a classification result.

[0136] Specifically, the module M3.2 includes:

[0137] Module M3.2.1: Depth-wise separable convolutional layer, which encodes each channel of the input feature map and extracts local information of the input feature map;

[0138] Module M3.2.2: Normalization layer, using group normalization to accelerate training convergence and improve model generalization ability;

[0139] Specifically, the group normalization is to group the channels of the sample feature map, calculate the mean and variance within each group, and perform the normalization operation using the following formula:

[0140]

[0141] Where X represents the sample input value after the input graph is grouped by channel, μ(X) and σ(X) represent the mean and standard deviation of the sample input respectively, γ and β are learnable affine transformation parameters, and ε is a small constant to avoid the denominator being zero;

[0142] In other words, where X represents the sample input value after the input graph is grouped by channel, N(X) represents the normalized output, μ(X) represents the mean of the sample input, σ(X) represents the standard deviation of the sample input, γ represents an affine transformation parameter, β represents another affine transformation parameter, and ε represents a constant used to avoid the denominator being zero;

[0143] Module M3.2.3: For the expansion layer and compression layer, use 1×1×1 convolution to expand the number of channels of the input feature map to twice the original input, use GELU function activation to introduce nonlinear characteristics, and then use 1×1×1 convolution to compress the number of channels of the input feature map to the number of channels of the original input.

[0144] Specifically, the module M3.2.1, the module M3.2.2 and the module M3.2.3 are arranged in sequence to form a convolutional network unit;

[0145] Specifically, the module M3.2, i.e., the CNN encoder CNN described in this embodiment M , is formed by combining M convolutional network encoding units;

[0146] In particular, the module M3.2, i.e. the CNN encoder CNN described in this embodiment M The connection methods of the included convolutional network coding units include: series connection, parallel connection, residual connection, dense connection, etc.;

[0147] Specifically, the module M3.3 includes:

[0148] Module M3.3.1: Feature map segmentation module, which segments the input feature map into multiple small-sized feature map blocks;

[0149] Module M3.3.2: Linear encoding module, which performs linear mapping on the feature map blocks obtained by segmentation to obtain the encoded image primitives;

[0150] Module M3.3.3: Multi-head attention module, which uses multi-head attention to extract global information in the encoded primitives.

[0151] Specifically, the module M3.4 includes:

[0152] Module M3.4.1: contains a stride-length depthwise separable convolutional layer that encodes the input feature map using a convolution kernel with a stride of 2. The size of the output feature map is half of the input feature map.

[0153] Module M3.4.2: For the expansion layer and compression layer, use 1×1×1 convolution to expand the number of channels of the input feature map to twice the original input, activate it with the GELU function, and then use 1×1×1 convolution to compress the number of channels of the input feature map to the number of channels of the original input.

[0154] Example 2: This example is based on multimodal magnetic resonance imaging data, using a deep learning feature extractor based on CNN and Transformer alternating encoding and an imaging genomics method to construct a method and system for classification and diagnosis of brain tumor diseases.

[0155] Specifically, the brain tumors targeted in this embodiment include gliomas, brain metastases and primary central nervous system lymphomas.

[0156] Specifically, the multimodal magnetic resonance imaging data used include T1 weighted sequence, namely T1 weighted imaging, referred to as T1WI, gadolinium-enhanced T1 weighted sequence, namely contrast-enhanced T1 weighted imaging, referred to as T1-CE, T2 weighted sequence, namely T2 weighted imaging, referred to as T2WI, T2 weighted fluid attenuated inversion recovery sequence, namely T2 weighted fluid attenuated inversion recovery imaging, referred to as T2-Flair, and apparent diffusion coefficient map, namely apparent diffusion coefficient map, referred to as ADC map.

[0157] This embodiment provides a brain tumor multimodal magnetic resonance imaging joint classification method combining radiomics and CNN-Transformer alternating coding deep feature extractor, the method comprising the following steps:

[0158] Step 1: Acquire brain tumor multimodal magnetic resonance imaging data, and preprocess the multimodal magnetic resonance imaging data to obtain unified brain tumor multimodal magnetic resonance imaging data;

[0159] Step 1.1: Acquire multimodal MRI data of brain tumors. Use an 8-channel head coil on a 3.0T MRI scanner to collect multimodal MRI data of brain tumor patients, including T1WI sequence, T1-CE sequence, T2WI sequence, T2-Flair sequence, and ADC image.

[0160] Step 1.2: Preprocess the acquired multimodal MRI data. Based on the spatial position of the T1-CE sequence images, rigidly align the T1WI sequence, T2WI sequence, T2-Flair sequence and ADC image to the space of the T1-CE sequence images. Perform n4 bias field correction, resample to 1mm×1mm×1mm, and z-score normalization on all multimodal MRI data to obtain unified brain tumor multimodal MRI data.

[0161] Step 2: Obtain the region of interest (ROI) mask of the multimodal MRI data and extract the imaging features of the multimodal MRI data of brain tumors;

[0162] Step 2.1: Two or more experienced radiologists annotate the regions of interest (ROIs) of the multimodal MRI data, and the final ROI mask selects the overlapping parts of the annotated results of all doctors;

[0163] Step 2.2: Based on the obtained region of interest, use radiomics methods to extract radiomics features on each magnetic resonance imaging sequence, including morphological features, first-order statistical features, texture features, etc., to obtain the radiomics feature set F of brain tumor multimodal magnetic resonance imaging data. rad .

[0164] Step 3: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature set F of brain tumor multimodal magnetic resonance imaging data deep ; The CNN-Transformer alternately encodes a deep feature extractor, that is, the steps of convolutional network encoding, downsampling, Transformer encoding and downsampling are used in sequence, and the feature extractor of the input image is extracted through multiple cycles.

[0165] Step 3.1: training the CNN-Transformer alternating coding deep feature extractor, inputting the brain tumor multimodal magnetic resonance images into the CNN-Transformer alternating coding deep feature extractor for training and saving the optimal network parameters;

[0166] Step 3.1.1: Load the multimodal MRI image of the brain tumor. In this embodiment, the multimodal MRI image data includes five MRI sequences, each of which occupies one channel, i.e., a total of five channels. The background portion is cut off and the size of the multimodal MRI image is adjusted to 128×128×32 to obtain the input data I m ;

[0167] Step 3.1.2: Input data I m Send it to the 1×1×1 convolution to expand the input data I m After the number of channels is increased to 64, the initial feature map is obtained

[0168] Step 3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through 6 convolutional network units The local features of

[0169] Step 3.1.4: Feature map Input to the downsampling module, and obtain the downsampling feature map through stride convolution and residual structure

[0170] Step 3.1.5: Downsample feature map Input to Transformer encoder Extract downsampled feature maps through 4 multi-head attention layers The global features of

[0171] Step 3.1.6: Feature map Input to the downsampling module, and obtain the downsampling feature map through stride convolution and residual structure

[0172] Step 3.1.7: Repeat steps S3.1.3 to S3.1.6 for a total of 6 times, performing feature extraction operations on the downsampled feature map until the final feature map is obtained. The final feature map That is, the 12th down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor;

[0173] Step 3.1.8: Final feature map Input to the deep feature output module, the final feature map is transformed into The dimension is reduced to one-dimensional features, and the one-dimensional feature vector f is obtained flatten , for the one-dimensional eigenvector f flatten The deep feature vector f is further refined using the fully connected layer deep ;

[0174] Step 3.1.9: Transform the depth feature vector f deep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, and back propagate to optimize the network parameters;

[0175] Step 3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating coding deep feature extractor, the deep features of the multimodal MRI images of brain tumors are extracted to obtain the deep feature set F of the multimodal MRI images of brain tumors. deep .

[0176] Step 4: Combine the imaging features and deep features of brain tumor multimodal MRI data, and use the Pearson's correlation coefficient (PCC) and the least absolute shrinkage and selection operator (LASSO) feature screening method to complete feature screening and obtain the joint feature set F of brain tumor multimodal MRI data. hybrid .

[0177] Specifically, PCC is used to calculate the correlation coefficient between each pair of features in the feature set, and to determine whether the correlation coefficient is greater than 0.9. If the result is yes, that is, for feature pairs with a correlation coefficient greater than 0.9, only the features with a stronger correlation with the dependent variable are retained; if the result is no, no processing is performed; and then the Lasso method is used to determine the optimal lambda value that minimizes the cross-validation error value through ten-fold cross validation, and finally the feature corresponding to the optimal lambda value is selected.

[0178] In other words, step S4 includes:

[0179] Step S4.1: Merge the deep feature set F deep and the radiomics feature set F rad , get the feature set;

[0180] Step S4.2: By using the Pearson correlation coefficient algorithm, calculate and obtain the correlation coefficient between the features in the feature set, and determine whether the correlation coefficient is greater than 0.9. If the result is yes, retain the feature with the strongest correlation with the dependent variable; if the result is no, do not process it;

[0181] Step S4.3: Determine the lambda value that can minimize the error value through the LASSO regression algorithm, and then select and obtain the feature corresponding to the lambda value to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ;

[0182] Step 5: Based on the filtered joint feature set F hybrid , a multi-layer perceptron is used to filter the joint feature set F of brain tumor multimodal magnetic resonance imaging data hybrid , the input multimodal magnetic resonance imaging data are classified into brain glioma, brain metastasis and primary central nervous system lymphoma.

[0183] The method proposed in the present invention is based on multimodal magnetic resonance imaging data, using a deep learning feature extractor based on CNN and Transformer alternating coding and an imaging omics method to construct a classification diagnosis method for brain tumor diseases. The advantages are:

[0184] The present invention constructs a combined method based on imaging omics methods and CNN-Transformer alternating encoders to jointly extract multimodal magnetic resonance imaging features and deep features of brain tumors. Compared with a single method, it not only retains the standardized process and good interpretability of imaging omics methods, but also utilizes the end-to-end adaptive training optimization characteristics of deep learning methods. It can more accurately capture the multi-level information of brain tumor lesions and surrounding structures, thereby improving the diagnostic accuracy of the model.

[0185] The present invention proposes a deep feature extractor using a CNN and Transformer alternating coding structure. The deep feature extractor combines the advantages of CNN and Transformer, so that the model can simultaneously extract local features and global features; the local features include the tumor body and edge morphology; the global features include the distribution of edema around the tumor and the anatomical structure.

[0186] The present invention also provides a brain tumor multimodal magnetic resonance imaging joint classification system, which can be implemented by executing the process steps of the brain tumor multimodal magnetic resonance imaging joint classification method, that is, those skilled in the art can understand the brain tumor multimodal magnetic resonance imaging joint classification method as a preferred implementation of the brain tumor multimodal magnetic resonance imaging joint classification system.

[0187] A brain tumor multimodal magnetic resonance imaging joint classification system provided by the present invention comprises:

[0188] Module M1: Acquire and preprocess multimodal magnetic resonance imaging data of brain tumors;

[0189] Module M2: Extracting the imaging feature set F of the multimodal magnetic resonance imaging data of the brain tumor rad ;

[0190] Module M3: Train a deep feature extractor to extract a deep feature set F from multimodal MRI data of brain tumors deep ;

[0191] Module M4: Merge the deep feature set F deep and the radiomics feature set F rad , filter out the joint feature set F hybrid ;

[0192] Module M5: Based on the joint feature set F hybrid ,Classification of brain tumors using multimodal magnetic resonance imaging.

[0193] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0194] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A brain tumor multimodal magnetic resonance imaging joint classification method, characterized in that: include: Step S1: acquiring and preprocessing brain tumor multimodal magnetic resonance imaging data; Step S2: Extracting the imaging feature set F of the brain tumor multimodal magnetic resonance imaging data rad ; Step S3: Train a deep feature extractor to extract a deep feature set F of brain tumor multimodal magnetic resonance imaging data deep ; Step S4: Merge the deep feature set F drep and the radiomics feature set F rad , filter out the joint feature set F hybrid ; Step S5: Based on the joint feature set F hybrid ,Classification of brain tumors using multimodal magnetic resonance imaging.

2. The brain tumor multimodal magnetic resonance imaging joint classification method according to claim 1, characterized in that: In the step S1, the brain tumor multimodal magnetic resonance imaging data includes: T1-CE sequence, T2-Flair sequence and ADC map; In the step S1, the preprocessing, i.e., taking the spatial position of the T1-CE sequence as a reference, rigidly aligns the T2-Flair sequence and the ADC image to the image space of the T1-CE sequence, and then obtains unified brain tumor multimodal magnetic resonance imaging data through n4 bias field correction and normalization.

3. The brain tumor multimodal magnetic resonance imaging joint classification method according to claim 1, characterized in that: In the step S2, it includes: Step S2.1: Based on the brain tumor multimodal magnetic resonance imaging data, annotating a region of interest in the brain tumor imaging data; Step S2.2: generating a mask label reflecting the position information according to the region of interest to obtain a labeling result; Step S2.3: Extract radiomics features one by one according to all the annotation results of the multimodal magnetic resonance images of the brain tumor to obtain a radiomics feature set F rad .

4. The brain tumor multimodal magnetic resonance imaging joint classification method according to claim 2, characterized in that: In the step S3, the deep feature extractor, i.e., the CNN-Transformer alternating encoding deep feature extractor; In the step S3, it includes: Step S3.1: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature vector f of brain tumor multimodal magnetic resonance imaging data deep ; Step S3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating encoding deep feature extractor, the deep feature vector f deep The elements on each vector dimension are used as the feature values, and then the deep feature set F of brain tumor multimodal magnetic resonance imaging data is obtained. deep ; In the step S3.1, the construction process of the CNN-Transformer alternating encoding deep feature extractor includes: Step S3.1.1: trimming the brain tumor multimodal magnetic resonance imaging data to obtain input data I m ; Step S3.1.2: 1×1×1 convolution of the input data I m , and expand the input data I m The number of channels is increased to C, and the initial feature map is obtained Step S3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through M convolutional network units The local features of Step S3.1.4: Transform the first feature map Input the downsampling module to get the first downsampling feature map Step S3.1.5: Subtract the first downsampled feature map Input to Transformer encoder Extract the first downsampled feature map through N multi-head attention layers The global features of the second feature map are obtained Step S3.1.6: The second feature map Input the downsampling module to get the second downsampling feature map Repeat steps S3.1.3 to S3.1.6 until the final feature map is obtained. The final feature map That is, the Lth down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor; Step S3.1.7: The final feature map Flatten to get the one-dimensional eigenvector f flatten , for the feature vector f flatten The deep feature vector f is further refined using the fully connected layer deep ; Step S3.1.8: Transform the depth feature vector f deep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, obtain and back-propagate the optimal network parameters; In step S3.1.2, the convergence of the training model is accelerated by group normalization; the mathematical expression of the group normalization is: Among them, X represents the sample input value after the input graph is grouped by channel, N(X) represents the normalized output, μ(X) represents the mean of the sample input, σ(X) represents the standard deviation of the sample input, γ represents an affine transformation parameter, β represents another affine transformation parameter, and ε represents a constant used to avoid the denominator being zero; In step S3.1.2, for the expansion layer and the compression layer, a 1×1×1 convolution is used to expand the number of channels of the input feature map to twice the original input, and after activation by the GELU function, a 1×1×1 convolution is used to compress the number of channels of the input feature map to the number of channels of the original input.

5. The brain tumor multimodal magnetic resonance imaging joint classification method according to claim 4, characterized in that: In step S4, feature screening is performed by a feature screening method to obtain a joint feature set F of brain tumor multimodal magnetic resonance imaging data. hybrid ; The feature screening method includes LASSO regression algorithm and Pearson correlation coefficient; In the step S4, it includes: Step S4.1: Merge the deep feature set F deep and the radiomics feature set F rad , get the feature set; Step S4.2: By using the Pearson correlation coefficient algorithm, calculate and obtain the correlation coefficient between the features in the feature set, and determine whether the correlation coefficient is greater than 0.

9. If the result is yes, retain the feature with the strongest correlation with the dependent variable; if the result is no, do not process it; Step S4.3: Determine the lambda value that can minimize the error value through the LASSO regression algorithm, and then select and obtain the feature corresponding to the lambda value to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ; In step S5, based on the joint feature set F obtained by screening hybrid , multimodal magnetic resonance imaging data of brain tumors were classified through a machine learning network model; the classification categories included: brain glioma, brain metastasis and primary central nervous system lymphoma.

6. A brain tumor multimodal magnetic resonance imaging joint classification system, characterized in that: include: Module M1: Acquire and preprocess multimodal magnetic resonance imaging data of brain tumors; Module M2: Extracting the imaging feature set F of the multimodal magnetic resonance imaging data of the brain tumor rad ; Module M3: Train a deep feature extractor to extract a deep feature set F from multimodal MRI data of brain tumors deep ; Module M4: Merge the deep feature set F deep and the radiomics feature set F rad , filter out the joint feature set F hybrid ; Module M5: Based on the joint feature set F hybrid ,Classification of brain tumors using multimodal magnetic resonance imaging.

7. The brain tumor multimodal magnetic resonance imaging joint classification system according to claim 6, characterized in that: In the module M1, the brain tumor multimodal magnetic resonance imaging data includes: T1-CE sequence, T2-Flair sequence and ADC map; In the module M1, the preprocessing, i.e., based on the spatial position of the T1-CE sequence, the T2-Flair sequence and the ADC image are rigidly registered to the image space of the T1-CE sequence, and then unified brain tumor multimodal magnetic resonance imaging data is obtained through n4 bias field correction and normalization.

8. The brain tumor multimodal magnetic resonance imaging joint classification system according to claim 6, characterized in that: The module M2 includes: Module M2.1: Based on the brain tumor multimodal magnetic resonance imaging data, annotating a region of interest in the brain tumor imaging data; Module M2.2: generating a mask label reflecting the position information according to the region of interest to obtain a labeling result; Module M2.3: Extract radiomics features one by one based on all the annotation results of the multimodal magnetic resonance images of the brain tumor to obtain the radiomics feature set F rad .

9. The brain tumor multimodal magnetic resonance imaging joint classification system according to claim 6, characterized in that: In the module M3, the deep feature extractor, i.e., the CNN-Transformer alternating encoding deep feature extractor; The module M3 includes: Module M3.1: Construct a CNN-Transformer alternating encoding deep feature extractor to extract the deep feature vector f of brain tumor multimodal magnetic resonance imaging data deep ; Module M3.2: Based on the optimal network parameters obtained by training the CNN-Transformer alternating encoding deep feature extractor, the deep feature vector f deep The elements on each vector dimension are used as the feature values, and then the deep feature set F of brain tumor multimodal magnetic resonance imaging data is obtained. deep ; In the module M3.1, the construction process of the CNN-Transformer alternating encoding deep feature extractor includes: Module M3.1.1: Trimming the brain tumor multimodal magnetic resonance imaging data to obtain input data I m ; Module M3.1.2: 1×1×1 convolution of the input data I m , and expand the input data I m The number of channels is increased to C, and the initial feature map is obtained Module M3.1.3: Initial feature map Input to the convolutional network encoder Extract the initial feature map through M convolutional network units The local features of Module M3.1.4: The first feature map Input the downsampling module to get the first downsampling feature map Module M3.1.5: Downsample the first feature map Input to Transformer encoder Extract the first downsampled feature map through N multi-head attention layers The global features of Module M3.1.6: The second feature map Input the downsampling module to get the second downsampling feature map Repeatedly trigger modules M3.1.3 to M3.1.6 until the final feature map is obtained. The final feature map That is, the Lth down-sampled feature map is the feature map output by the last down-sampling module of the deep feature extractor; Module M3.1.7: The final feature map Flatten to get the one-dimensional eigenvector f flatten , for the feature vector f flatten The deep feature vector f is further refined using the fully connected layer deep ; Module M3.1.8: Transform the deep feature vector f deep Perform linear mapping to obtain the predicted label, use the cross entropy loss function to calculate the loss value, obtain and back-propagate the optimal network parameters; In the module M3.1.2, the convergence of the training model is accelerated by group normalization; the mathematical expression of the group normalization is: Among them, X represents the sample input value after the input graph is grouped by channel, N(X) represents the normalized output, μ(X) represents the mean of the sample input, σ(X) represents the standard deviation of the sample input, γ represents an affine transformation parameter, β represents another affine transformation parameter, and ε represents a constant used to avoid the denominator being zero; In the module M3.1.2, for the expansion layer and the compression layer, a 1×1×1 convolution is used to expand the number of channels of the input feature map to twice the original input, and after activation by the GELU function, a 1×1×1 convolution is used to compress the number of channels of the input feature map to the number of channels of the original input.

10. The brain tumor multimodal magnetic resonance imaging joint classification system according to claim 9, characterized in that: In the module M4, feature screening is performed by a feature screening method to obtain a joint feature set F of brain tumor multimodal magnetic resonance imaging data. hybrid ; The feature screening method includes LASSO regression algorithm and Pearson correlation coefficient; The module M4 includes: Module M4.1: Merge the deep feature set F deep and the radiomics feature set F rad , get the feature set; Module M4.2: By using the Pearson correlation coefficient algorithm, calculate and obtain the correlation coefficient between the features in the feature set, and determine whether the correlation coefficient is greater than 0.

9. If the result is yes, retain the feature with the strongest correlation with the dependent variable; if the result is no, do not process it; Module M4.3: Determine the lambda value that minimizes the error value through the LASSO regression algorithm, and then select and obtain the features corresponding to the lambda value to obtain the joint feature set F of the brain tumor multimodal magnetic resonance imaging data. hybrid ; In the module M5, based on the joint feature set F obtained by the screening hybrid , multimodal magnetic resonance imaging data of brain tumors were classified through a machine learning network model; the classification categories included: brain glioma, brain metastasis and primary central nervous system lymphoma.

Images