A multimodal breast magnetic resonance image classification method and system

Abstract

The present invention discloses a multimodal breast magnetic resonance image classification method and system, which relates to the field of medical and image processing technology. The method outputs the acquired magnetic resonance image to be classified and the parameter map to be classified into a target classification network to obtain the prediction probability of the lesion category, and determines the lesion category according to the prediction probability of the lesion category; the target classification network is obtained by training based on training samples and calibration convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network. The present invention combines the attention mechanism, multi-scale features and multi-modal information, so that the neural network learns more effective information that is helpful for feature extraction, thereby improving the accuracy of lesion category determination.

Description

Technical Field

[0001] The present invention relates to the field of medical treatment and image processing technology, and in particular to a multi-modal breast magnetic resonance image classification method and system. Background Art

[0002] In recent years, deep learning methods, especially convolutional neural networks (CNN), have been increasingly used in the field of medical imaging, playing an important role in medical image classification, reconstruction, and segmentation.

[0003] Adding an attention mechanism to CNN can make the network pay more attention to useful information and ignore useless information during the learning process, thereby effectively improving the classification performance of the model. The deepening process of CNN layers is the process of feature extraction from low-level to high-level semantic features. However, as the network deepens, each layer will lose some information, and more information will be lost in the last layer. In order to solve this problem, the concept of "multi-scale feature fusion" is introduced. Its basic idea is to fuse the features of different scales extracted from the features together, fuse high- and low-level information, and better complete the classification task. Common fusion methods include the average fusion of multiple scale prediction results and the splicing of multiple scale features. However, the poor features in these methods may affect the final prediction effect. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal breast magnetic resonance image classification method and system, which combines attention mechanism, multi-scale features and multimodal information to enable the neural network to learn more effective information suitable for feature extraction and improve the accuracy of target detection.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A multimodal breast magnetic resonance image classification method, comprising:

[0007] acquiring a magnetic resonance image to be classified;

[0008] Acquiring a parameter map corresponding to the magnetic resonance image to be classified to obtain a parameter map to be classified;

[0009] Outputting the magnetic resonance image to be classified and the parameter map to be classified into a target classification network to obtain a predicted probability of a lesion category;

[0010] determining a lesion category according to the predicted probability of the lesion category;

[0011] The target classification network is trained based on training samples and calibration convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is the lesion category of the primary lesion area in the first image; the structure of the calibration convolutional neural network is a dual-input single-output network structure; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network;

[0012] An input end of the first attention network is used to input the first image and the label;

[0013] The first output end of the first attention network is used to output a first feature map; the first feature map is determined after extracting features from the first image using the attention mechanism;

[0014] The second output end of the first attention network is used to output a third feature map; the third feature map is obtained by processing the first image using the attention mechanism and the multi-scale fusion algorithm;

[0015] The input end of the second attention network is used to input the first parameter image and the label;

[0016] The first output terminal of the second attention network is used to output a second feature map; the second feature map is determined after extracting features from the first parameter image using the attention mechanism;

[0017] The second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained by processing the first parameter image using the attention mechanism and the multi-scale fusion algorithm;

[0018] The first input end of the fusion network is used to input a fifth feature map; the fifth feature map is obtained by concatenating the first feature map and the second feature map;

[0019] The second input end of the fusion network is used to input the third feature map;

[0020] The third input terminal of the fusion network is used to input the fourth feature map;

[0021] The output end of the fusion network is used to output the predicted probability of the lesion category.

[0022] Optionally, the first attention network includes a first fusion module and a plurality of first residual attention modules;

[0023] The connection mode of the plurality of the first residual attention modules is a series connection; the output end of the first residual attention module at the end is the first output end of the first attention network, and the input end of the first residual attention module at the head end is the input end of the first attention network;

[0024] The input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

[0025] Optionally, the second attention network includes a second fusion module and a plurality of second residual attention modules;

[0026] The connection mode of the plurality of the second residual attention modules is a series connection; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head end second residual attention module is the input end of the second attention network;

[0027] The input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

[0028] Optionally, the structures of the multiple first residual attention modules are the same;

[0029] Among them, the data processing process of the first residual attention module is:

[0030] The input image is sequentially subjected to feature extraction through a 3*3*3 convolutional layer, a GN layer, a Dropout layer, and a rectified linear unit to obtain the first feature subgraph;

[0031] Performing feature extraction on the input image through a 1*1*1 convolutional layer, a GN layer, and a Dropout layer in sequence to obtain a second feature subgraph;

[0032] The first feature subgraph is subjected to feature extraction through a 3*3*3 convolutional layer, a GN layer, and a Dropout layer in sequence to obtain a third feature subgraph;

[0033] Performing feature extraction on the third feature subgraph through an attention module to obtain a fourth feature subgraph;

[0034] The second feature sub-graph is jump-connected with the fourth feature sub-graph and added to obtain an output image.

[0035] Optionally, the data processing process of the attention module is:

[0036] Performing feature extraction on the third feature subgraph through a channel attention module to obtain a channel attention feature graph;

[0037] Performing an element-wise multiplication operation on the channel attention feature map and the third feature sub-map to obtain an intermediate feature sub-map;

[0038] Performing feature extraction on the intermediate feature sub-graph through a spatial attention module to obtain a spatial attention feature graph;

[0039] An element-wise multiplication operation is performed on the spatial attention feature map and the intermediate feature sub-map to obtain a fourth feature sub-map.

[0040] Optionally, the process of determining the fifth characteristic map specifically includes:

[0041] The first feature map and the second feature map are concatenated in a channel series manner to obtain a fifth feature map.

[0042] Optionally, the training sample construction process specifically includes:

[0043] Acquire multiple original images; the original images include original three-dimensional dynamic enhanced magnetic resonance images and original parameter images corresponding to the original three-dimensional dynamic enhanced magnetic resonance images;

[0044] Preprocessing the plurality of original three-dimensional dynamic enhanced magnetic resonance images to obtain a first image; the preprocessing includes resampling processing, cropping processing, standardization processing and data enhancement processing;

[0045] Preprocessing the plurality of original parameter images to obtain a first parameter image;

[0046] Tag information is determined according to the first image.

[0047] To achieve the above object, the present invention also provides the following technical solutions:

[0048] A multimodal breast magnetic resonance image classification system, comprising:

[0049] A first data acquisition unit, used for acquiring a magnetic resonance image to be classified;

[0050] A second data acquisition unit is used to acquire a parameter map corresponding to the magnetic resonance image to be classified to obtain a parameter map to be classified;

[0051] A prediction probability determination unit, used for outputting the magnetic resonance image to be classified and the parameter map to be classified to a target classification network to obtain a prediction probability of a lesion category;

[0052] A lesion category determination unit, used to determine the lesion category according to the predicted probability of the lesion category;

[0053] The target classification network is trained based on training samples and calibration convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is the lesion category of the primary lesion area in the first image; the structure of the calibration convolutional neural network is a dual-input single-output network structure; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network;

[0054] An input end of the first attention network is used to input the first image and the label;

[0055] The first output end of the first attention network is used to output a first feature map; the first feature map is determined after extracting features from the first image using the attention mechanism;

[0056] The second output end of the first attention network is used to output a third feature map; the third feature map is obtained by processing the first image using the attention mechanism and the multi-scale fusion algorithm;

[0057] The input end of the second attention network is used to input the first parameter image and the label;

[0058] The first output terminal of the second attention network is used to output a second feature map; the second feature map is determined after extracting features from the first parameter image using the attention mechanism;

[0059] The second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained by processing the first parameter image using the attention mechanism and the multi-scale fusion algorithm;

[0060] The first input end of the fusion network is used to input a fifth feature map; the fifth feature map is determined by concatenating the first feature map and the second feature map;

[0061] The second input end of the fusion network is used to input the third feature map;

[0062] The third input terminal of the fusion network is used to input the fourth feature map;

[0063] The output end of the fusion network is used to output the predicted probability of the lesion category.

[0064] Optionally, the first attention network includes a first fusion module and a plurality of first residual attention modules;

[0065] The connection mode of the plurality of the first residual attention modules is a series connection; the output end of the first residual attention module at the end is the first output end of the first attention network, and the input end of the first residual attention module at the head end is the input end of the first attention network;

[0066] The input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

[0067] Optionally, the second attention network includes a second fusion module and a plurality of second residual attention modules;

[0068] The connection mode of the plurality of the second residual attention modules is a series connection; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head end second residual attention module is the input end of the second attention network;

[0069] The input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

[0070] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0071] The target classification network is obtained by training the training samples and the calibration convolutional neural network, wherein the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network; the first attention network outputs a first feature map determined after feature extraction of the first image using the attention mechanism, and also outputs a third feature map obtained after processing the first image using the attention mechanism and the multiscale fusion algorithm; the second attention network outputs a second feature map determined after feature extraction of the first parameter image using the attention mechanism, and also outputs a fourth feature map obtained after processing the first parameter image using the attention mechanism and the multiscale fusion algorithm; the fifth feature map obtained after multimodal fusion of the first feature map and the second feature map is input into the fusion network, in which the fifth feature map is fused with the third feature map and the fourth feature map, and finally the predicted probability of the lesion category is obtained, and then the lesion category is determined according to the predicted probability of the lesion category. The present invention combines the attention mechanism, multiscale features and multimodal information, so that the neural network learns more effective information that is helpful for feature extraction, and improves the accuracy of lesion category determination in the primary lesion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0073] Figure 1 It is a schematic diagram of the process of the multimodal breast magnetic resonance image classification method of the present invention;

[0074] Figure 2 It is a structural schematic diagram of the first residual attention module of the multimodal breast magnetic resonance image classification method of the present invention;

[0075] Figure 3 It is a structural schematic diagram of the attention module of the multimodal breast magnetic resonance image classification method of the present invention;

[0076] Figure 4 It is a schematic diagram of the structure of a multi-scale fusion module of the multi-modal breast magnetic resonance image classification method of the present invention;

[0077] Figure 5 It is a structural schematic diagram of a multimodal breast magnetic resonance image classification system of the present invention;

[0078] Figure 6 Schematic diagram of the structure of the convolutional neural network in the multimodal breast magnetic resonance image classification system of the present invention. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The purpose of the present invention is to provide a multimodal breast magnetic resonance image classification method and system, which uses an attention mechanism to fuse multi-scale features, and fuses information from multiple modalities of dynamic contrast enhanced-MRI (DCE-MRI) images and parameter images, so that the target classification network can learn more effective information suitable for the task, improve the accuracy of classification prediction, and assist doctors in better distinguishing benign and malignant breast lesions.

[0081] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0082] Embodiment 1

[0083] like Figure 1 As shown, this embodiment provides a multimodal breast magnetic resonance image classification method, including: step 100, obtaining a magnetic resonance image to be classified; step 101, obtaining a parameter map corresponding to the magnetic resonance image to be classified to obtain a parameter map to be classified; step 102, outputting the magnetic resonance image to be classified and the parameter map to be classified to a target classification network to obtain a predicted probability of a lesion category; step 103, determining a lesion category according to the predicted probability of the lesion category.

[0084] The target classification network is trained based on training samples and calibrated convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is a lesion category of the primary lesion area in the first image.

[0085] The structure of the calibration convolutional neural network is a dual-input single-output network structure; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network.

[0086] The input end of the first attention network is used to input the first image and the label; the first output end of the first attention network is used to output a first feature map; the first feature map is determined after feature extraction of the first image using the attention mechanism; the second output end of the first attention network is used to output a third feature map; the third feature map is obtained after processing the first image using the attention mechanism and a multi-scale fusion algorithm.

[0087] The input end of the second attention network is used to input the first parameter image and the label; the first output end of the second attention network is used to output a second feature map; the second feature map is determined after feature extraction of the first parameter image using the attention mechanism; the second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained after processing the first parameter image using the attention mechanism and a multi-scale fusion algorithm.

[0088] The first input end of the fusion network is used to input the fifth feature map; the fifth feature map is obtained by splicing the first feature map and the second feature map; specifically, the first feature map and the second feature map are spliced ​​in series through channels to obtain the fifth feature map. The second input end of the fusion network is used to input the third feature map; the third input end of the fusion network is used to input the fourth feature map; the output end of the fusion network is used to output the predicted probability of the lesion category, specifically outputting a predicted probability value of 0-1.

[0089] Preferably, the first attention network includes a first fusion module and multiple first residual attention modules; the multiple first residual attention modules are connected in series; the output end of the terminal first residual attention module is the first output end of the first attention network, and the input end of the head first residual attention module is the input end of the first attention network; the input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

[0090] In an example: multiple first residual attention modules are respectively the first residual attention sub-module BL1, the second residual attention sub-module BL2, the third residual attention sub-module BL3 and the fourth residual attention sub-module BL4; wherein, the first residual attention sub-module BL1 is the first residual attention module at the head end, and the fourth residual attention sub-module BL4 is the first residual attention module at the end.

[0091] Specifically, the structures of the first residual attention modules are the same, that is, the structures of the first residual attention submodule BL1, the second residual attention submodule BL2, the third residual attention submodule BL3 and the fourth residual attention submodule BL4 are exactly the same. However, the number of channels of the feature maps generated by the first residual attention submodule BL1, the second residual attention submodule BL2, the third residual attention submodule BL3 and the fourth residual attention submodule BL4 are different, which are 32, 64, 128 and 256 respectively.

[0092] Furthermore, if Figure 2 As shown, the data processing process of the first residual attention module is:

[0093] A) The input image is sequentially subjected to feature extraction through a 3*3*3 convolutional layer, a GN layer, a Dropout layer, and a rectified linear unit to obtain the first feature sub-graph. Specifically, the GN layer is used to normalize the data, the Dropout layer is used to prevent overfitting of the data, and the rectified linear unit (ReLU) added later is used for nonlinear mapping.

[0094] If the first residual attention module is the first residual attention sub-module BL1, the input image is the first image; if the first residual attention module is the second residual attention sub-module BL2, the input image is the image output by the first residual attention sub-module BL1; if the first residual attention module is the data processing of the third residual attention sub-module BL3, the input image is the image output by the second residual attention sub-module BL2, and so on.

[0095] B) performing feature extraction on the input image through a 1*1*1 convolutional layer, a GN layer and a Dropout layer in sequence to obtain a second feature subgraph.

[0096] C) extracting features from the first feature subgraph through a 3*3*3 convolution layer, a GN layer, and a Dropout layer in sequence to obtain a third feature subgraph. In step C), not only the number of channels of the convolution operation is the same as that of step A), but also the size of the generated feature graph is the same as that of the input image, that is, the size of the first feature subgraph is the same as that of the third feature subgraph, and the size of the first feature subgraph is the same as that of the input image.

[0097] D) extracting features from the third feature subgraph through an attention module to obtain a fourth feature subgraph; wherein the introduction of the attention mechanism (CBAM, Convolutional block attention module) can make the network more focused on useful information. The attention module includes a channel attention module and a spatial attention module.

[0098] E) performing a jump connection and adding the second feature subgraph to the fourth feature subgraph to obtain an output image. Specifically, the output image is the input image of the second residual attention module; and in the data processing process of the fourth residual attention module, the output image is the first feature graph.

[0099] Among them, Figure 3 As shown, the data processing process of the attention module in step D) is:

[0100] D1) extracting features from the third feature subgraph through a channel attention module to obtain a channel attention feature graph. Specifically, the third feature subgraph undergoes maximum pooling and average pooling respectively, and then enters a shared multi-layer perceptron. After the feature graphs after the shared multi-layer perceptron undergo maximum pooling and average pooling respectively, the maximum pooling results and the average pooling results are sequentially subjected to element-wise addition operations and sigmoid activation operations to obtain a channel attention feature graph.

[0101] D2) performing an element-wise multiplication operation on the channel attention feature map and the third feature sub-map to obtain an intermediate feature sub-map.

[0102] D3) Extract features from the intermediate feature sub-graphs through the spatial attention module to obtain a spatial attention feature graph. Specifically, the intermediate feature sub-graphs are concatenated by maximum pooling and average pooling in sequence, and then the feature graphs are reduced to one dimension by a 1×1×1 convolution operation, and then the spatial attention feature graphs are obtained by a sigmoid operation.

[0103] D4) performing an element-wise multiplication operation on the spatial attention feature map and the intermediate feature sub-map to obtain a fourth feature sub-map.

[0104] Furthermore, the second attention network includes a second fusion module and multiple second residual attention modules; the multiple second residual attention modules are connected in series; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head second residual attention module is the input end of the second attention network; the input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

[0105] In a specific embodiment of the present invention, since the structure of the second attention network is the same as that of the first attention network, the processing steps of the second attention network for the first parameter image are also the same as the processing steps of the first image in the first attention network, which will not be repeated here.

[0106] Preferably, in a specific embodiment of the present invention, the process of constructing a training sample specifically includes:

[0107] 1) Acquire multiple original images; the original images include an original three-dimensional dynamic enhanced magnetic resonance image and an original parameter image corresponding to the original three-dimensional dynamic enhanced magnetic resonance image, that is, the original three-dimensional dynamic enhanced magnetic resonance image and the original parameter image corresponding to the original three-dimensional dynamic enhanced magnetic resonance image belong to the same patient. Specifically, the original three-dimensional dynamic enhanced magnetic resonance image is composed of multiple two-dimensional slices. Among them, one two-dimensional slice shows the pathological condition at a certain section, and multiple two-dimensional slices show multiple places. The original three-dimensional dynamic enhanced magnetic resonance image is obtained by combining multiple two-dimensional slices. In a specific embodiment of the present invention, patients who underwent breast DCE-MRI examinations in a hospital from May 2017 to April 2020 were collected, and a total of 1,230 cases were collected. The most obviously enhanced phase in the DCE-MRI image of each patient is selected as the original three-dimensional dynamic enhanced magnetic resonance image.

[0108] There are 8 types of original parameter images; the original parameter images include 4 types of semi-quantitative parameter maps and 4 types of quantitative parameter maps; the semi-quantitative parameter maps are obtained by describing the shape and structure of the tissue signal intensity-time curve in the primary lesion area, and the semi-quantitative parameters include the initial area under the curve (AUC), the time to peak (TTP), the maximum slope (MaxSlope) and the maximum concentration of the contrast agent (MaxConc). The quantitative parameter map needs to select a pharmacokinetic model that matches the tissue blood supply status, and the quantitative parameters include the volume transfer constant (Ktrans), the outflow rate constant (Kep), the extravascular extracellular space volume fraction (Ve) and the plasma space volume fraction (Vp). These parameters reveal the hemodynamic characteristics of the lesion, evaluate the formation of tumor angiogenesis, and play an important role in the diagnosis of breast cancer. In addition, since the original parameter images include 8 categories, the number of channels for feature extraction in the second attention network is also 8.

[0109] 2) Preprocessing the plurality of original three-dimensional dynamic enhanced magnetic resonance images to obtain a first image; the preprocessing includes resampling, cropping, standardization and data enhancement. Specifically, it includes: a) resampling each original three-dimensional dynamic enhanced magnetic resonance image so that the pixel intervals of the images generated by different scanning devices remain consistent. b) The radiologist annotates the lesion area of ​​each two-dimensional slice of the resampled original three-dimensional dynamic enhanced magnetic resonance image with a two-dimensional rectangular frame to obtain a plurality of two-dimensional rectangular frames; and synthesizes the plurality of two-dimensional rectangular frames into a three-dimensional rectangular block containing the lesion area. Specifically, the three-dimensional rectangular block containing the lesion area is a mask containing only 0 and 1. The primary lesion area of ​​the resampled original three-dimensional dynamic enhanced magnetic resonance image is cropped according to the size of the above-mentioned three-dimensional rectangular block (mask) to obtain a three-dimensional image block. The cropped size is the smallest rectangular block that can cover the primary lesion area, and the primary lesion area is located at the center of the rectangular block. Because the size of each primary lesion area is different, the size of the three-dimensional rectangular block (mask) obtained by annotating each original three-dimensional dynamic enhanced magnetic resonance image is also different. Therefore, the cropped image needs to be unified before being input into the convolutional neural network. In this embodiment, the size of each three-dimensional image block is unified to (8, 80, 80); specifically, the unified size of each three-dimensional image block is the average of all mask sizes. c) The three-dimensional image blocks after the unified size are sequentially standardized and data enhanced to obtain a first image; wherein the data enhancement processing includes flipping, rotation, translation and scaling operations to prevent overfitting caused by data volume problems.

[0110] 3) Preprocessing the plurality of original parameter images to obtain a first parameter image. Specifically, the method comprises: resampling the eight parameter images corresponding to each original DCE-MRI image so that the pixel intervals of the images generated by different scanning devices remain consistent. Then, the eight resampled parameter images are cropped according to the size of the three-dimensional rectangular blocks obtained after annotating the original DCE-MRI image to generate eight three-dimensional parameter image blocks. The eight three-dimensional parameter image blocks are then subjected to standardization processing and data enhancement processing in turn to obtain a first parameter image.

[0111] 4) Determining label information according to the first image. Specifically, the label information is obtained by performing histopathological detection on the first image.

[0112] Furthermore, in a specific embodiment of the present invention, the data processing process of obtaining the third feature map after processing the first image using the attention mechanism and the multi-scale fusion algorithm specifically includes: inputting the feature map F1 generated by the second residual attention sub-module BL2 and the feature map F2 generated by the third residual attention sub-module BL3 into the multi-scale fusion module to perform the multi-scale fusion algorithm to obtain the third feature map.

[0113] like Figure 4 As shown, specifically, by downsampling, the scale of the feature map F1 and the scale of the feature map F2 are matched and spliced ​​to obtain a merged feature map; the merged feature map is processed by a 1*1*1 convolution layer and a binary classification model in sequence to obtain a first spatial weight factor S2 and a second spatial weight factor S3; a sigmoid operation is performed on the feature map F1 to obtain a first soft attention feature map; a sigmoid operation is performed on the feature map F2 to obtain a second soft attention feature map; the first spatial weight factor S2 and the first soft attention feature map are element-wise multiplied to obtain a first fine-grained feature map; the second spatial weight factor S3 and the second soft attention feature map are element-wise multiplied to obtain a second fine-grained feature map; the first fine-grained feature map and the second fine-grained feature map are element-wise added to obtain a third feature map F3.

[0114] In a specific embodiment of the present invention, the process of using the attention mechanism and the multi-scale fusion algorithm to process the first parameter image to obtain the fourth feature map is the same as the process of using the attention mechanism and the multi-scale fusion algorithm to process the first parameter image to obtain the third feature map, and will not be repeated here.

[0115] Furthermore, in a specific embodiment of the present invention, the data processing process of the fusion network is as follows:

[0116] The third feature map is input into the first fully connected module to obtain a first result; the first result is the predicted probability of the lesion category of the primary lesion area in the first image.

[0117] The fourth feature map is input into the second fully connected module to obtain a second result; the second result is the predicted probability of the lesion category of the primary lesion area in the first parameter image.

[0118] The fifth feature map is input into the third fully connected module to obtain a third result; the third result is the predicted probability of the lesion category of the primary lesion area in the fifth feature map; wherein the fifth feature map is obtained by splicing the first feature map obtained according to the first image and the second feature map obtained according to the first parameter image.

[0119] The first result, the second result and the third result are averaged to obtain the predicted probability of the lesion category that is finally output.

[0120] The first fully connected module, the second fully connected module and the third fully connected module have the same structure; the first fully connected module includes a global average pooling layer, a fully connected layer and a softmax classification layer connected in sequence. And global average pooling can avoid overfitting and realize lightweight design of the network.

[0121] In a specific embodiment of the present invention, the specific operation steps of the multimodal breast magnetic resonance image classification method are as follows:

[0122] Step 1, obtaining an original three-dimensional DCE-MRI image of a patient; Step 2, obtaining 8 parameter maps corresponding to the original three-dimensional DCE-MRI image; Step 3, sequentially performing resampling, cropping and standardization processing on the original three-dimensional DCE-MRI image of a patient to obtain a first image; Step 4, sequentially performing resampling, cropping and standardization processing on the 8 parameter maps of a patient to obtain a first parameter image; Step 5, inputting both the first image and the first parameter image into a trained target classification network to obtain a predicted probability of a lesion category; Step 6, determining a lesion category according to the predicted probability of the lesion category.

[0123] Embodiment 2

[0124] like Figure 5 As shown, this embodiment provides a multimodal breast magnetic resonance image classification system, including:

[0125] The first data acquisition unit 200 is used to acquire the magnetic resonance image to be classified.

[0126] The second data acquisition unit 201 is used to acquire a parameter map corresponding to the magnetic resonance image to be classified, so as to obtain a parameter map to be classified.

[0127] The prediction probability determination unit 202 is used to output the magnetic resonance image to be classified and the parameter map to be classified to the target classification network to obtain the prediction probability of the lesion category.

[0128] The lesion category determination unit 203 is used to determine the lesion category according to the predicted probability of the lesion category.

[0129] The target classification network is trained based on training samples and calibrated convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is the lesion category of the primary lesion area in the first image; the structure of the calibrated convolutional neural network is a dual-input single-output network structure.

[0130] like Figure 6 As shown, the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network.

[0131] The input end of the first attention network is used to input the first image and the label; the first output end of the first attention network is used to output a first feature map; the first feature map is determined after feature extraction of the first image using the attention mechanism; the second output end of the first attention network is used to output a third feature map; the third feature map is obtained after processing the first image using the attention mechanism and a multi-scale fusion algorithm.

[0132] The input end of the second attention network is used to input the first parameter image and the label; the first output end of the second attention network is used to output a second feature map; the second feature map is determined after feature extraction of the first parameter image using the attention mechanism; the second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained after processing the first parameter image using the attention mechanism and a multi-scale fusion algorithm.

[0133] The first input end of the fusion network is used to input the fifth feature map; the fifth feature map is determined by splicing the first feature map and the second feature map; the second input end of the fusion network is used to input the third feature map; the third input end of the fusion network is used to input the fourth feature map; the output end of the fusion network is used to output the predicted probability of the lesion category.

[0134] Preferably, the first attention network includes a first fusion module and multiple first residual attention modules; the multiple first residual attention modules are connected in series; the output end of the terminal first residual attention module is the first output end of the first attention network, and the input end of the head first residual attention module is the input end of the first attention network; the input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

[0135] The second attention network includes a second fusion module and multiple second residual attention modules; the multiple second residual attention modules are connected in series; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head second residual attention module is the input end of the second attention network; the input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

[0136] Compared with the prior art, the present invention also has the following advantages:

[0137] (1) The present invention performs multimodal fusion of parameter maps of quantitative parameters and semi-quantitative parameters with original DCE-MRI images, so that the network can learn richer features and the features can complement each other.

[0138] (2) The present invention adds an attention mechanism based on the proposed residual network model, so that the network pays more attention to useful information and improves training efficiency.

[0139] (3) The present invention simultaneously integrates multi-scale features, and the multi-scale attention mechanism enables the network to adaptively learn more refined features in space, so that the network can obtain more accurate classification results.

[0140] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0141] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal breast magnetic resonance image classification method, characterized in that: The multimodal breast magnetic resonance image classification method comprises: acquiring a magnetic resonance image to be classified; Obtaining a parameter map corresponding to the magnetic resonance image to be classified to obtain a parameter map to be classified; wherein the parameter map includes 4 types of semi-quantitative parameter maps and 4 types of quantitative parameter maps; the semi-quantitative parameter map is obtained by describing the shape and structure of the tissue signal intensity-time curve in the primary lesion area, and the semi-quantitative parameters include the initial area under the curve, the peak time, the maximum slope and the maximum concentration value of the contrast agent; the quantitative parameter map selects a pharmacokinetic model that matches the tissue blood supply status, and the quantitative parameters include the volume transfer constant, the outflow rate constant, the extravascular extracellular space volume fraction and the plasma space volume fraction; Outputting the magnetic resonance image to be classified and the parameter map to be classified into a target classification network to obtain a predicted probability of a lesion category; determining a lesion category according to the predicted probability of the lesion category; The target classification network is trained based on training samples and calibration convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is the lesion category of the primary lesion area in the first image; the structure of the calibration convolutional neural network is a dual-input single-output network structure; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network; An input end of the first attention network is used to input the first image and the label; The first output end of the first attention network is used to output a first feature map; the first feature map is determined after extracting features from the first image using the attention mechanism; The second output end of the first attention network is used to output a third feature map; the third feature map is obtained by processing the first image using the attention mechanism and the multi-scale fusion algorithm; The input end of the second attention network is used to input the first parameter image and the label; The first output terminal of the second attention network is used to output a second feature map; the second feature map is determined after extracting features from the first parameter image using the attention mechanism; The second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained by processing the first parameter image using the attention mechanism and the multi-scale fusion algorithm; The first input end of the fusion network is used to input a fifth feature map; the fifth feature map is obtained by concatenating the first feature map and the second feature map; The second input end of the fusion network is used to input the third feature map; The third input terminal of the fusion network is used to input the fourth feature map; The output end of the fusion network is used to output the predicted probability of the lesion category.

2. The multimodal breast magnetic resonance image classification method according to claim 1, characterized in that: The first attention network includes a first fusion module and a plurality of first residual attention modules; The connection mode of the multiple first residual attention modules is a series connection; the output end of the terminal first residual attention module is the first output end of the first attention network, and the input end of the head end first residual attention module is the input end of the first attention network; The input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

3. The multimodal breast magnetic resonance image classification method according to claim 1, characterized in that: The second attention network includes a second fusion module and a plurality of second residual attention modules; The connection mode of the plurality of the second residual attention modules is a series connection; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head end second residual attention module is the input end of the second attention network; The input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

4. The multimodal breast magnetic resonance image classification method according to claim 2, characterized in that: The structures of the plurality of first residual attention modules are the same; Among them, the data processing process of the first residual attention module is: The input image is sequentially subjected to feature extraction through a 3*3*3 convolutional layer, a GN layer, a Dropout layer, and a rectified linear unit to obtain the first feature subgraph; Performing feature extraction on the input image through a 1*1*1 convolutional layer, a GN layer, and a Dropout layer in sequence to obtain a second feature subgraph; The first feature subgraph is subjected to feature extraction through a 3*3*3 convolutional layer, a GN layer, and a Dropout layer in sequence to obtain a third feature subgraph; Performing feature extraction on the third feature subgraph through an attention module to obtain a fourth feature subgraph; The second feature sub-graph is jump-connected with the fourth feature sub-graph and added to obtain an output image.

5. The multimodal breast magnetic resonance image classification method according to claim 4, characterized in that: The data processing process of the attention module is: Performing feature extraction on the third feature subgraph through a channel attention module to obtain a channel attention feature graph; Performing an element-wise multiplication operation on the channel attention feature map and the third feature sub-map to obtain an intermediate feature sub-map; Performing feature extraction on the intermediate feature sub-graph through a spatial attention module to obtain a spatial attention feature graph; An element-wise multiplication operation is performed on the spatial attention feature map and the intermediate feature sub-map to obtain a fourth feature sub-map.

6. The multimodal breast magnetic resonance image classification method according to claim 1, characterized in that: The process of determining the fifth characteristic map specifically includes: The first feature map and the second feature map are concatenated in a channel series manner to obtain a fifth feature map.

7. The multimodal breast magnetic resonance image classification method according to claim 1, characterized in that: The training sample construction process specifically includes: Acquire multiple original images; the original images include original three-dimensional dynamic enhanced magnetic resonance images and original parameter images corresponding to the original three-dimensional dynamic enhanced magnetic resonance images; Preprocessing the plurality of original three-dimensional dynamic enhanced magnetic resonance images to obtain a first image; the preprocessing includes resampling processing, cropping processing, standardization processing and data enhancement processing; Preprocessing the plurality of original parameter images to obtain a first parameter image; Tag information is determined according to the first image.

8. A multimodal breast magnetic resonance image classification system, characterized in that: The multimodal breast magnetic resonance image classification system comprises: A first data acquisition unit, used for acquiring a magnetic resonance image to be classified; The second data acquisition unit is used to acquire the parameter map corresponding to the magnetic resonance image to be classified to obtain the parameter map to be classified; wherein the parameter map includes 4 types of semi-quantitative parameter maps and 4 types of quantitative parameter maps; the semi-quantitative parameter map is obtained by describing the shape and structure of the tissue signal intensity-time curve in the primary lesion area, and the semi-quantitative parameters include the initial area under the curve, the peak time, the maximum slope and the maximum concentration value of the contrast agent; the quantitative parameter map selects a pharmacokinetic model that matches the tissue blood supply status, and the quantitative parameters include the volume transfer constant, the outflow rate constant, the extravascular extracellular space volume fraction and the plasma space volume fraction; A prediction probability determination unit, used for outputting the magnetic resonance image to be classified and the parameter map to be classified to a target classification network to obtain a prediction probability of a lesion category; A lesion category determination unit, used to determine the lesion category according to the predicted probability of the lesion category; The target classification network is trained based on training samples and calibration convolutional neural networks; the training samples include input data and labels; the input data includes a first image and a first parameter image corresponding to the first image; the first image is a three-dimensional dynamic enhanced magnetic resonance image with a primary lesion area; the label is the lesion category of the primary lesion area in the first image; the structure of the calibration convolutional neural network is a dual-input single-output network structure; the structure of the calibration convolutional neural network includes a first attention network, a second attention network and a fusion network; An input end of the first attention network is used to input the first image and the label; The first output end of the first attention network is used to output a first feature map; the first feature map is determined after extracting features from the first image using the attention mechanism; The second output end of the first attention network is used to output a third feature map; the third feature map is obtained by processing the first image using the attention mechanism and the multi-scale fusion algorithm; The input end of the second attention network is used to input the first parameter image and the label; The first output terminal of the second attention network is used to output a second feature map; the second feature map is determined after extracting features from the first parameter image using the attention mechanism; The second output end of the second attention network is used to output a fourth feature map; the fourth feature map is obtained by processing the first parameter image using the attention mechanism and the multi-scale fusion algorithm; The first input end of the fusion network is used to input a fifth feature map; the fifth feature map is determined by concatenating the first feature map and the second feature map; The second input end of the fusion network is used to input the third feature map; The third input terminal of the fusion network is used to input the fourth feature map; The output end of the fusion network is used to output the predicted probability of the lesion category.

9. The multimodal breast magnetic resonance image classification system according to claim 8, characterized in that: The first attention network includes a first fusion module and a plurality of first residual attention modules; The connection mode of the plurality of the first residual attention modules is a series connection; the output end of the first residual attention module at the end is the first output end of the first attention network, and the input end of the first residual attention module at the head end is the input end of the first attention network; The input end of the first fusion module is connected to the first calibrated residual attention module; the first calibrated residual attention module is any of the first residual attention modules, and the number of the first calibrated residual attention modules is greater than or equal to 2; the output end of the first fusion module is the second output end of the first attention network.

10. The multimodal breast magnetic resonance image classification system according to claim 8, characterized in that: The second attention network includes a second fusion module and a plurality of second residual attention modules; The connection mode of the plurality of the second residual attention modules is a series connection; the output end of the terminal second residual attention module is the first output end of the second attention network, and the input end of the head end second residual attention module is the input end of the second attention network; The input end of the second fusion module is connected to the second calibrated residual attention module; the second calibrated residual attention module is any second residual attention module, and the number of the second calibrated residual attention modules is greater than or equal to 2; the output end of the second fusion module is the second output end of the second attention network.

Images